Task-Manager pattern (in elixir)
Imagine you are executing a long running Task that is communicating with external services, doing work, etc. You also want to inspect the progress. Problem is your long running Task isn’t running in a loop, can’t receive your requests to inspect the state and answer your questions.
When I was facing this problem I came up with what I call the Task-Manager pattern. The idea is that we attach a “manager” to the task.
Just like in real life, the actual job is being done by the worker process. Meanwhile the manager has time to answer emails, phone calls, report to his superiors. And if it’s a good manager (ours will be a good manager) protects the worker from the constant nagging of upper management (you in this case).
Create the manager
We will of course start with a GenServer
defmodule Manager do
use GenServer
def start_link(args) do
GenServer.start_link(__MODULE__, args, name: args[:task_name])
end
@impl true
def init(args) do
task = start_work(args[:task_name])
{:ok, %{task_name: task_name, task: task, done: false, result: nil}}
end
@doc """
Starts a Task under the task supervisor, returns the %Task{}
"""
def start_work(task_name) do
# the actual work being done, irrelevant for this demonstration
fun = fun_for_task_name(task_name)
# task is started with async, but instead of calling Task.await we will listen to the
# raw message emited by the task when finished (see later)
Task.Supervisor.async(MyApp.TaskSupervisor, fun)
end
end
Manager is a GenServer that will take a task_name and run the work associated with it in a supervised async task. This is not so useful yet. As you can see the GenServer’s state contains done: false, we will want to flip this to true when the Task finishes. When an asynchronous task finished it send a message to the parent process. This message is usually processed by Task.await, but we don’t want to block our manager, otherwise it couldn’t answer those important questions from higher ups. Instead we will be listening to the raw message emited by the task:
@impl true
def handle_info({ref, result}, state) do
# task finished
# ref is the task reference, stored at state.task.ref
# if you need you can check it with ref == task.ref
{:noreply, %{state | done: true, result: result}}
end
When the {ref, result} message is received from the task we save the result and set done to true.
Now we only need some boilerplate code so our manager can actually answer questions:
# interface
def get_task(manager) do
GenServer.call(manager, :get_task_name)
end
def done?(manager) do
GenServer.call(manager, :done?)
end
def get_result(manager) do
GenServer.call(manager, :get_result)
end
# backend functions, just send back the values from the GenServer state
@impl true
def handle_call(:get_task_name, _from, %{get_task_name: get_task_name} = state) do
{:reply, get_task_name, state}
end
@impl true
def handle_call(:done?, _from, %{done: done} = state) do
{:reply, done, state}
end
@impl true
def handle_call(:get_result, _from, %{result: result} = state) do
{:reply, result, state}
end
Now our manager can answer questions like “is it done yet?” “what is the worker working on?”
Send reports from the worker
What if we want to report progress or decisions made by the worker?
For that our worker will send reports to its manager when relevant steps are done
First our manager will provide a function for sending reports (and it will merge all previous reports together):
def send_report(server, report) do
GenServer.cast(server, {:send_report, report})
end
@impl true
def handle_cast({:send_report, new_report}, %{report: existing_report} = state) do
report = Map.merge(existing_report, new_report)
{:noreply, %{state | report: report}}
end
@impl true
def handle_call(:done?, _from, %{done: done} = state) do
{:reply, done, state}
end
@impl true
def handle_call(:get_report, _from, %{report} = state) do
{:reply, report, state}
end
And our worker will start sending reports:
def long_running_work(manager) do
Manager.send_report(manager, %{started_work: true})
# do some work
Manager.send_report(manager, %{phase1_done: true})
database_records_generated = generate_records() # irrelevant
Manager.send_report(manager, %{database_records_generated: database_records_generated})
# do some more work
end
Summary
We implemented a manager that can start a worker task, receive reports from the worker, handle when the worker task finished and answer questions about the current state of the work. The worker process can do its job, won’t be interrupted and doesn’t need to stop its work to see if someone is asking about the progress.
Miscellaneous
Above I only presented the minimal code to explain the patter. I’ll describe some minor improvements here.
Labelling our manager and worker processes will show a clear name in :observer or Phoenix LiveDashboard:
Process.set_label({:manager, task_name})
Logger.metadata(task_name: task_name)
It’s also a good idea to shut down the worker when the manager quits. But pay attention to how does this affect your business logic
@impl true
def terminate(reason, %{task: task} = state) do
Logger.info("Manager terminating", reason: reason, state: state)
# clean up resources
if task do
Task.shutdown(task)
end
end
I also like to save the report into Logger.metadata of the Task as well (note that send_report is called from the task, so Logger.metadata refers to the task's metadata):
def send_report(server, report) do
if Logger.metadata()[:report] do
key_values = Keyword.merge(Logger.metadata()[:state_report], key_values)
Logger.metadata(state_report: key_values)
else
Logger.metadata(state_report: key_values)
end
GenServer.cast(server, {:report_state, key_values})
end