hangyas.me

Personal website of hangyas (Krisztián Ádám) about programming and whatever I feel like


css: base html | fun | blue | renaissance

Task-Manager pattern (in elixir)

Imagine you are executing a long running Task that is communicating with external services, doing work, etc. You also want to inspect the progress. Problem is your long running Task isn’t running in a loop, can’t receive your requests to inspect the state and answer your questions.

When I was facing this problem I came up with what I call the Task-Manager pattern. The idea is that we attach a “manager” to the task.

Just like in real life, the actual job is being done by the worker process. Meanwhile the manager has time to answer emails, phone calls, report to his superiors. And if it’s a good manager (ours will be a good manager) protects the worker from the constant nagging of upper management (you in this case).

Create the manager

We will of course start with a GenServer

defmodule Manager do
  use GenServer

  def start_link(args) do
    GenServer.start_link(__MODULE__, args, name: args[:task_name])
  end

  @impl true
  def init(args) do
    task = start_work(args[:task_name])

    {:ok, %{task_name: task_name, task: task, done: false, result: nil}}
  end

  @doc """
  Starts a Task under the task supervisor, returns the %Task{}
  """
  def start_work(task_name) do
    # the actual work being done, irrelevant for this demonstration
    fun = fun_for_task_name(task_name)

    # task is started with async, but instead of calling Task.await we will listen to the
    # raw message emited by the task when finished (see later)
    Task.Supervisor.async(MyApp.TaskSupervisor, fun)
  end
end

Manager is a GenServer that will take a task_name and run the work associated with it in a supervised async task. This is not so useful yet. As you can see the GenServer’s state contains done: false, we will want to flip this to true when the Task finishes. When an asynchronous task finished it send a message to the parent process. This message is usually processed by Task.await, but we don’t want to block our manager, otherwise it couldn’t answer those important questions from higher ups. Instead we will be listening to the raw message emited by the task:

@impl true
def handle_info({ref, result}, state) do
  # task finished
  # ref is the task reference, stored at state.task.ref
  # if you need you can check it with ref == task.ref

  {:noreply, %{state | done: true, result: result}}
end

When the {ref, result} message is received from the task we save the result and set done to true.

Now we only need some boilerplate code so our manager can actually answer questions:

# interface

def get_task(manager) do
  GenServer.call(manager, :get_task_name)
end

def done?(manager) do
  GenServer.call(manager, :done?)
end

def get_result(manager) do
  GenServer.call(manager, :get_result)
end

# backend functions, just send back the values from the GenServer state

@impl true
def handle_call(:get_task_name, _from, %{get_task_name: get_task_name} = state) do
  {:reply, get_task_name, state}
end

@impl true
def handle_call(:done?, _from, %{done: done} = state) do
  {:reply, done, state}
end

@impl true
def handle_call(:get_result, _from, %{result: result} = state) do
  {:reply, result, state}
end

Now our manager can answer questions like “is it done yet?” “what is the worker working on?”

Send reports from the worker

What if we want to report progress or decisions made by the worker?

For that our worker will send reports to its manager when relevant steps are done

First our manager will provide a function for sending reports (and it will merge all previous reports together):

def send_report(server, report) do
  GenServer.cast(server, {:send_report, report})
end

@impl true
def handle_cast({:send_report, new_report}, %{report: existing_report} = state) do
  report = Map.merge(existing_report, new_report)
  {:noreply, %{state | report: report}}
end

@impl true
def handle_call(:done?, _from, %{done: done} = state) do
  {:reply, done, state}
end

@impl true
def handle_call(:get_report, _from, %{report} = state) do
  {:reply, report, state}
end

And our worker will start sending reports:

def long_running_work(manager) do
  Manager.send_report(manager, %{started_work: true})

  # do some work

  Manager.send_report(manager, %{phase1_done: true})

  database_records_generated = generate_records() # irrelevant

  Manager.send_report(manager, %{database_records_generated: database_records_generated})

  # do some more work
end

Summary

We implemented a manager that can start a worker task, receive reports from the worker, handle when the worker task finished and answer questions about the current state of the work. The worker process can do its job, won’t be interrupted and doesn’t need to stop its work to see if someone is asking about the progress.

Miscellaneous

Above I only presented the minimal code to explain the patter. I’ll describe some minor improvements here.

Labelling our manager and worker processes will show a clear name in :observer or Phoenix LiveDashboard:

Process.set_label({:manager, task_name})
Logger.metadata(task_name: task_name)

It’s also a good idea to shut down the worker when the manager quits. But pay attention to how does this affect your business logic

@impl true
def terminate(reason, %{task: task} = state) do
    Logger.info("Manager terminating", reason: reason, state: state)

    # clean up resources
    if task do
        Task.shutdown(task)
    end
end

I also like to save the report into Logger.metadata of the Task as well (note that send_report is called from the task, so Logger.metadata refers to the task's metadata):

def send_report(server, report) do
    if Logger.metadata()[:report] do
        key_values = Keyword.merge(Logger.metadata()[:state_report], key_values)
        Logger.metadata(state_report: key_values)
    else
        Logger.metadata(state_report: key_values)
    end

    GenServer.cast(server, {:report_state, key_values})
end