Priority and Fairness

When more flows are ready to run than the Foreman has workers to execute, something has to decide what runs next. By default every flow is treated equally, which is fine until an urgent flow is stuck behind a long batch job, or one busy tenant’s flood of work crowds out everyone else. Microbus gives you two dials to control this: priority and fairness. Both are set per flow through workflow.FlowOptions at create time — you never tune the scheduler itself.

Priority

Priority decides which flow runs first. Priorities run from 1 to the Foreman’s MaxPriority config (5 by default), and lower numbers run first — a flow at priority 1 is dispatched ahead of a flow at priority 5. Leaving it at 0 (the zero value) is not priority zero; it means “unset”, and the flow takes the midpoint of the range, 3 by default. Create and Run reject a priority above MaxPriority with a 400.

Priority is strict over capacity, not absolute. The scheduler fills the best band that has due work first and gives the capacity it leaves unused to the next band, and so on down. A light but steady stream of urgent flows therefore never leaves workers idle while a backlog waits below it, but lower-priority work gets only what higher-priority work does not use: as long as higher-priority work keeps every worker busy, lower-priority flows wait. There is no aging. This is intentional — it lets you guarantee that an interactive, user-facing flow is never delayed by background batch work. Choose your numbering with that in mind: reserve the low numbers for genuinely urgent flows and let bulk work default.

Priority is not preemptive: a task that is already running finishes. Because the worker pool grows on demand, newly arrived urgent work normally starts as soon as the scheduler sees it, and waits for a running task only when every worker the Foreman may run is busy. Where that can happen and urgent latency matters, keep lower-priority tasks short so workers free up often.

A flow that doesn’t set a priority runs at the midpoint, so the common pattern is to leave most flows unset, mark the few that need to jump the queue with 1 or 2, and push background work back with 4 or 5:

import "github.com/microbus-io/dwarf/workflow"

// Expedite this flow ahead of unmarked work, which runs at 3.
flowID, err := foremanapi.NewClient(svc).Create(ctx, myserviceapi.CreditApproval.URL(),
    map[string]any{"applicant": applicant},
    &workflow.FlowOptions{Priority: 1},
)

Fairness

Priority alone doesn’t stop a single tenant from monopolizing the workers — if one customer submits ten thousand flows at the default priority, everyone else’s default-priority flows wait behind them. Fairness solves this by sharing dispatch within a priority band across fairness keys.

A fairness key is just a label that groups related flows — most often a tenant or customer ID. Among flows of the same priority, the Foreman shares dispatch across the distinct fairness keys rather than draining them in submission order, so a tenant with a huge backlog gets its turn but not the whole pool. Within a single key, dispatch is oldest-first. Crucially, a key’s share does not grow with its backlog: submitting more work does not buy more throughput, which is what keeps one noisy tenant from starving the rest.

If you don’t set a fairness key, the Foreman derives one from the request’s tid or tenant actor claim when present, and otherwise places the flow in a shared default bucket. In a multi-tenant service where calls already carry a tenant claim, fairness works correctly with no code changes.

Set it explicitly when the grouping isn’t the tenant claim — for example, fairness per end-user within a tenant:

&workflow.FlowOptions{FairnessKey: "user-" + userID}

Fairness Weight

By default every fairness key gets an equal share. FairnessWeight changes a key’s relative share: a key with weight 2.0 receives roughly twice the dispatch share of a key at the default weight of 1.0, when both have work waiting. Use it for tiered service levels — say, premium tenants getting a larger slice than free-tier ones:

&workflow.FlowOptions{FairnessKey: tenantID, FairnessWeight: 2.0}

Weight only matters relative to the other keys competing at the same priority; a key with no waiting work consumes none of its share. If flows sharing a key carry different weights, the key dispatches at the lowest weight among its due work, so give one key one weight.

Setting Flow Options

workflow.FlowOptions is passed when a flow is created. All fields are optional — a nil *workflow.FlowOptions, or any zero-valued field, falls back to the Foreman’s defaults.

type FlowOptions struct {
    Priority           int           // 1 to MaxPriority, lower runs first; 0 means unset → the midpoint
    FairnessKey        string        // groups flows; empty derives from the tenant claim
    FairnessWeight     float64       // relative share of the key; 0 means weight 1
    TimeBudget         time.Duration // per-task time budget for this flow, at most 15m; 0 uses the Foreman's default
    DeleteOnCompletion bool          // delete the flow shortly after it completes successfully
    Baggage            any           // opaque context for every task; the Foreman fills it with the caller's actor claims
    ThreadKey          string        // join an existing thread; empty starts a fresh one
}

These fields are frozen at Create and immutable for the life of the flow. Priority, FairnessKey, FairnessWeight, and TimeBudget are inherited unchanged by Continue, Fork, and subgraphs. ThreadKey is the explicit-policy way to add a turn to an existing thread — pass any flow key already in the thread, and the new flow joins it while you set its scheduling explicitly. (Continue is the inherit-everything convenience that copies the thread’s policy; the two can be mixed.) A flow whose start should be deferred is authored in the workflow itself — an entry task that calls flow.Interrupt, released by Resume — not via a scheduling option.

TimeBudget overrides the Foreman’s default per-task time budget for every ExecuteTask dispatch in this flow — a per-task ceiling, not a flow-wide deadline — and bounds each task’s context. It is frozen at Create. A task’s own sub.TimeBudget endpoint declaration still applies; the effective deadline is the smaller of the two.

A workflow is invoked by submitting it to the Foreman. Pass the options as the last argument to Create or Run (use nil when you want the defaults):

client := foremanapi.NewClient(svc)

outcome, err := client.Run(ctx, workflowURL, initialState,
    &workflow.FlowOptions{Priority: 1, FairnessKey: tenantID},
)

The same final argument applies to client.Create when you start and await the flow separately.

Operating Notes

The range lives on the Foreman. MaxPriority is a foreman.core config, at most 100, that sets how many bands there are; unmarked flows always sit at its midpoint. Set the same value on every replica. Raising it to 9, say, gives four bands on either side of the default.
Priority is soft across replicas. The scheduler works from periodic snapshots of what is due, and each Foreman replica serves its own share of the work first, so for arrivals within a cycle or two of each other a little lower-priority work can start while higher-priority work waits briefly on another replica. Ordering is honored closely but not as a single global queue. Treat priority as “strongly preferred”, not a hard real-time guarantee.
Observability. The engine exports pending steps and the age of the oldest due step per shard and priority band (dwarf_steps_pending, dwarf_steps_oldest_pending_age_seconds), and flows started and ended per band (dwarf_flows_started_total, dwarf_flows_ended_total). A lower band whose age climbs while dwarf_steps_dispatch_lag_seconds stays low is being held back by better-priority work, as designed. There are no per-fairness-key metrics, since a label per key would be unbounded. These feed the bundled Workflow Overview Grafana dashboard, and Production Readiness explains how to aggregate them.