foreman

The foreman is the orchestration engine for agentic workflows in Microbus. It manages the lifecycle of flows - instances of a workflow that progress through a series of steps, where each step executes a task defined by a microservice. The foreman persists all state in a SQL database, making workflows durable across restarts and recoverable after failures.

Under the hood the foreman is a thin Microbus adapter over Dwarf, the standalone workflow-orchestration engine that owns durable flow execution, scheduling, and recovery. This page documents the foreman’s bus API - the engine internals live in the Dwarf repository.

Workflows are defined as directed graphs by individual microservices. When a flow is created via Create, the foreman fetches the workflow’s graph, stores it, and immediately runs the flow — create-and-run is one transaction, so there is no separate start step and the flow is running the moment Create returns. A pool of worker goroutines picks up pending steps and dispatches them to the corresponding task endpoints. After a task completes, the foreman evaluates the graph’s transitions to determine the next step(s) and enqueues them. This continues until the flow reaches a terminal status: completed, failed, terminated, or cancelled.

Flow Lifecycle

A flow progresses through these statuses:

created - a transient internal state during Create; flows are returned already running
running - actively executing steps
interrupted - paused, waiting for external input via Resume
completed - all steps finished successfully
failed - a step returned an error
terminated - forcefully stopped via Terminate
cancelled - gracefully stopped via Cancel, with every unrecovered loss in the flow a cancellation

The status constants live in the workflow package as workflow.StatusCreated, workflow.StatusRunning, etc.

Snapshot returns a *workflow.FlowOutcome with the current status, state, and any populated side-channel field (Error, InterruptPayload, TerminateReason, CancelReason) for the matching status. History returns the step-by-step execution history. HistoryMermaid renders the history as an HTML page with a Mermaid diagram.

Awaiting and Polling

Await and Run block until the flow stops and return its *workflow.FlowOutcome — Run creates-and-blocks in one call, Await blocks on an already-created flow. Poll is the long-poll sibling of Await: it waits up to the request’s time budget for the flow to stop, but a timeout is not an error. A still-running flow comes back as a non-error outcome with outcome.Stopped() == false, so a caller bridging an open-ended flow to a bounded HTTP request can answer within its own budget and re-poll immediately, holding the connection efficiently instead of fixed-interval hammering. PollAndParse additionally unmarshals the terminal state into a typed struct, the way AwaitAndParse does for Await.

A non-nil Go error on any of these means a transport, foreman, or timeout failure; a workflow failure is signalled by outcome.Status == "failed" with outcome.Error populated. An Await or Run that runs out of time before the flow stops fails with a 408 that also satisfies errors.Is(err, context.DeadlineExceeded). To read a flow’s current outcome without blocking at all, use Snapshot. The bank support example drives a browser off Poll end-to-end.

Cancelling and Terminating

There are two ways to stop a flow, and they differ in whether the workflow gets a say.

Cancel(flowKey, reason) is graceful. Every step in progress when it is called is covered: a step that has not started does not run, and a step that is already running finishes, after which its next transition is not taken. A covered step receives the cancellation as an error on its node’s onError transition, like any task error except that the task’s own changes are kept. workflow.IsCancelled(onErr) tells a cancellation apart from a real failure, so the handler can compensate and let the flow carry on, or stop. With no onError transition the flow ends cancelled, or failed if a real error is mixed in. Cancel returns once the request is recorded, so Await the flow to see it stop. A sleeping or backed-off step is reached only when it next wakes, and an interrupted flow stays interrupted until it is resumed or terminated. A flow is cancelled once: a repeat Cancel does nothing, which makes a call whose response was lost safe to retry.

Terminate(flowKey, reason) is forceful. It unconditionally stops a running or interrupted flow and its whole subgraph hierarchy, abandoning in-flight work rather than awaiting it, and the flow ends terminated. Use it to stop a flow that recovered from a Cancel, a parked flow, or one that must stop now regardless of what it is doing.

Both are addressed by the root flow key and carry their reason to the outcome, as CancelReason and TerminateReason respectively. A reason longer than 4 KB is rejected with a 400. Terminate of a flow that has already ended does nothing and returns no error, so it too is safe to retry.

Fork, Continue, and Lineage

A terminal flow (completed, failed, terminated, or cancelled) is immutable — the foreman never re-runs it in place. The only operations on a terminal flow are read (Snapshot, History, Step) and removal (Delete, Purge). To recover from a failure or explore an alternative, Fork(stepKey, stateOverrides) clones the flow’s prefix up to the chosen step into a new, self-contained running flow and re-executes from that step with stateOverrides applied to it; the original is never touched. The fork point may be any recorded step, including one inside a subgraph — the clone re-runs from there and bubbles back up to the root. The fork inherits the origin flow’s scheduling and baggage. Pass nil for no overrides.

Continue creates a new flow from the latest completed flow in a thread, chaining multi-turn workflows under a single threadKey. The new flow is returned already running and inherits the thread’s policy (priority, fairness, time budget, baggage) from the latest completed flow. To join a thread but set policy explicitly, use Create with FlowOptions.ThreadKey instead.

Failure Recovery

The foreman uses a lease-based model for step execution. When a worker picks up a step, it leases it for the step’s time budget plus a 30-second margin. If the replica running it crashes, the lease lapses and a sweep that runs every minute returns the step to pending for another worker, so the step runs again within TimeBudget + 30 s + 1 minute. Execution is therefore at least once, and tasks should be idempotent. A replica that dies after recording a step’s outcome but before creating the step that follows is repaired by the same sweep: the transition is replayed from the recorded outcome without running the task again.

A flow that has already failed is recovered with Fork: address the failed step (its key comes from History), apply any state overrides that let it succeed, and Fork clones the prefix up to that step into a new running flow that re-executes from there. Because the fork is an ordinary new flow rather than a mutation of the original, a partially-failed fan-out is recovered by forking one failed branch at a time.

The List → read Error → fix → Fork loop ties this together for recovering failed flows.

Backpressure Is Task-Owned

The foreman dispatches every step on the worker’s hot path but runs no dispatch-rate controller of its own. It does not inspect a task’s status code or error text: any error a task returns is terminal for that attempt, routing via the graph’s onError transition if one exists, otherwise failing the step. Backing off a transient failure (a 429 from a rate-limited provider, a 503 from a momentarily-unavailable downstream) is the task’s job — it detects the condition and arms flow.Retry, whose wall-clock horizon parks the step rather than holding a worker. A 503 or 529 from a present microservice is an ordinary step failure, because that microservice owns its own retry against its downstream — the engine never reads status codes.

The one exception is a task whose hosting microservice is absent: a 404 ack-timeout (no responder), which the task body cannot catch because it never ran. The foreman arms flow.Retry on the carrier and re-probes on a backoff bounded by the step’s time budget, so a short absence such as a rolling deployment heals transparently. A 404 that persists past the budget becomes an ordinary step failure with the original ack timeout: error. This is the only engine-level retry. It is surfaced via microbus_foreman_timeout_requests_total{task_url, outcome} — outcome="retry" per re-probe, outcome="giveup" when the horizon is spent (the alertable “a microservice is missing” signal) — and retry-dispatch churn shows up in dwarf_steps_executed_total{task_name, status="retried"}.

Debugging

Step(stepKey) returns a single step’s full payload — state, changes, status, error, and timings — for operator tooling and the agentstudio dev UI, useful for inspecting state mid-flow during development. To pause a flow at a chosen point, author the pause into the workflow with flow.Interrupt and release it with Resume; to explore an alternative path from a recorded step, Fork from that step into a new flow.

Querying Flows

List filters by status, workflow URL or name, the task the flow stopped at (StoppedTaskName), fairness key (typically the tenant), thread, priority, shard, or age (OlderThan / NewerThan), and returns results in reverse chronological order. StoppedTaskName is the task whose step completed, failed or cancelled the flow, or the one it is interrupted at; it is empty while the flow is running, so a StoppedTaskName filter never matches a running flow. The free-text Search field is a case-insensitive substring match over the workflow URL and name, StoppedTaskName, the error, the terminate and cancel reasons, and the flow key — the way to find failed flows by why they failed (e.g. Search: "ack timeout" for missing-microservice give-ups). Each FlowSummary carries the failed flow’s Error text and its StoppedTaskName. It uses cursor-based pagination: List returns (flows []FlowSummary, nextCursor string); pass nextCursor back as Query.Cursor on the next call to fetch the next page. Default page size is 100. The dwarf_flows and dwarf_steps tables carry a created_at index so time-window queries (recent flows, archival scripts, dashboards) remain efficient as history grows. These queries are how an operator detects failed flows before recovering them.

By default List returns only root flows. Query.IncludeSubgraphs adds subgraph-child flows to the results, and each FlowSummary.Subgraph marks which kind a flow is — combined with a WorkflowURL filter, this finds every run of a graph that executed as a subgraph.

ShardInfo returns a ShardSummary per database shard: its Index, the Latency of a trivial query as a time.Duration, and the query’s Error, empty when the shard answered. Use it for operator dashboards and readiness probes.

Flow Cleanup

The foreman does not auto-purge flows - every row remains potentially resurrectable (an interrupted flow via Resume, a completed flow via Continue into the same thread, a failed, terminated or cancelled flow via Fork), so a fixed-timer purge would silently break any of these patterns. Two operator-driven endpoints take their place:

Delete(flowKey) removes a flow and its whole subgraph subtree (the root flow, its steps, and every descendant) from the database. The flow must not be running; an interrupted flow is terminated first. It must be addressed by the root flow key — like the other lifecycle operations, it rejects a subgraph-child key with 400, since it acts on the whole tree. Thread lineage references to the deleted flow become dangling.
Purge(query) bulk-deletes flows matching the same Query shape as List, except those currently running, and returns how many it deleted. It selects matching root flows and deletes each one’s whole subtree, so IncludeSubgraphs is rejected with 400, as is a query that sets no filter. Interrupted flows are terminated first. A call deletes at most 4096 root flows (fewer when Query.Limit is lower), so an operator script calls it until it returns 0 without locking up a shard.

Database-side partitioning or TTL is still a valid alternative for very high-volume deployments; Delete / Purge are the in-protocol option for everything else.

Database Sharding

The Shards config distributes flows across multiple database instances, one entry per shard. Each shard is opened and migrated independently, and each names its own database — there is no templated connection string. Shards can be added but never removed, and applying a change is a coordinated restart of the whole fleet. A subgraph’s child flow is always created on the same shard as its parent. The full sharding model is part of getting a deployment production-ready.

Flow Scheduling

Create and Run accept an optional *workflow.FlowOptions argument that sets the flow’s priority, fairness key/weight, time budget, and thread membership (ThreadKey). When more flows are ready than there are workers, the foreman fills the best priority band first (lower number first) and gives the capacity it leaves unused to the next band, and shares each band fairly across fairness keys so one busy tenant cannot starve the rest. Priorities run from 1 to the MaxPriority config; Create and Run reject a higher one with a 400. A nil FlowOptions, or any zero field, uses the foreman’s defaults (the midpoint of the priority range, the tenant actor claim as the fairness key, weight 1). Policy is authored once at genesis (Create/Run); the derived operations take no options — Continue inherits the thread’s policy, Fork inherits the origin’s. The priority and fairness guide covers the concepts and usage in full.

A deferred start is expressed in the workflow itself — an entry task that calls flow.Interrupt, released by Resume — not via a scheduling option.

Configuration

foreman.core:
  Shards: '[{"index":1,"dsn":"postgres://foreman:secret@db1.internal:5432/flows"}]'
  TimeBudget: 2m
  MaxPriority: 5

Shards declares the database shards that hold flows and steps, one entry per shard, as a JSON array. It is a secret config, since every entry carries a database credential. A shard needs only its index and dsn; the other fields are optional:

Field	Meaning
`index`	Identifies the shard; `>= 1`, unique, and stable across restarts. It is encoded into every flow key created on the shard, so the index-to-DSN mapping must be identical on every replica. Indices need not be contiguous.
`dsn`	The shard’s connection string, dialect auto-detected. Used verbatim — never formatted or rewritten, so a percent-encoded credential survives intact. MySQL, PostgreSQL, SQL Server and SQLite are supported.
`maxConnections`	The shard’s connection budget, shared by the replicas that dispatch it. Left at `0`, the engine derives it from the measured round trip and the server’s `max_connections`. Declare the same value on every replica.
`weight`	The shard’s relative share of new flows (default `1`). Any weight other than `1` switches the fleet to placement over all shards in proportion to their weights.
`cordoned`	Excludes the shard from new-flow placement while resident flows keep executing. The way to retire a shard.

Left empty, a LOCAL deployment falls back to a single file:shard_1.local.sqlite file shard, and TESTING opens isolated in-memory databases without consulting the config at all — so neither needs any configuration to run workflows.

Shards is applied once during startup and has no config callback: a change takes effect on the next restart, and only after a coordinated restart of every replica.

TimeBudget is the hard ceiling on the execution time of any task step, applied as the timeout on the task dispatch call (default 2m, capped at 15m). A flow may override it per-flow via FlowOptions.TimeBudget, and a task endpoint may declare a shorter budget of its own via sub.TimeBudget; the effective deadline is the smallest of these.

MaxPriority is the number of priority levels flows may use (default 5, at most 100). Priorities run from 1, served first, to MaxPriority, and a flow that sets none gets the midpoint, 3 at the default. Create and Run reject a priority above it with a 400. Set the same value on every replica; it takes effect on restart. The priority and fairness guide covers how the bands share capacity.

No single value the engine stores may exceed 256 MB: initial state, resume data, a step’s changes, or a flow’s final state. Create, Continue, Resume and Fork reject an oversized value with a 413 and create nothing; a step whose changes are too large fails, and so does a flow whose final state is.

Memory

The engine keeps large state fields in memory for the steps that read them next, in a cache sized at a tenth of the process’s memory limit: GOMEMLIMIT when set, otherwise the container’s cgroup limit, otherwise 1 GiB. Set GOMEMLIMIT on foreman hosts, a little below the memory the process may use (about 90% of its container’s limit), to bound the cache and the heap together. The dwarf_state_cache_bytes metric reports the cache’s size.

Expert Overrides

Workers caps the concurrent workers that process flow steps. At its default of -1 the engine derives the ceiling from each shard’s connection budget and the measured round-trip time, and grows the pool into it on demand. 0 is a distinct and meaningful setting rather than a derivation hint: it stands up a replica that creates flows, awaits them, and serves reads, but never executes a task. Performance tuning walks through the connection budget and when to set a shard’s maxConnections.