foreman

The foreman is the orchestration engine for agentic workflows in Microbus. It manages the lifecycle of flows - instances of a workflow that progress through a series of steps, where each step executes a task defined by a microservice. The foreman persists all state in a SQL database, making workflows durable across restarts and recoverable after failures.

Under the hood the foreman is a thin Microbus adapter over Dwarf, the standalone workflow-orchestration engine that owns durable flow execution, scheduling, and recovery. This page documents the foreman’s bus API - the engine internals live in the Dwarf repository.

Workflows are defined as directed graphs by individual microservices. When a flow is created via Create, the foreman fetches the workflow’s graph, stores it, and immediately runs the flow — create-and-run is one transaction, so there is no separate start step and the flow is running the moment Create returns. A pool of worker goroutines picks up pending steps and dispatches them to the corresponding task endpoints. After a task completes, the foreman evaluates the graph’s transitions to determine the next step(s) and enqueues them. This continues until the flow reaches a terminal status: completed, failed, or cancelled.

Flow Lifecycle

A flow progresses through these statuses:

created - a transient internal state during Create; flows are returned already running
running - actively executing steps
interrupted - paused, waiting for external input via Resume
completed - all steps finished successfully
failed - a step returned an error
cancelled - explicitly cancelled via Cancel

The status constants live in the workflow package as workflow.StatusCreated, workflow.StatusRunning, etc.

Snapshot returns a *workflow.FlowOutcome with the current status, state, and any populated side-channel field (Error, InterruptPayload, CancelReason) for the matching status. History returns the step-by-step execution history. HistoryMermaid renders the history as an HTML page with a Mermaid diagram.

Awaiting and Polling

Await and Run block until the flow stops and return its *workflow.FlowOutcome — Run creates-and-blocks in one call, Await blocks on an already-created flow. Poll is the long-poll sibling of Await: it waits up to the request’s time budget for the flow to stop, but a timeout is not an error. A still-running flow comes back as a non-error outcome with outcome.Stopped() == false, so a caller bridging an open-ended flow to a bounded HTTP request can answer within its own budget and re-poll immediately, holding the connection efficiently instead of fixed-interval hammering. PollAndParse additionally unmarshals the terminal state into a typed struct, the way AwaitAndParse does for Await.

A non-nil Go error on any of these means a transport, foreman, or timeout failure; a workflow failure is signalled by outcome.Status == "failed" with outcome.Error populated. To read a flow’s current outcome without blocking at all, use Snapshot. The bank support example drives a browser off Poll end-to-end.

Fork, Continue, and Lineage

A terminal flow (completed, failed, or cancelled) is immutable — the foreman never re-runs it in place. The only operations on a terminal flow are read (Snapshot, History, Step) and removal (Delete, Purge). To recover from a failure or explore an alternative, Fork(stepKey, stateOverrides) clones the flow’s prefix up to the chosen step into a new, self-contained running flow and re-executes from that step with stateOverrides applied to it; the original is never touched. The fork point may be any recorded step, including one inside a subgraph — the clone re-runs from there and bubbles back up to the root. The fork inherits the origin flow’s scheduling and baggage. Pass nil for no overrides.

Continue creates a new flow from the latest completed flow in a thread, chaining multi-turn workflows under a single threadKey. The new flow is returned already running and inherits the thread’s policy (priority, fairness, time budget, baggage) from the latest completed flow. To join a thread but set policy explicitly, use Create with FlowOptions.ThreadKey instead.

Failure Recovery

The foreman uses a lease-based model for step execution. When a worker picks up a step, it reserves it for a time window. If the worker crashes or times out, the lease expires and a periodic poll recovers the step so another worker can pick it up.

A flow that has already failed is recovered with Fork: address the failed step (its key comes from History), apply any state overrides that let it succeed, and Fork clones the prefix up to that step into a new running flow that re-executes from there. Because the fork is an ordinary new flow rather than a mutation of the original, a partially-failed fan-out is recovered by forking one failed branch at a time.

The List → read Error → fix → Fork loop ties this together for recovering failed flows.

Backpressure Is Task-Owned

The foreman dispatches every step on the worker’s hot path but runs no dispatch-rate controller of its own. It does not inspect a task’s status code or error text: any error a task returns is terminal for that attempt, routing via the graph’s onError transition if one exists, otherwise failing the step. Backing off a transient failure (a 429 from a rate-limited provider, a 503 from a momentarily-unavailable downstream) is the task’s job — it detects the condition and arms flow.Retry, whose wall-clock horizon parks the step rather than holding a worker. A 503 or 529 from a present microservice is an ordinary step failure, because that microservice owns its own retry against its downstream — the engine never reads status codes.

The one exception is a task whose hosting microservice is absent: a 404 ack-timeout (no responder), which the task body cannot catch because it never ran. The foreman arms flow.Retry on the carrier and re-probes on a backoff bounded by the step’s time budget, so a short absence such as a rolling deployment heals transparently. A 404 that persists past the budget becomes an ordinary step failure with the original ack timeout: error. This is the only engine-level retry. It is surfaced via microbus_foreman_timeout_requests_total{task_url, outcome} — outcome="retry" per re-probe, outcome="giveup" when the horizon is spent (the alertable “a microservice is missing” signal) — and retry-dispatch churn shows up in dwarf_steps_executed_total{task_name, status="retried"}.

Debugging

Step(stepKey) returns a single step’s full payload — state, changes, status, error, and timings — for operator tooling and the agentstudio dev UI, useful for inspecting state mid-flow during development. To pause a flow at a chosen point, author the pause into the workflow with flow.Interrupt and release it with Resume; to explore an alternative path from a recorded step, Fork from that step into a new flow.

Querying Flows

List filters by status, workflow name, task name, tenant, thread, or age (OlderThan / NewerThan), and returns results in reverse chronological order. Its free-text Search field is a case-insensitive substring match over the workflow_url, workflow_name, current task_name, error, and cancel_reason columns — the way to find failed flows by why they failed (e.g. Search: "ack timeout" for missing-microservice give-ups). Each FlowSummary carries the failed flow’s Error text. It uses cursor-based pagination: List returns (flows []FlowSummary, nextCursor string); pass nextCursor back as Query.Cursor on the next call to fetch the next page. Default page size is 100. The microbus_flows and microbus_steps tables carry a created_at index so time-window queries (recent flows, archival scripts, dashboards) remain efficient as history grows. These queries are how an operator detects failed flows before recovering them.

By default List returns only root flows. Query.IncludeSubgraphs adds subgraph-child flows to the results, and each FlowSummary.Subgraph marks which kind a flow is — combined with a WorkflowURL filter, this finds every run of a graph that executed as a subgraph.

ShardInfo returns per-shard health (latency, row counts, last error) for every database shard. Use it for operator dashboards and readiness probes.

Flow Cleanup

The foreman does not auto-purge flows - every row remains potentially resurrectable (an interrupted flow via Resume, a completed flow via Continue into the same thread, a failed or cancelled flow via Fork), so a fixed-timer purge would silently break any of these patterns. Two operator-driven endpoints take their place:

Delete(flowKey) removes a flow and its whole subgraph subtree (the root flow, its steps, and every descendant) from the database. The flow must not be running. It must be addressed by the root flow key — like the other lifecycle operations, it rejects a subgraph-child key with 400, since it acts on the whole tree. Thread lineage references to the deleted flow become dangling.
Purge(query) bulk-deletes flows matching the same Query shape as List, except those currently running. It selects matching root flows and deletes each one’s whole subtree, so IncludeSubgraphs is rejected with 400. Capped at 1000 root flows per call so an operator script can paginate without locking up a shard.

Database-side partitioning or TTL is still a valid alternative for very high-volume deployments; Delete / Purge are the in-protocol option for everything else.

Database Sharding

The NumShards config distributes flows across multiple database instances. Each shard is opened and migrated independently. Shards can be added dynamically but never removed. A subgraph’s child flow is always created on the same shard as its parent. The full sharding model is part of getting a deployment production-ready.

Flow Scheduling

Create and Run accept an optional *workflow.FlowOptions argument that sets the flow’s priority, fairness key/weight, time budget, and thread membership (ThreadKey). When more flows are ready than there are workers, the foreman dispatches strictly by priority (lower number first) and shares each priority level fairly across fairness keys so one busy tenant cannot starve the rest. A nil FlowOptions, or any zero field, uses the foreman’s defaults (DefaultPriority config, the tenant actor claim as the fairness key, weight 1). Policy is authored once at genesis (Create/Run); the derived operations take no options — Continue inherits the thread’s policy, Fork inherits the origin’s. The priority and fairness guide covers the concepts and usage in full.

A deferred start is expressed in the workflow itself — an entry task that calls flow.Interrupt, released by Resume — not via a scheduling option.

Configuration

foreman.core:
  SQLDataSourceName: root:root@tcp(127.0.0.1:3306)/
  Workers: 64
  TimeBudget: 2m
  DefaultPriority: 5
  NumShards: 1
  SQLConnectionPool: 8

SQLDataSourceName is the connection string for the backing database. The foreman supports MySQL, PostgreSQL, SQL Server and SQLite.

Workers controls the number of concurrent goroutines that process steps (default 64). Increase this for high-throughput workloads; lower it to limit database connection pressure.

TimeBudget is the hard ceiling on the execution time of any task step, applied as the timeout on the task dispatch call (default 2m). A task endpoint may declare a shorter budget of its own via sub.TimeBudget; the effective deadline is the smaller of the two.

DefaultPriority is the priority assigned to a flow when the caller does not specify one via workflow.FlowOptions (default 5; lower numbers run first).

NumShards is the number of database shards to distribute flows across.

SQLConnectionPool is the number of database connections kept open per shard (default 8). The default is deliberately small: workers spend most of their wall time awaiting downstream task responses, not holding a SQL connection, so a Foreman replica with 64 workers rarely needs more than a handful of connections per shard. Raise this when DB-hold times are longer (remote database, slow queries) or when you have headroom in the database’s max_connections budget. Performance tuning walks through the connection-budget arithmetic.