Production Readiness

Agentic workflows are durable: every step’s input, output, and state delta is persisted to a SQL database by the Foreman. That durability is what lets a flow survive a process crash or a deploy mid-execution — but it also means the Foreman’s database is on the critical path for every running flow, and the Foreman itself is on the critical path for every dispatched step.

Production-grade operation comes down to four cross-cutting properties:

• Scalability — handle growing load without compromising performance.
• Availability — stay ready to serve, with minimal downtime.
• Reliability — consistently produce correct results across failure modes.
• Performance — carry out work at the expected rate under peak load.

The structural backbone covering scalability and availability is three layers:

1. Multiple database shards — distribute flows across more than one database instance so write throughput and storage scale horizontally.
2. Hot/cold replicas per shard — give each shard a standby. The flows on an unreachable shard stop, and listing and purging fail across the whole deployment, even with just one shard.
3. Multiple Foreman replicas — run several Foreman instances against the same shard set so step dispatch survives a worker crash and total worker capacity scales with replica count.

The later sections cover what to monitor (Performance), how the system behaves under failure (Reliability), and the security isolation that protects the Foreman’s trust-elevated credentials.

The diagram above shows the structural backbone for a deployment with three Foreman replicas and two database shards: every Foreman replica reaches every shard (the bus-bar across the top), and each shard has its own hot/cold pair with the cold standby kept in sync from the hot primary at the database layer (the bottom arrow). Failover from hot to cold is handled outside the framework.

Database Sharding

A single database eventually becomes the throughput ceiling for everything: state persistence, step claims, history queries. The Foreman lets you distribute flows across multiple database instances via its Shards config, a list in which every shard is declared explicitly.

Each shard is an independent database — usually on a separate instance, but at minimum a separate logical database with its own write capacity. Migrations run independently on each shard at startup.

foreman.core:
  Shards: '[{"index":1,"dsn":"postgres://foreman:secret@db1.internal:5432/flows"},
            {"index":2,"dsn":"postgres://foreman:secret@db2.internal:5432/flows"},
            {"index":3,"dsn":"postgres://foreman:secret@db3.internal:5432/flows","weight":2},
            {"index":4,"dsn":"postgres://foreman:secret@db4.internal:5432/flows","weight":2}]'

A shard needs only its index and dsn; the other fields are optional:

• index identifies the shard and is encoded into every flow key created on it. Indices must be >= 1 and unique, but need not be contiguous — shards 1 and 99 are fine. Because the index drives routing, the index-to-DSN mapping must be identical on every replica and stable across restarts.
• dsn is the shard’s connection string, with the dialect auto-detected. It is used exactly as given — the engine never formats or rewrites it, so a percent-encoded credential (a password p@ss written p%40ss) survives intact.
• maxConnections is the shard’s connection budget, shared by the replicas that dispatch it. Left at 0, the engine derives it from the measured round trip to the database and the server’s own max_connections, which suits a dedicated database of up to 32 vCPU. Declare the same value on every replica.
• weight is the shard’s share of new flows relative to the other shards, 1 by default.
• cordoned excludes the shard from new-flow placement while everything already resident proceeds normally. This is how a shard is retired, without draining it by hand.

Because each shard names its own database, shards do not have to be uniform. Say the four-shard example above pairs two 16-vCPU instances with two 32-vCPU ones: giving the larger pair a weight of 2 places roughly twice as much new work on each of them.

New flows are assigned to a shard when they’re created. While every shard’s weight is 1, a replica places a new flow on a shard it dispatches, so the flow’s first step runs at once and flows spread by where the Create calls arrive. Any weight other than 1 switches every replica to placement over all non-cordoned shards in proportion to their weights; a flow placed on a shard its creating replica does not dispatch waits for that shard’s next scan, well under a second. Once a flow lands on a shard, it stays there for life — every step, every retry, every interrupt, every history row lives on the same shard as the parent flow. Subgraphs and forked flows inherit their parent’s shard so an entire flow tree is colocated, which keeps cross-shard joins out of the critical path.

Shards can be added but never removed: a flow created on a shard a peer does not know about is unroutable there. Retire a shard by setting "cordoned": true on its entry — resident flows keep executing, and shard-pinned creations (subgraph children, Continue on a thread, forks) still land on it, but no new flow tree starts there.

Changing the shard set is a coordinated restart of the whole fleet, not a live config edit. Shards is applied once during startup and has no config callback, and the engine rejects a re-shard on a running engine outright. Every replica must come back on the same shard list.

Choosing a Database Engine

The Foreman is portable across four SQL engines: PostgreSQL, Microsoft SQL Server, MySQL/MariaDB, and SQLite. All four are first-class targets in the sense that the framework’s test suite passes against each, but they behave very differently under the Foreman’s concurrent INSERT and UPDATE workload. Pick the engine for the shape of your deployment rather than by what you already have running, and budget for the per-engine tuning below.

PostgreSQL — Recommended for Production

PostgreSQL’s MVCC model means concurrent INSERTs never lock each other on secondary indexes, and the default READ COMMITTED isolation level does not take gap locks. The Foreman’s hot path, which is many parallel INSERTs against the same shard’s dwarf_steps table during fan-out, runs without engine-level deadlocks at any concurrency your worker pool can produce. PostgreSQL 13 or newer is required for the partial indexes the schema declares.

PostgreSQL also provides the most efficient indexing for the Foreman’s scheduling reads. The schema declares the indexes that serve pollPendingSteps and the refiller’s candidate scan as partial indexes filtered to status IN ('pending', 'running'), so the working set in each index stays bounded to active steps regardless of how many completed or failed steps accumulate in the underlying table. Index lookups stay fast over the long term without depending on aggressive retention.

No special tuning is needed for correctness. For headroom and steady operation:

• max_connections at least twice the shard’s connection budget, plus two for every Foreman replica beyond the shard’s two dispatchers. The engine takes no more than half the server’s limit, so a stock limit of 100 caps a shard’s budget at 50.
• shared_buffers at roughly 25% of the database host’s RAM.
• autovacuum_work_mem = 1GB on PostgreSQL 16 and older. The step table updates and deletes rows continuously, and each time vacuum’s memory fills with dead rows it rescans every index of the table before it can go on. At the 64 MB many managed services default to, a long pass falls behind the churn. It is a server setting: postgresql.conf, ALTER SYSTEM, or a database flag on a managed service. PostgreSQL 17 and later need much less.
• No statement_timeout, transaction_timeout or idle_in_transaction_session_timeout on the shard’s database, the Foreman’s role or its DSN, even where general guidance recommends them. Some of the engine’s background queries grow with the backlog, so no value is safe for every backlog, and a timeout that cuts them off can stop a shard’s work from being dispatched at all. An idle-in-transaction timeout ends transactions when the engine is busiest.
• Keep long-running queries off the shard databases. While an analytics or ad hoc query runs for minutes, PostgreSQL cannot clean up the step table behind it, and dispatch slows until it ends. A read replica with hot_standby_feedback on holds cleanup back the same way.

The Foreman’s database sessions run in UTC unless the DSN sets a timezone parameter.

Microsoft SQL Server

SQL Server requires one database-level change for the Foreman to behave well: enable Read Committed Snapshot Isolation on each shard database.

ALTER DATABASE <shard_database> SET READ_COMMITTED_SNAPSHOT ON;

This gives SQL Server PostgreSQL-style non-blocking reads and eliminates the pessimistic locking that would otherwise behave similarly to MySQL’s REPEATABLE READ. Without it, expect 1205 deadlocks under sustained parallel flow creation. Beyond enabling RCSI on every shard, no other server-level tuning is mandatory.

SQL Server’s filtered indexes give the foreman the same scheduling-index containment benefit as PostgreSQL partial indexes: the selection and saturation indexes on dwarf_steps only carry entries for non-terminal steps, so their size and lookup cost stay bounded to active work rather than growing with the total number of steps ever recorded.

MySQL and MariaDB

InnoDB at its default REPEATABLE READ isolation level takes next-key locks (row plus gap) on every secondary-index touch. Two flows being created on the same shard at the same time can lock overlapping ranges of the step table’s selection or saturation indexes in different orders, and InnoDB aborts the loser with Error 1213 (40001) Deadlock found when trying to get lock. This is correct engine behavior, not a Foreman bug. The Foreman’s flow-creation path retries on lock contention so an individual Error 1213 is invisible to callers, but a high sustained deadlock rate caps throughput and inflates p99 latency.

Operators running MySQL or MariaDB should set the following in the server’s [mysqld] section before putting any real load on the deployment:

Add interpolateParams=true to each shard’s DSN, e.g. foreman:secret@tcp(db1.internal:3306)/flows?interpolateParams=true. Without it the driver sends every statement that has arguments as a prepare and then an execute, which roughly doubles the database round trips each step makes. The driver refuses it together with the big5, gbk, sjis and cp932 collations, so use a utf8mb4 connection collation.

MariaDB 10.6 or newer is recommended. 10.5 is the minimum version that handles the schema’s JSON columns correctly; older versions accept the DDL but ship with less mature deadlock detection.

Unlike PostgreSQL and SQL Server, neither MySQL nor MariaDB supports partial (or filtered) indexes, so the foreman schema falls back to full composite indexes on dwarf_steps for both of those engines. The indexes that drive scheduling carry an entry for every step, terminal or not, and grow proportionally to the total number of steps ever recorded rather than to the size of the active working set. Operators running MySQL or MariaDB should plan for more aggressive flow retention than they would on the other engines, or accept that scheduling lookup cost and index maintenance work will increase over time. The Operational Notes section below discusses the retention surface the framework exposes.

For multi-shard deployments, every database named in a shard’s dsn must exist before the Foreman starts. The framework runs schema migrations on each but does not issue CREATE DATABASE itself.

SQLite — Testing and Development Only

SQLite uses a single database-level writer lock, which makes engine-level deadlocks structurally impossible but caps write throughput at roughly one transaction at a time. The framework reaches for SQLite on both non-deployed paths, and Shards is not consulted on either: under TESTING the engine opens isolated in-memory databases keyed by the Microbus plane, so every replica in a test app resolves to the same throwaway set; under LOCAL a Foreman that declares no shards falls back to a single file:shard_1.local.sqlite file shard. The connection layer injects a one-second busy timeout so the worker pool does not immediately fail on SQLITE_BUSY during fan-out. SQLite is intentionally not a production target; the single-writer ceiling will be the bottleneck before anything else in the stack.

Sizing the Shard Count

Engine choice directly affects how many shards you need to hit a given target throughput. The numbers below are rough sizing starting points and should be calibrated against your own workload using the signals listed in What to Monitor.

The wide MySQL/MariaDB range reflects how much the isolation-level change matters. The same hardware delivers roughly an order of magnitude more sustained flow creation rate at READ-COMMITTED than at the default REPEATABLE READ, simply because most of the otherwise-deadlocking INSERTs commit on the first try.

Connection Pools

Each shard has a connection budget: the connections that the replicas dispatching it may hold there, together. The engine derives it at startup from two things it measures:

• The round trip to the database. The budget is 250 connections up to 0.25 ms, rising evenly to 500 at 1 ms and staying there beyond. A connection held while a packet is on the wire does the database no good, so a more distant database needs more of them to stay as busy.
• The server’s own max_connections. The budget never exceeds half of it, leaving the rest for other clients, administration and the server’s reserved slots. Raise the server’s limit and the budget follows.

On SQLite the budget is a fixed 8, since SQLite writes one transaction at a time. Connections open only as work needs them and close after two minutes idle, so a lightly loaded shard holds far fewer than its budget.

The fleet agrees, per shard, on two replicas that dispatch it: they split its budget between them and run its steps. Every other replica holds one or two connections to the shard, enough to create flows, answer reads and await outcomes. A shard’s connections therefore stay near its budget plus a connection or two per extra replica however large the fleet grows, and different shards pick different dispatchers, so the work spreads across the fleet. The split resizes itself as replicas join and leave.

The derived budget suits a dedicated database of 4 to 32 vCPU with no pooler in front. Performance Tuning below covers when to set maxConnections instead.

Hot/Cold Replicas per Shard

Each shard’s availability is its database’s. When a shard’s database is unreachable, the flows on it stop until it returns, Create calls placed on it fail, and List and Purge, which span every shard, fail as a whole. Flows on the healthy shards keep running, but sharding multiplies capacity, not availability: with four shards there are four databases whose loss is an outage for someone. Hot/cold per shard is therefore mandatory at any shard count.

The pattern is the same regardless of shard count: each shard gets its own database standby with failover handled at the database layer.

Microbus does not manage database failover itself. The Foreman holds one connection string per shard and treats whatever that string resolves to as the authoritative endpoint for that shard. The operator-side pattern is:

• Each shard is provisioned as a primary plus one or more standbys, with replication configured in the database (MySQL semi-sync, PostgreSQL streaming replication, SQL Server Always On, etc.).
• The DSN points at a stable endpoint that routes to the current primary — typically a managed-database failover endpoint, an HAProxy / ProxySQL in front of the cluster, or a DNS name that’s repointed by your failover tooling.
• When the primary fails, the endpoint flips to the new primary. The Foreman’s connection pool reconnects and resumes.

This pattern is intentionally outside the framework: every cloud and on-prem environment has its own preferred way to do database HA, and the Foreman’s role is only to keep one DSN per shard and reconnect cleanly when the connection drops.

A reasonable production posture is N declared shards with a primary plus a synchronous standby each, so a single database failure costs reconnect latency rather than a full Foreman outage. Because the DSN is used verbatim, the failover endpoint goes straight into the shard’s dsn field.

Managed Database on Cloud Providers

The framework’s “shard = independent database instance” model maps directly onto cloud-managed Postgres clusters. Each Foreman shard’s DSN points at one managed cluster, and the cloud provider handles per-shard HA, backups, failover, patching, and monitoring that would otherwise be operator work. The operational cost of N Foreman shards stays mostly proportional to N (one IaC definition per shard), rather than scaling with operator headcount.

All three major clouds offer the same broad tiers of managed Postgres, with meaningful gaps at the upper tiers:

The three tiers represent meaningfully different architectures, not just different price points:

• Basic managed Postgres is plain Postgres with the cloud handling HA. One primary, one or more standbys, failover in tens of seconds to a couple of minutes. Same per-primary ceilings as self-hosted Postgres, with the operator burden lifted.
• High-throughput rewrite (Aurora and AlloyDB) keeps the single-writer architecture but replaces the storage layer with a distributed one, offloading checkpoint and WAL work. Roughly 2 to 4 times the write throughput of basic managed Postgres on equivalent compute. The single-primary considerations from Performance Tuning (WAL throughput, autovacuum lag, the connection budget) all still apply per cluster, just with more headroom.
• Horizontally sharded Postgres (Aurora Limitless, Cosmos DB for PostgreSQL) has multiple writer instances internally, with the managed service partitioning tables across them. With this tier, the Foreman can declare a single shard and rely on the cloud to handle write distribution behind one DSN.

Read replicas at the basic or rewrite tiers (RDS read replicas, Aurora reader instances, AlloyDB read pools) do not help the Foreman directly. The refiller’s candidate scans and step claims need read-after-write consistency, so they go to the primary regardless of how many readers are configured.

Configuration Shape

Each managed cluster is one entry in Shards, carrying that cluster’s own endpoint:

foreman.core:
  Shards: '[{"index":1,"dsn":"postgres://user:pw@shard-1.example-host:5432/flows"},
            {"index":2,"dsn":"postgres://user:pw@shard-2.example-host:5432/flows"},
            {"index":3,"dsn":"postgres://user:pw@shard-3.example-host:5432/flows"},
            {"index":4,"dsn":"postgres://user:pw@shard-4.example-host:5432/flows"}]'

The exact endpoint format differs per cloud (*.rds.amazonaws.com, *.cloudsql.googleapis.com, *.postgres.database.azure.com), but the shape is the same. A four-shard deployment is, operationally, four managed clusters defined in IaC. Resizing a cluster within 4 to 32 vCPU needs no change to its entry. Past 32 vCPU, set its maxConnections (a 64-vCPU database gains throughput up to about 500), and in a fleet of unequal clusters give the larger ones a larger weight. Managed services also size max_connections by instance class, and the derived budget never takes more than half of it.

Per-Cloud Specifics

AWS. RDS Multi-AZ is the default per shard, with the newer Multi-AZ DB Cluster variant offering sub-35-second failover when that matters. Aurora PostgreSQL lifts the per-shard ceiling 2-3x further when needed. Aurora Limitless Database (GA late 2024) is AWS’s horizontally-sharded offering, but is a relatively new service: plain Aurora has many more production references at any given workload shape.

GCP. Cloud SQL for PostgreSQL with regional HA is the default per shard. AlloyDB for PostgreSQL is Google’s high-throughput rewrite (the Aurora equivalent), GA since 2022 and well past the early-adopter phase. GCP does not have a direct horizontally-sharded equivalent: Spanner with the PostgreSQL interface exists, but Spanner is architecturally a different database (Paxos-replicated global transactions, much higher base latency and cost) and is not a drop-in. Deployments on GCP scale by adding AlloyDB clusters as Foreman shards.

Azure. Azure Database for PostgreSQL Flexible Server with zone-redundant HA is the default per shard. Azure does not have a high-throughput-rewrite tier equivalent to Aurora or AlloyDB; deployments stay on Flexible Server until they need horizontal sharding. However, Azure Cosmos DB for PostgreSQL is managed Citus (Microsoft acquired Citus in 2019), and Citus has been production-tested for horizontal sharding since 2016, so Azure’s sharded tier is the most mature of the three clouds. On Azure, Cosmos DB for PostgreSQL is a viable “skip app-level sharding” option today with relatively low risk.

Choosing the Configuration Shape

Rough guidance, calibrate to your own benchmarks:

The choice is rarely about pure cost: it is about how many operational units (clusters, runbooks, dashboards) your team can comfortably operate. Managed offerings compress that count substantially compared to self-hosted Postgres, but only the horizontally-sharded tier eliminates the Foreman’s sharding model.

Multiple Foreman Replicas

The Foreman is a normal Microbus microservice and runs with as many Foreman replicas as you want. Every Foreman replica connects to every shard — the shard set is a property of the Foreman service, not of an individual Foreman replica — and any replica can create flows on, read from and await flows on any shard. Per shard, the fleet picks two replicas to dispatch its steps, different ones for different shards, so the work spreads across the fleet. Steps are leased at claim time so only one worker across all Foreman replicas executes a given step, and an expired lease is recovered automatically if a Foreman replica crashes mid-step. If a dispatcher goes away, another replica takes its place. No Foreman replica is a single point of failure for dispatch.

Plan for two Foreman replicas per shard, each able to carry its shard alone, so that every shard has its own pair of dispatchers. Replicas beyond that add availability and capacity for creating and reading flows, but a shard’s dispatch capacity is bounded by its database and its connection budget, not by the replica count. Scaling out is just adding Foreman replicas — no shard rebalancing, no coordinator to configure, no client-side changes. The fleet agrees on dispatchers through the shard databases it already shares, and the Foreman replicas are interchangeable.

# Same config on every Foreman replica.
foreman.core:
  Shards: '[{"index":1,"dsn":"postgres://foreman:secret@db1.internal:5432/flows"},
            {"index":2,"dsn":"postgres://foreman:secret@db2.internal:5432/flows"},
            {"index":3,"dsn":"postgres://foreman:secret@db3.internal:5432/flows","weight":2},
            {"index":4,"dsn":"postgres://foreman:secret@db4.internal:5432/flows","weight":2}]'

The shard list must be byte-for-byte agreed on across the fleet: a flow created on a shard a peer does not know about is unroutable there.

A note on database connections: each Foreman process opens one connection pool per shard, and workers reuse those connections rather than each holding one. Because a shard’s budget is split between its two dispatchers and every other replica holds only a connection or two there, total load on a shard is bounded by the shard’s own budget rather than growing with the fleet. Performance Tuning below covers the symptoms that indicate the derived budget needs overriding.

The framework’s scheduling rules — Priority and Fairness — still apply across replicas, but they work from periodic snapshots of what is due rather than from a single global queue. Each dispatcher serves its own share of the due work first, so for arrivals within a scheduling cycle or two of each other a lower-priority step can start on one replica while a higher-priority one waits briefly for another. That’s an accepted trade-off for the horizontal scaling.

Performance Tuning

Once the structural backbone is in place, throughput is shaped by a small number of configurable levers and the relationships among them. Most production deployments leave most of these at defaults. Knowing which symptom indicates which lever is the difference between profiling a real bottleneck and over-provisioning blind.

The Levers

The Connection Budget

The budget belongs to the shard, not to the replica. The engine derives it — 250 connections up to a 0.25 ms round trip, rising to 500 at 1 ms, never more than half the server’s max_connections — and splits it between the shard’s two dispatchers. So a shard holds about the same number of connections whether two replicas or twenty are running, and the number to check against the database’s max_connections is the shard’s budget plus a connection or two for each other replica. It does not move when the fleet scales.

Under-sizing a budget costs throughput; it does not destabilize the database. Past the budget, throughput keeps climbing slowly as connections are added, so a larger pool is diminishing returns rather than a cliff. Set maxConnections on a shard’s entry when the derivation cannot see the real constraint:

• The database is larger than 32 vCPU. The derived budget stops growing with the server there; a 64-vCPU database gains throughput up to about 500.
• The database is shared with other heavy workloads, and the Foreman should take less of it.
• A connection pooler such as PgBouncer sits in front, so the server’s reported limit is not the real one.
• Several shards are databases on one server. Each would take half the server’s limit without knowing about the others, so split the server between them explicitly.

Declare the same maxConnections for a shard on every replica, so every dispatcher takes the same share of it. Compensation for distance stops at 1 ms by design: past that, the answer is to place the Foreman closer to its databases, or to add shards. The engine warns in the log when a fleet’s connections to a shard approach the server’s max_connections.

Symptom-to-Lever Map

Read this table alongside the metrics in What to Monitor.

Why the Defaults Are What They Are

Workers and each shard’s maxConnections default to deriving because both are functions of facts the engine can measure — the round trip to each database and the server’s own connection limit — and a hand-picked constant goes stale the moment the hardware or the workload moves. Both remain settable as overrides, and neither should be the first thing reached for.

One value of Workers is not a derivation hint but a distinct shape: Workers: 0 stands up a replica that creates flows, awaits them, and serves reads, but never executes a task. That is why the “derive” sentinel is -1 rather than 0 — the config has to stay able to express a dispatch-free replica.

Each worker is a goroutine and a socket, both cheap, and a worker blocked on a slow or parked dispatch is just an idle goroutine, not a held thread. Backing off an overloaded or rate-limited downstream is the task’s own job via flow.Retry, which parks the step rather than tying up a worker, so the derived ceiling can be generous without the engine needing to throttle it.

A single shard is the right starting point for single-database deployments. Declaring extra shards ahead of demand is cheap and saves a future fleet restart, but starting at one avoids coordination a single-shard deployment does not need.

TimeBudget: 2m matches a typical HTTP dispatch ceiling, comfortably longer than most tasks but short enough that a wedged task replica is recovered without hours of stall. Tasks that genuinely need longer should declare it on the endpoint via sub.TimeBudget rather than raising the global ceiling.

MaxPriority: 5 gives five bands, and a flow that sets no priority runs at the midpoint, 3, so callers can mark work as more urgent (1, 2) or less (4, 5) without touching the default. Priority is strict over capacity, not absolute: the scheduler fills the best due band first and gives the capacity it leaves unused to the next, so a light stream of urgent flows never idles workers while a backlog waits below it, but a band that keeps every worker busy holds the bands below it back entirely. If multiple workload classes need to coexist, set each class’s priority explicitly at Create.

Isolate Foreman Replicas

The Foreman mints actor access tokens before dispatching each step so the task runs under the original caller’s identity. That mint happens through the access-token service on the trust-root tier (:666), which means the Foreman holds a publish grant to :666 — a privilege that only a small, named set of microservices in any deployment ever has. Co-bundling the Foreman with ordinary application services would extend that privilege to every sibling in the bundle’s address space: a compromise of any of them gives the attacker the Foreman’s in-memory credentials and, transitively, the ability to call the trust-root tier.

Run Foreman replicas as their own application bundle, separate from the bundles that hold your business microservices. This is the same isolation rule that already applies to the trust-root services themselves, one tier removed — the Foreman is a trust-elevated caller and should be treated accordingly. Other :666 callers in the deployment (for example a control-plane microservice that orchestrates token issuance) belong in their own bundle for the same reason.

The cost is modest: an extra deployment artifact and an extra process per Foreman replica. The benefit is that a compromised business microservice has no in-process path to the Foreman’s privileged credentials, and the blast radius of any single bundle compromise stays bounded by the broker’s ACLs instead of by process memory.

What to Monitor

Operating in production means knowing which dial to turn before users notice. The Foreman exposes a ShardInfo endpoint that reports each shard’s round-trip latency and any error, and keeps answering when a shard is down. The scheduling metrics come from the embedded engine and carry a dwarf_ prefix; the Foreman contributes one microbus_-prefixed series of its own.

Most engine gauges are per replica and sum across the fleet. Three describe a shard instead, and only the replicas responsible for that shard report them: dwarf_steps_dispatch_lag_seconds comes from each replica dispatching the shard, and dwarf_steps_pending and dwarf_steps_oldest_pending_age_seconds come from one leading replica per shard, refreshed every 30 seconds. More than one replica can report the same shard, so take the max per shard first, then sum across shards where a total makes sense — summing across replicas overstates the backlog. For example, sum by (priority) (max by (shard, priority) (dwarf_steps_pending)) is the fleet’s pending steps per band.

• dwarf_steps_dispatch_lag_seconds (per shard, aggregate with max) — how long the most overdue step in the priority bands being served has been due. Under a second while dispatch keeps up, so this is the backlog signal to alert on. A band that better-priority work leaves no capacity for is excluded, so when spare capacity reaches a band that has waited a long time the lag jumps to its age; read it beside the per-band ages.
• dwarf_steps_pending (per shard and priority, aggregate with max per shard and priority, then sum) — due pending steps in each band. A rising count with a flat lag is just more work arriving.
• dwarf_steps_oldest_pending_age_seconds (per shard and priority, aggregate with max) — how long the longest-due step in each band has been due. A lower band whose age climbs while the dispatch lag stays low is being held back by better-priority work, as designed, not by a slow engine.
• dwarf_steps_queue_depth — current size of each replica’s in-memory candidate cache. Plot it on the same panel as the pending series: when pending stays high but queue depth stays at zero, candidate selection is the bottleneck (database shard latency, not worker capacity).
• dwarf_operation_seconds (histogram, by op and shard) — the time a Foreman API operation such as create or list spent on a shard, its wait for a turn plus its database work. Compare one op across shards to find a slow database; a shard that refuses connections fails fast and reads as fast, which is what ShardInfo is for.
• dwarf_shard_dispatching (per replica, per shard) — 1 where this replica dispatches the shard. Summed per shard it should read 2 once there are two replicas; a shard summing to 0 runs nothing.
• dwarf_peer_replicas (per replica, per shard) — how many replicas this one currently sees holding connections to that shard, from which the fleet picks the shard’s dispatchers. Replicas should agree, so a spread across the fleet is itself the signal. It is deliberately slow to fall, so a drop lags a real departure by a reading or two.
• dwarf_peer_blind_seconds (per replica, per shard) — how long since that shard’s peer registry was last read successfully; zero on a healthy replica. Past two read cadences the replica is blind on that shard and holds its last known view of the fleet. Check it first whenever a shard’s counts look frozen.
• dwarf_state_in_flight_bytes, over dwarf_state_in_flight_steps — flow state this replica is holding across in-flight task dispatches. The ratio is the mean state a task carries: a large mean says state size is what loads the replica, a small mean against a large count says fan-out width is. This is the series that catches a workflow carrying large blobs forward, so pair it with the housekeeping guidance in State.
• dwarf_state_cache_bytes (per replica) — memory the engine’s state cache holds, bounded at a tenth of the process’s memory limit. Pinned at that size means the cache is full; set GOMEMLIMIT on Foreman hosts so the bound matches the memory the process may really use.
• dwarf_flows_started_total and dwarf_flows_ended_total (by workflow, shard and priority; the second also by terminal status) — the difference of their increases over a window, per priority, is how many flows each band gained or shed in flight. Read it as a trend: each replica’s counters restart from zero with the replica.
• microbus_foreman_timeout_requests_total (labelled by task_url and outcome) — 404 ack-timeout dispatches, i.e. a task whose hosting microservice did not answer. outcome="retry" counts re-probes during a brief absence; outcome="giveup" counts steps failed because the microservice stayed absent past the step’s time budget — alert on this series. These dispatches are the engine’s one engine-level retry for missing microservices.
• dwarf_steps_executed_total{status="retried"} (labelled by task_name and shard) — retry-dispatch churn: every re-dispatch a flow.Retry armed, whether task-owned backoff or an ack-timeout re-probe. A sustained climb means something is backing off repeatedly; pair it with the timeout series to tell a missing microservice from a task riding out a rate limit.
• dwarf_steps_recovered_total — steps recovered because their worker was lost. A steady trickle usually means tasks are overrunning their time budget, or that the platform’s termination grace period is cutting a replica’s drain short.

Three engine counters are zero in a healthy deployment and should page someone when they are not: dwarf_steps_unwedged_total, dwarf_flows_orphaned_total and dwarf_steps_write_failed_total.

The bundled Workflow Overview Grafana dashboard charts these series with the aggregation each needs: flows started and ended, missing-microservice retries and give-ups, pending steps and oldest pending age by priority, dispatch lag and API operation latency by shard, replicas and dispatching replicas per shard, worker and database pressure, candidate selection, in-flight state and state cache memory. Wire it in alongside Foreman-replica CPU and database connection counts. The combination tells you whether your bottleneck is dispatch capacity, database write throughput, the task endpoints themselves, or a single hot tenant.

Failure Modes

A quick reference for what each failure looks like in production:

Putting It Together

A typical production deployment combines all the layers:

• N database shards sized for your write throughput and storage targets, each declared in Shards with its own DSN, and a weight to match its size when the shards are unequal.
• Each shard backed by a primary + standby with failover routed through the DSN endpoint. This is non-optional at any shard count — the flows on an unreachable shard stop until it returns.
• Two Foreman replicas per shard, every one of them carrying an identical Shards list and MaxPriority, so every shard has two dispatchers and the fleet survives any single Foreman-replica restart. Set GOMEMLIMIT on each.
• Foreman replicas isolated in their own application bundle, separate from business microservices, so the Foreman’s :666 privilege does not leak through process memory.

A small but resilient starting point might be two declared shards, two databases each with a standby, and four Foreman replicas. From there you scale shards for throughput, keeping two Foreman replicas per shard.

Operational Notes

• Engine upgrades are a maintenance window, not a rolling deploy. Schema migrations are forward-only and run at startup, so the first replica on a new release migrates the databases that every still-running old replica is reading, and there is no downgrade path. Back up every shard, drain the fleet, bring up a single replica and confirm it comes up clean, then start the rest. Flows survive it untouched: pending steps stay pending and interrupted flows stay parked. Allow extra time on that first restart — it migrates every shard before it serves.
• Deploy every microservice and every Foreman replica on the same Microbus release. Each task microservice decodes and re-encodes the flow it is handed, and replicas of different releases can disagree on which of them dispatch a shard, so task microservices and Foreman replicas of different releases must not run side by side.
• Allow a drain window longer than the largest time budget. A Foreman replica that shuts down waits for the tasks it has in flight. If the platform kills it first (on Kubernetes, past terminationGracePeriodSeconds), those steps are recovered on another replica and their tasks run again.
• Flow lifecycle is operator-managed. The Foreman does not auto-purge flows — every row remains potentially resurrectable (Resume an interrupted flow, Continue into a completed thread, Fork a failed/terminated/cancelled flow), and no single retention policy fits both short batch jobs and long human-approval workflows. Plan archival, partitioning, or manual cleanup at the database layer, per shard. Retention discipline matters more on MySQL/MariaDB than on PostgreSQL or SQL Server because those two engines confine the scheduling indexes to non-terminal steps via partial/filtered indexes, while MySQL and MariaDB index every row regardless of status — a difference the engine choice turns on.
• History queries hit the shard the flow lives on. Snapshot, History, and HistoryMermaid look up the flow’s shard by the shardNum-flowID-flowToken composite key embedded in the flow key, so a query for a specific flow only touches one shard. List enumerates across all shards and merges in reverse chronological order; the shards are queried in parallel so latency stays roughly flat, but aggregate database load grows linearly with shard count.
• You can pre-provision empty shards. Declaring four shards on day one when you only need two-shard throughput is fine — the extra ones open empty and start receiving flows immediately. Since any change to Shards costs a coordinated fleet restart, provisioning ahead of demand is how you avoid paying for one later.
• Subgraphs and forks share the parent’s shard. Cross-shard joins never appear in normal operation because every flow tree is shard-local by construction.

Further Reading

• Foreman — full configuration reference, including Workers, TimeBudget and MaxPriority.
• Priority and Fairness — per-flow scheduling controls that work alongside this scaling model.
• Application Bundling — how to package microservices into the bundles this page recommends.
• Deploy to Production — broader production-deployment checklist for a Microbus solution, including NATS HA and other deployment-wide concerns.