Production Readiness

Agentic workflows are durable: every step’s input, output, and state delta is persisted to a SQL database by the Foreman. That durability is what lets a flow survive a process crash or a deploy mid-execution — but it also means the Foreman’s database is on the critical path for every running flow, and the Foreman itself is on the critical path for every dispatched step.

Production-grade operation comes down to four cross-cutting properties:

• Scalability — handle growing load without compromising performance.
• Availability — stay ready to serve, with minimal downtime.
• Reliability — consistently produce correct results across failure modes.
• Performance — carry out work at the expected rate under peak load.

The structural backbone covering scalability and availability is three layers:

1. Multiple database shards — distribute flows across more than one database instance so write throughput and storage scale horizontally.
2. Hot/cold replicas per shard — give each shard a standby. Any unreachable shard halts Foreman scheduling across the whole deployment, even with just one shard.
3. Multiple Foreman replicas — run several Foreman instances against the same shard set so step dispatch survives a worker crash and total worker capacity scales with replica count.

The later sections cover what to monitor (Performance), how the system behaves under failure (Reliability), and the security isolation that protects the Foreman’s trust-elevated credentials.

The diagram above shows the structural backbone for a deployment with three Foreman replicas and two database shards: every Foreman replica reaches every shard (the bus-bar across the top), and each shard has its own hot/cold pair with the cold standby kept in sync from the hot primary at the database layer (the bottom arrow). Failover from hot to cold is handled outside the framework.

Database Sharding

A single database eventually becomes the throughput ceiling for everything: state persistence, step claims, history queries. The Foreman lets you distribute flows across multiple database instances via its NumShards config.

When NumShards > 1, the SQLDataSourceName connection string must contain a %d placeholder that the Foreman fills with the shard index (1-based). Each shard is an independent database — usually on a separate instance, but at minimum a separate logical database with its own write capacity. Migrations run independently on each shard at startup.

foreman.core:
  SQLDataSourceName: foreman:secret@tcp(db-shard-%d.internal:3306)/flows
  NumShards: 4

That config opens four shards: db-shard-1.internal, db-shard-2.internal, db-shard-3.internal, db-shard-4.internal. The placeholder is plain fmt.Sprintf formatting, so it works the same way for database names (/flows_%d) or any other position in the DSN where the shard index makes sense for your deployment.

New flows are assigned to a shard at random when they’re created. Once a flow lands on a shard, it stays there for life — every step, every retry, every interrupt, every history row lives on the same shard as the parent flow. Subgraphs and forked flows inherit their parent’s shard so an entire flow tree is colocated, which keeps cross-shard joins out of the critical path.

Shards can be added by raising NumShards and rolling the Foreman; the new shards open and migrate at startup and immediately become candidates for new flows. Shards cannot be removed — reducing NumShards is rejected at startup because in-flight flows on the dropped shards would be orphaned. Plan capacity assuming the shard count only grows.

Choosing a Database Engine

The Foreman is portable across four SQL engines: PostgreSQL, Microsoft SQL Server, MySQL/MariaDB, and SQLite. All four are first-class targets in the sense that the framework’s test suite passes against each, but they behave very differently under the Foreman’s concurrent INSERT and UPDATE workload. Pick the engine for the shape of your deployment rather than by what you already have running, and budget for the per-engine tuning below.

PostgreSQL — recommended for production

PostgreSQL’s MVCC model means concurrent INSERTs never lock each other on secondary indexes, and the default READ COMMITTED isolation level does not take gap locks. The Foreman’s hot path, which is many parallel INSERTs against the same shard’s microbus_steps table during fan-out, runs without engine-level deadlocks at any concurrency your worker pool can produce. PostgreSQL 13 or newer is required for the partial indexes the schema declares.

PostgreSQL also provides the most efficient indexing for the Foreman’s scheduling reads. The schema declares the indexes that serve pollPendingSteps and the refiller’s candidate scan as partial indexes filtered to status IN ('pending', 'running'), so the working set in each index stays bounded to active steps regardless of how many completed or failed steps accumulate in the underlying table. Index lookups stay fast over the long term without depending on aggressive retention.

No special tuning is needed for correctness. For headroom, size the cluster’s max_connections to comfortably accommodate NumShards connection pools per Foreman replica, multiplied by the number of replicas, and raise shared_buffers to roughly 25% of the database host’s RAM.

Microsoft SQL Server

SQL Server requires one database-level change for the Foreman to behave well: enable Read Committed Snapshot Isolation on each shard database.

ALTER DATABASE microbus_N SET READ_COMMITTED_SNAPSHOT ON;

This gives SQL Server PostgreSQL-style non-blocking reads and eliminates the pessimistic locking that would otherwise behave similarly to MySQL’s REPEATABLE READ. Without it, expect 1205 deadlocks under sustained parallel flow creation. Beyond enabling RCSI on every shard, no other server-level tuning is mandatory.

SQL Server’s filtered indexes give the foreman the same scheduling-index containment benefit as PostgreSQL partial indexes: the selection and saturation indexes on microbus_steps only carry entries for non-terminal steps, so their size and lookup cost stay bounded to active work rather than growing with the total number of steps ever recorded.

MySQL and MariaDB

InnoDB at its default REPEATABLE READ isolation level takes next-key locks (row plus gap) on every secondary-index touch. Two flows being created on the same shard at the same time can lock overlapping ranges of the step table’s selection or saturation indexes in different orders, and InnoDB aborts the loser with Error 1213 (40001) Deadlock found when trying to get lock. This is correct engine behavior, not a Foreman bug. The Foreman’s flow-creation path retries on lock contention so an individual Error 1213 is invisible to callers, but a high sustained deadlock rate caps throughput and inflates p99 latency.

Operators running MySQL or MariaDB should set the following in the server’s [mysqld] section before putting any real load on the deployment:

MariaDB 10.6 or newer is recommended. 10.5 is the minimum version that handles the schema’s JSON columns correctly; older versions accept the DDL but ship with less mature deadlock detection.

Unlike PostgreSQL and SQL Server, neither MySQL nor MariaDB supports partial (or filtered) indexes, so the foreman schema falls back to full composite indexes on microbus_steps for both of those engines. The indexes that drive scheduling carry an entry for every step, terminal or not, and grow proportionally to the total number of steps ever recorded rather than to the size of the active working set. Operators running MySQL or MariaDB should plan for more aggressive flow retention than they would on the other engines, or accept that scheduling lookup cost and index maintenance work will increase over time. The Operational Notes section below discusses the retention surface the framework exposes.

For multi-shard deployments, the per-shard databases (microbus_1 through microbus_N) must exist before the Foreman starts. The framework runs schema migrations on each but does not issue CREATE DATABASE itself.

SQLite — testing and development only

SQLite uses a single database-level writer lock, which makes engine-level deadlocks structurally impossible but caps write throughput at roughly one transaction at a time. The framework selects SQLite automatically when Deployment == TESTING and SQLDataSourceName is empty: each shard becomes a separate in-memory database, and the connection layer injects a one-second busy timeout so the worker pool does not immediately fail on SQLITE_BUSY during fan-out. SQLite is intentionally not a production target; the single-writer ceiling will be the bottleneck before anything else in the stack.

Sizing the shard count

Engine choice directly affects how many shards you need to hit a given target throughput. The numbers below are rough sizing starting points and should be calibrated against your own workload using the signals listed in What to Monitor.

The wide MySQL/MariaDB range reflects how much the isolation-level change matters. The same hardware delivers roughly an order of magnitude more sustained flow creation rate at READ-COMMITTED than at the default REPEATABLE READ, simply because most of the otherwise-deadlocking INSERTs commit on the first try.

Connection pools

Each Foreman replica opens a connection pool per shard, sized by the SQLConnectionPool config (default 8 connections per shard). Defaults usually work because workers spend most of their time awaiting downstream task responses, not the database, so a single replica with 64 workers and four shards typically holds only a handful of connections to each shard at steady state. The Performance Tuning section below covers when to raise SQLConnectionPool, the symptoms that indicate the current pool is too small, and the connection-budget arithmetic that bounds how high you can take it without overwhelming the database.

Hot/Cold Replicas per Shard

The Foreman’s database is a single point of failure for the entire deployment at every shard count. Scheduling, recovery polling, and retry re-dispatch fan out across every shard on each cycle, and any shard that errors fails the whole cycle. With one shard, losing it halts everything; with four shards, losing any one of them also halts everything — sharding doesn’t degrade you gracefully to 75% capacity, it multiplies the failure surface. Hot/cold per shard is therefore mandatory at any shard count.

The pattern is the same regardless of NumShards: each shard gets its own database standby with failover handled at the database layer.

Microbus does not manage database failover itself. The Foreman holds one connection string per shard and treats whatever that string resolves to as the authoritative endpoint for that shard. The operator-side pattern is:

• Each shard is provisioned as a primary plus one or more standbys, with replication configured in the database (MySQL semi-sync, PostgreSQL streaming replication, SQL Server Always On, etc.).
• The DSN points at a stable endpoint that routes to the current primary — typically a managed-database failover endpoint, an HAProxy / ProxySQL in front of the cluster, or a DNS name that’s repointed by your failover tooling.
• When the primary fails, the endpoint flips to the new primary. The Foreman’s connection pool reconnects and resumes.

This pattern is intentionally outside the framework: every cloud and on-prem environment has its own preferred way to do database HA, and the Foreman’s role is only to keep one DSN per shard and reconnect cleanly when the connection drops.

A reasonable production posture is NumShards: N with a primary plus a synchronous standby per shard, so a single database failure costs reconnect latency rather than a full Foreman outage.

Managed Database on Cloud Providers

The framework’s “shard = independent database instance” model maps directly onto cloud-managed Postgres clusters. Each Foreman shard’s DSN points at one managed cluster, and the cloud provider handles per-shard HA, backups, failover, patching, and monitoring that would otherwise be operator work. The operational cost of N Foreman shards stays mostly proportional to N (one IaC definition per shard), rather than scaling with operator headcount.

All three major clouds offer the same broad tiers of managed Postgres, with meaningful gaps at the upper tiers:

The three tiers represent meaningfully different architectures, not just different price points:

• Basic managed Postgres is plain Postgres with the cloud handling HA. One primary, one or more standbys, failover in tens of seconds to a couple of minutes. Same per-primary ceilings as self-hosted Postgres, with the operator burden lifted.
• High-throughput rewrite (Aurora and AlloyDB) keeps the single-writer architecture but replaces the storage layer with a distributed one, offloading checkpoint and WAL work. Roughly 2 to 4 times the write throughput of basic managed Postgres on equivalent compute. The single-primary considerations from Performance Tuning (WAL throughput, autovacuum lag, the connection budget) all still apply per cluster, just with more headroom.
• Horizontally sharded Postgres (Aurora Limitless, Cosmos DB for PostgreSQL) has multiple writer instances internally, with the managed service partitioning tables across them. With this tier, the Foreman can set NumShards: 1 and rely on the cloud to handle write distribution behind a single DSN.

Read replicas at the basic or rewrite tiers (RDS read replicas, Aurora reader instances, AlloyDB read pools) do not help the Foreman directly. The refiller’s candidate scans and step claims need read-after-write consistency, so they go to the primary regardless of how many readers are configured.

Configuration shape

The framework’s SQLDataSourceName with a %d placeholder resolves to per-shard endpoints, one per managed cluster:

foreman.core:
  SQLDataSourceName: postgres://user:pw@foreman-shard-%d.example-host:5432/flows
  NumShards: 4

The exact endpoint format differs per cloud (*.rds.amazonaws.com, *.cloudsql.googleapis.com, *.postgres.database.azure.com), but the substitution is the same. A four-shard deployment is, operationally, four managed clusters defined in IaC.

Per-cloud specifics

AWS. RDS Multi-AZ is the default per shard, with the newer Multi-AZ DB Cluster variant offering sub-35-second failover when that matters. Aurora PostgreSQL lifts the per-shard ceiling 2-3x further when needed. Aurora Limitless Database (GA late 2024) is AWS’s horizontally-sharded offering, but is a relatively new service: plain Aurora has many more production references at any given workload shape.

GCP. Cloud SQL for PostgreSQL with regional HA is the default per shard. AlloyDB for PostgreSQL is Google’s high-throughput rewrite (the Aurora equivalent), GA since 2022 and well past the early-adopter phase. GCP does not have a direct horizontally-sharded equivalent: Spanner with the PostgreSQL interface exists, but Spanner is architecturally a different database (Paxos-replicated global transactions, much higher base latency and cost) and is not a drop-in. Deployments on GCP scale by adding AlloyDB clusters as Foreman shards.

Azure. Azure Database for PostgreSQL Flexible Server with zone-redundant HA is the default per shard. Azure does not have a high-throughput-rewrite tier equivalent to Aurora or AlloyDB; deployments stay on Flexible Server until they need horizontal sharding. However, Azure Cosmos DB for PostgreSQL is managed Citus (Microsoft acquired Citus in 2019), and Citus has been production-tested for horizontal sharding since 2016, so Azure’s sharded tier is the most mature of the three clouds. On Azure, Cosmos DB for PostgreSQL is a viable “skip app-level sharding” option today with relatively low risk.

Choosing the configuration shape

Rough guidance, calibrate to your own benchmarks:

The choice is rarely about pure cost: it is about how many operational units (clusters, runbooks, dashboards) your team can comfortably operate. Managed offerings compress that count substantially compared to self-hosted Postgres, but only the horizontally-sharded tier eliminates the Foreman’s sharding model.

Multiple Foreman Replicas

The Foreman is a normal Microbus microservice and runs with as many Foreman replicas as you want. Every Foreman replica connects to every shard — the shard set is a property of the Foreman service, not of an individual Foreman replica — and each Foreman replica’s worker pool draws steps from all shards. Steps are leased at claim time so only one worker across all Foreman replicas executes a given step, and an expired lease is recovered automatically if a Foreman replica crashes mid-step. No Foreman replica is a single point of failure for dispatch.

Total worker capacity is Workers x Foreman replicas. Scaling out is just adding Foreman replicas — no shard rebalancing, no coordinator election, no client-side changes. The Foreman replicas are interchangeable.

# Same config on every Foreman replica.
foreman.core:
  SQLDataSourceName: foreman:secret@tcp(db-shard-%d.internal:3306)/flows
  NumShards: 4
  Workers: 64

A note on database connections: each Foreman process opens one connection pool per shard, sized explicitly by the SQLConnectionPool config (default 8 connections per shard). Workers reuse those connections from the pool; they do not each hold one. A 64-worker Foreman replica typically holds only a handful of database connections per shard, because workers spend most of their time awaiting downstream task responses rather than the database. Total database connection load grows with Foreman replica count and shard count, not with Workers. The Performance Tuning section below has the full arithmetic and the symptoms that indicate the pool needs adjustment.

Run as many Foreman replicas of this config as your throughput budget needs. The framework’s scheduling rules — Priority and Fairness — still apply, but they are evaluated per Foreman replica rather than globally. With N Foreman replicas each draining independently there is no global ordering across them, only within each; that’s an accepted trade-off for the horizontal scaling.

Performance Tuning

Once the structural backbone is in place, throughput is shaped by a small number of configurable levers and the relationships among them. Most production deployments leave most of these at defaults. Knowing which symptom indicates which lever is the difference between profiling a real bottleneck and over-provisioning blind.

The levers

The connection budget

The total database connection load on any single shard is the product of two configs and the Foreman replica count:

shard_connections = ForemanReplicas * SQLConnectionPool

Aggregate database-wide connection load is that multiplied by NumShards:

total_db_connections = ForemanReplicas * NumShards * SQLConnectionPool

Three Foreman replicas with the default SQLConnectionPool: 8 on a single shard open 24 connections to that shard. The same three replicas on a four-shard deployment open 96 connections aggregate. Raising SQLConnectionPool to 32 in that four-shard setup lands at 384 connections, well above the default max_connections on a stock PostgreSQL install. Raise the pool only when the database has been sized to absorb the extra load. The database side of the equation, max_connections on PostgreSQL or the equivalent setting on other engines, is the limit that bounds how high any of these knobs can usefully go.

Symptom-to-lever map

Read this table alongside the metrics in What to Monitor.

Why the defaults are what they are

Workers: 64 is generous on purpose. Each worker is a goroutine and a socket, both cheap, and a worker blocked on a slow or parked dispatch is just an idle goroutine, not a held thread. Backing off an overloaded or rate-limited downstream is the task’s own job via flow.Retry (which parks the step rather than tying up a worker), so the pool can be sized generously without the engine needing to throttle it.

SQLConnectionPool: 8 is deliberately small. Workers spend most of their wall time awaiting downstream task responses, not the database, so a Foreman replica with 64 workers rarely holds more than a handful of connections per shard at once. The small default also keeps total database load proportional to Foreman replica count rather than to worker count, which means scaling out replicas does not surprise the database. Operators with longer DB-hold times, remote databases with higher per-query latency, or aggressive throughput targets should raise this pool deliberately and budget the database side at the same time.

NumShards: 1 is the right starting point for single-database deployments. Pre-provisioning empty shards is cheap and avoids a future config change, but starting at one avoids coordination overhead a single-shard deployment does not need.

TimeBudget: 2m matches a typical HTTP dispatch ceiling, comfortably longer than most tasks but short enough that a wedged task replica is recovered without hours of stall. Tasks that genuinely need longer should declare it on the endpoint via sub.TimeBudget rather than raising the global ceiling.

DefaultPriority: 5 sits in the middle of the integer space so callers can express both higher-priority work (priority < 5) and lower-priority work (priority > 5) without re-tuning the default. Strict priority means a steady stream of high-priority flows can starve lower bands, by design; if multiple workload classes need to coexist, set each class’s priority explicitly at Create.

Isolate Foreman Replicas

The Foreman mints actor access tokens before dispatching each step so the task runs under the original caller’s identity. That mint happens through the access-token service on the trust-root tier (:666), which means the Foreman holds a publish grant to :666 — a privilege that only a small, named set of microservices in any deployment ever has. Co-bundling the Foreman with ordinary application services would extend that privilege to every sibling in the bundle’s address space: a compromise of any of them gives the attacker the Foreman’s in-memory credentials and, transitively, the ability to call the trust-root tier.

Run Foreman replicas as their own application bundle, separate from the bundles that hold your business microservices. This is the same isolation rule that already applies to the trust-root services themselves, one tier removed — the Foreman is a trust-elevated caller and should be treated accordingly. Other :666 callers in the deployment (for example a control-plane microservice that orchestrates token issuance) belong in their own bundle for the same reason.

The cost is modest: an extra deployment artifact and an extra process per Foreman replica. The benefit is that a compromised business microservice has no in-process path to the Foreman’s privileged credentials, and the blast radius of any single bundle compromise stays bounded by the broker’s ACLs instead of by process memory.

What to Monitor

Operating in production means knowing which dial to turn before users notice. The Foreman exposes a ShardInfo endpoint that reports per-shard latency, row counts, and last error, and emits scheduling metrics that surface saturation. The Prometheus names below are grouped under microbus_flows_*, microbus_steps_*, and microbus_task_*:

• microbus_steps_pending (labelled by priority) — a sustained backlog at a given priority means worker capacity is below the sustained inbound rate. Add Foreman replicas or raise Workers.
• microbus_steps_queue_depth — current size of each replica’s in-memory candidate cache. Plot it on the same panel as microbus_steps_pending: when the pending series stays high but the queue-depth series stays at zero, the refiller is the bottleneck (database shard latency, not worker capacity).
• microbus_steps_oldest_pending_age_seconds (labelled by priority) — a steady increase signals lower-priority work being starved, often paired with a high backlog at higher priorities.
• microbus_steps_fairness_keys (labelled by priority) — number of distinct fairness keys (typically tenants) competing in the most recent refill. Spikes can correlate with one tenant flooding the queue.
• microbus_foreman_timeout_requests_total (labelled by task_url and outcome) — 404 ack-timeout dispatches, i.e. a task whose hosting microservice did not answer. outcome="retry" counts re-probes during a brief absence; outcome="giveup" counts steps failed because the microservice stayed absent past the step’s time budget — alert on this series. These dispatches are the engine’s one engine-level retry for missing microservices.
• dwarf_steps_executed_total{status="retried"} (labelled by task_name) — retry-dispatch churn: every re-dispatch a flow.Retry armed, whether task-owned backoff or an ack-timeout re-probe. A sustained climb means something is backing off repeatedly; pair it with the timeout series to tell a missing microservice from a task riding out a rate limit.

Wire these into your dashboards alongside Foreman-replica CPU and database connection counts. The combination tells you whether your bottleneck is dispatch capacity, database write throughput, the task endpoints themselves, or a single hot tenant.

Failure Modes

A quick reference for what each failure looks like in production:

Putting It Together

A typical production deployment combines all the layers:

• N database shards sized for your write throughput and storage targets, with SQLDataSourceName containing %d and NumShards set to N.
• Each shard backed by a primary + standby with failover routed through the DSN endpoint. This is non-optional at any shard count — any unreachable shard halts Foreman scheduling deployment-wide.
• M Foreman replicas, every one of them connected to all N shards, providing M x Workers total dispatch capacity and surviving any single Foreman-replica restart.
• Foreman replicas isolated in their own application bundle, separate from business microservices, so the Foreman’s :666 privilege does not leak through process memory.

A small but resilient starting point might be NumShards: 2, two databases each with a standby, and two Foreman replicas. From there you scale shards for throughput and Foreman replicas for dispatch capacity, independently.

Operational Notes

• Rolling upgrades. When a Microbus release adds Foreman schema changes, the first Foreman replica to come up after the upgrade runs the migration on every shard before it starts serving — allow extra time on that one restart. Subsequent Foreman replicas detect that migrations are already current and start immediately. The standard upgrade sequence is to roll one Foreman replica with the new release, wait for it to report healthy, then roll the remaining Foreman replicas one at a time.
• Flow lifecycle is operator-managed. The Foreman does not auto-purge flows — every row remains potentially resurrectable (Resume an interrupted flow, Continue into a completed thread, Fork a failed/cancelled flow), and no single retention policy fits both short batch jobs and long human-approval workflows. Plan archival, partitioning, or manual cleanup at the database layer, per shard. Retention discipline matters more on MySQL/MariaDB than on PostgreSQL or SQL Server because those two engines confine the scheduling indexes to non-terminal steps via partial/filtered indexes, while MySQL and MariaDB index every row regardless of status — a difference the engine choice turns on.
• History queries hit the shard the flow lives on. Snapshot, History, and HistoryMermaid look up the flow’s shard by the shardNum-flowID-flowToken composite key embedded in the flow key, so a query for a specific flow only touches one shard. List enumerates across all shards and merges in reverse chronological order; the shards are queried in parallel so latency stays roughly flat, but aggregate database load grows linearly with shard count.
• You can pre-provision empty shards. Setting NumShards: 4 on day one when you only need two-shard throughput is fine — the extra shards open empty and start receiving flows immediately. This lets you add database capacity ahead of demand without a config change later.
• Subgraphs and forks share the parent’s shard. Cross-shard joins never appear in normal operation because every flow tree is shard-local by construction.

Further Reading

• Foreman — full configuration reference, including Workers and TimeBudget.
• Priority and Fairness — per-flow scheduling controls that work alongside this scaling model.
• Application Bundling — how to package microservices into the bundles this page recommends.
• Deploy to Production — broader production-deployment checklist for a Microbus solution, including NATS HA and other deployment-wide concerns.