Production Readiness

Agentic workflows are durable: every step’s input, output, and state delta is persisted to a SQL database by the Foreman. That durability is what lets a flow survive a process crash or a deploy mid-execution — but it also means the Foreman’s database is on the critical path for every running flow, and the Foreman itself is on the critical path for every dispatched step.

Production-grade operation comes down to four cross-cutting properties:

• Scalability — handle growing load without compromising performance.
• Availability — stay ready to serve, with minimal downtime.
• Reliability — consistently produce correct results across failure modes.
• Performance — carry out work at the expected rate under peak load.

The structural backbone covering scalability and availability is three layers:

1. Multiple database shards — distribute flows across more than one database instance so write throughput and storage scale horizontally.
2. Hot/cold replicas per shard — give each shard a standby. Any unreachable shard halts Foreman scheduling across the whole deployment, even with just one shard.
3. Multiple Foreman replicas — run several Foreman instances against the same shard set so step dispatch survives a worker crash and total worker capacity scales with replica count.

The later sections cover what to monitor (Performance), how the system behaves under failure (Reliability), and the security isolation that protects the Foreman’s trust-elevated credentials.

The diagram above shows the structural backbone for a deployment with three Foreman replicas and two database shards: every Foreman replica reaches every shard (the bus-bar across the top), and each shard has its own hot/cold pair with the cold standby kept in sync from the hot primary at the database layer (the bottom arrow). Failover from hot to cold is handled outside the framework.

Database Sharding

A single database eventually becomes the throughput ceiling for everything: state persistence, step claims, history queries. The Foreman lets you distribute flows across multiple database instances via its Shards config, a list in which every shard is declared explicitly.

Each shard is an independent database — usually on a separate instance, but at minimum a separate logical database with its own write capacity. Migrations run independently on each shard at startup.

foreman.core:
  Shards: '[{"index":1,"dsn":"foreman:secret@tcp(db1.internal:3306)/flows","virtualCPUs":16},
            {"index":2,"dsn":"foreman:secret@tcp(db2.internal:3306)/flows","virtualCPUs":16},
            {"index":3,"dsn":"foreman:secret@tcp(db3.internal:3306)/flows","virtualCPUs":32},
            {"index":4,"dsn":"foreman:secret@tcp(db4.internal:3306)/flows","virtualCPUs":32}]'

Every entry carries four fields:

• index identifies the shard and is encoded into every flow key created on it. Indices must be >= 1 and unique, but need not be contiguous — shards 1 and 99 are fine. Because the index drives routing, the index-to-DSN mapping must be identical on every replica and stable across restarts.
• dsn is the shard’s connection string, with the dialect auto-detected. It is used exactly as given — the engine never formats or rewrites it, so a percent-encoded credential (a password p@ss written p%40ss) survives intact.
• virtualCPUs is the CPU count of that shard’s database server, a fact off the instance’s spec sheet. It sizes the shard’s connection budget and weights how many new flows land there. Left at 0 the engine assumes 2 — the smallest current-generation instance any major cloud sells — so the assumed pool stays safe on a small machine but badly under-uses a large one. Declare it.
• cordoned excludes the shard from new-flow placement while everything already resident proceeds normally. This is how a shard is retired, without draining it by hand.

Because each shard names its own database, shards do not have to be uniform. The four-shard example above pairs two 16-vCPU instances with two 32-vCPU ones, and the engine places roughly twice as much new work on each of the larger pair.

New flows are assigned to a shard in proportion to its declared capacity when they’re created. Once a flow lands on a shard, it stays there for life — every step, every retry, every interrupt, every history row lives on the same shard as the parent flow. Subgraphs and forked flows inherit their parent’s shard so an entire flow tree is colocated, which keeps cross-shard joins out of the critical path.

Shards can be added but never removed: a flow created on a shard a peer does not know about is unroutable there. Retire a shard by setting "cordoned": true on its entry — resident flows keep executing, and shard-pinned creations (subgraph children, Continue on a thread, forks) still land on it, but no new flow tree starts there.

Changing the shard set is a coordinated restart of the whole fleet, not a live config edit. Shards is applied once during startup and has no config callback, and the engine rejects a re-shard on a running engine outright. Every replica must come back on the same shard list.

Choosing a Database Engine

The Foreman is portable across four SQL engines: PostgreSQL, Microsoft SQL Server, MySQL/MariaDB, and SQLite. All four are first-class targets in the sense that the framework’s test suite passes against each, but they behave very differently under the Foreman’s concurrent INSERT and UPDATE workload. Pick the engine for the shape of your deployment rather than by what you already have running, and budget for the per-engine tuning below.

PostgreSQL — recommended for production

PostgreSQL’s MVCC model means concurrent INSERTs never lock each other on secondary indexes, and the default READ COMMITTED isolation level does not take gap locks. The Foreman’s hot path, which is many parallel INSERTs against the same shard’s dwarf_steps table during fan-out, runs without engine-level deadlocks at any concurrency your worker pool can produce. PostgreSQL 13 or newer is required for the partial indexes the schema declares.

PostgreSQL also provides the most efficient indexing for the Foreman’s scheduling reads. The schema declares the indexes that serve pollPendingSteps and the refiller’s candidate scan as partial indexes filtered to status IN ('pending', 'running'), so the working set in each index stays bounded to active steps regardless of how many completed or failed steps accumulate in the underlying table. Index lookups stay fast over the long term without depending on aggressive retention.

No special tuning is needed for correctness. For headroom, declare each cluster’s real virtualCPUs so the engine sizes its pool against the machine it actually has, size the cluster’s max_connections above that derived pool (the engine splits it across replicas, so the cap does not grow with the fleet), and raise shared_buffers to roughly 25% of the database host’s RAM.

Microsoft SQL Server

SQL Server requires one database-level change for the Foreman to behave well: enable Read Committed Snapshot Isolation on each shard database.

ALTER DATABASE <shard_database> SET READ_COMMITTED_SNAPSHOT ON;

This gives SQL Server PostgreSQL-style non-blocking reads and eliminates the pessimistic locking that would otherwise behave similarly to MySQL’s REPEATABLE READ. Without it, expect 1205 deadlocks under sustained parallel flow creation. Beyond enabling RCSI on every shard, no other server-level tuning is mandatory.

SQL Server’s filtered indexes give the foreman the same scheduling-index containment benefit as PostgreSQL partial indexes: the selection and saturation indexes on dwarf_steps only carry entries for non-terminal steps, so their size and lookup cost stay bounded to active work rather than growing with the total number of steps ever recorded.

MySQL and MariaDB

InnoDB at its default REPEATABLE READ isolation level takes next-key locks (row plus gap) on every secondary-index touch. Two flows being created on the same shard at the same time can lock overlapping ranges of the step table’s selection or saturation indexes in different orders, and InnoDB aborts the loser with Error 1213 (40001) Deadlock found when trying to get lock. This is correct engine behavior, not a Foreman bug. The Foreman’s flow-creation path retries on lock contention so an individual Error 1213 is invisible to callers, but a high sustained deadlock rate caps throughput and inflates p99 latency.

Operators running MySQL or MariaDB should set the following in the server’s [mysqld] section before putting any real load on the deployment:

MariaDB 10.6 or newer is recommended. 10.5 is the minimum version that handles the schema’s JSON columns correctly; older versions accept the DDL but ship with less mature deadlock detection.

Unlike PostgreSQL and SQL Server, neither MySQL nor MariaDB supports partial (or filtered) indexes, so the foreman schema falls back to full composite indexes on dwarf_steps for both of those engines. The indexes that drive scheduling carry an entry for every step, terminal or not, and grow proportionally to the total number of steps ever recorded rather than to the size of the active working set. Operators running MySQL or MariaDB should plan for more aggressive flow retention than they would on the other engines, or accept that scheduling lookup cost and index maintenance work will increase over time. The Operational Notes section below discusses the retention surface the framework exposes.

For multi-shard deployments, every database named in a shard’s dsn must exist before the Foreman starts. The framework runs schema migrations on each but does not issue CREATE DATABASE itself.

SQLite — testing and development only

SQLite uses a single database-level writer lock, which makes engine-level deadlocks structurally impossible but caps write throughput at roughly one transaction at a time. The framework reaches for SQLite on both non-deployed paths, and Shards is not consulted on either: under TESTING the engine opens isolated in-memory databases keyed by the Microbus plane, so every replica in a test app resolves to the same throwaway set; under LOCAL a Foreman that declares no shards falls back to a single file:shard_1.local.sqlite file shard. The connection layer injects a one-second busy timeout so the worker pool does not immediately fail on SQLITE_BUSY during fan-out. SQLite is intentionally not a production target; the single-writer ceiling will be the bottleneck before anything else in the stack.

Sizing the shard count

Engine choice directly affects how many shards you need to hit a given target throughput. The numbers below are rough sizing starting points and should be calibrated against your own workload using the signals listed in What to Monitor.

The wide MySQL/MariaDB range reflects how much the isolation-level change matters. The same hardware delivers roughly an order of magnitude more sustained flow creation rate at READ-COMMITTED than at the default REPEATABLE READ, simply because most of the otherwise-deadlocking INSERTs commit on the first try.

Connection pools

Each Foreman replica opens a connection pool per shard, and the engine sizes that pool itself from the shard’s declared virtualCPUs. The cap is the measured knee beyond which added connections only queue: roughly 12x the CPU count on a server of 32 vCPUs or more, 6x below that, where extra connections actively destabilize throughput rather than merely failing to improve it. The engine then divides the shard’s budget among the replicas registered against it, so the pool shrinks as the fleet grows rather than multiplying against the database.

The practical consequence is that the pool follows the hardware without an operator knob, as long as virtualCPUs is declared honestly. The MaxOpenConns override exists for the cases the engine cannot see — a database shared with something else, an external pooler, a deliberate global cap — and Performance Tuning below covers when to reach for it.

Hot/Cold Replicas per Shard

The Foreman’s database is a single point of failure for the entire deployment at every shard count. Scheduling, recovery polling, and retry re-dispatch fan out across every shard on each cycle, and any shard that errors fails the whole cycle. With one shard, losing it halts everything; with four shards, losing any one of them also halts everything — sharding doesn’t degrade you gracefully to 75% capacity, it multiplies the failure surface. Hot/cold per shard is therefore mandatory at any shard count.

The pattern is the same regardless of shard count: each shard gets its own database standby with failover handled at the database layer.

Microbus does not manage database failover itself. The Foreman holds one connection string per shard and treats whatever that string resolves to as the authoritative endpoint for that shard. The operator-side pattern is:

• Each shard is provisioned as a primary plus one or more standbys, with replication configured in the database (MySQL semi-sync, PostgreSQL streaming replication, SQL Server Always On, etc.).
• The DSN points at a stable endpoint that routes to the current primary — typically a managed-database failover endpoint, an HAProxy / ProxySQL in front of the cluster, or a DNS name that’s repointed by your failover tooling.
• When the primary fails, the endpoint flips to the new primary. The Foreman’s connection pool reconnects and resumes.

This pattern is intentionally outside the framework: every cloud and on-prem environment has its own preferred way to do database HA, and the Foreman’s role is only to keep one DSN per shard and reconnect cleanly when the connection drops.

A reasonable production posture is N declared shards with a primary plus a synchronous standby each, so a single database failure costs reconnect latency rather than a full Foreman outage. Because the DSN is used verbatim, the failover endpoint goes straight into the shard’s dsn field.

Managed Database on Cloud Providers

The framework’s “shard = independent database instance” model maps directly onto cloud-managed Postgres clusters. Each Foreman shard’s DSN points at one managed cluster, and the cloud provider handles per-shard HA, backups, failover, patching, and monitoring that would otherwise be operator work. The operational cost of N Foreman shards stays mostly proportional to N (one IaC definition per shard), rather than scaling with operator headcount.

All three major clouds offer the same broad tiers of managed Postgres, with meaningful gaps at the upper tiers:

The three tiers represent meaningfully different architectures, not just different price points:

• Basic managed Postgres is plain Postgres with the cloud handling HA. One primary, one or more standbys, failover in tens of seconds to a couple of minutes. Same per-primary ceilings as self-hosted Postgres, with the operator burden lifted.
• High-throughput rewrite (Aurora and AlloyDB) keeps the single-writer architecture but replaces the storage layer with a distributed one, offloading checkpoint and WAL work. Roughly 2 to 4 times the write throughput of basic managed Postgres on equivalent compute. The single-primary considerations from Performance Tuning (WAL throughput, autovacuum lag, the connection budget) all still apply per cluster, just with more headroom.
• Horizontally sharded Postgres (Aurora Limitless, Cosmos DB for PostgreSQL) has multiple writer instances internally, with the managed service partitioning tables across them. With this tier, the Foreman can declare a single shard and rely on the cloud to handle write distribution behind one DSN.

Read replicas at the basic or rewrite tiers (RDS read replicas, Aurora reader instances, AlloyDB read pools) do not help the Foreman directly. The refiller’s candidate scans and step claims need read-after-write consistency, so they go to the primary regardless of how many readers are configured.

Configuration shape

Each managed cluster is one entry in Shards, carrying that cluster’s own endpoint and the vCPU count of the instance class it was provisioned at:

foreman.core:
  Shards: '[{"index":1,"dsn":"postgres://user:pw@shard-1.example-host:5432/flows","virtualCPUs":16},
            {"index":2,"dsn":"postgres://user:pw@shard-2.example-host:5432/flows","virtualCPUs":16},
            {"index":3,"dsn":"postgres://user:pw@shard-3.example-host:5432/flows","virtualCPUs":16},
            {"index":4,"dsn":"postgres://user:pw@shard-4.example-host:5432/flows","virtualCPUs":16}]'

The exact endpoint format differs per cloud (*.rds.amazonaws.com, *.cloudsql.googleapis.com, *.postgres.database.azure.com), but the shape is the same. A four-shard deployment is, operationally, four managed clusters defined in IaC. Resizing a cluster’s instance class is a change to that entry’s virtualCPUs, which is how the engine learns about the new capacity.

Per-cloud specifics

AWS. RDS Multi-AZ is the default per shard, with the newer Multi-AZ DB Cluster variant offering sub-35-second failover when that matters. Aurora PostgreSQL lifts the per-shard ceiling 2-3x further when needed. Aurora Limitless Database (GA late 2024) is AWS’s horizontally-sharded offering, but is a relatively new service: plain Aurora has many more production references at any given workload shape.

GCP. Cloud SQL for PostgreSQL with regional HA is the default per shard. AlloyDB for PostgreSQL is Google’s high-throughput rewrite (the Aurora equivalent), GA since 2022 and well past the early-adopter phase. GCP does not have a direct horizontally-sharded equivalent: Spanner with the PostgreSQL interface exists, but Spanner is architecturally a different database (Paxos-replicated global transactions, much higher base latency and cost) and is not a drop-in. Deployments on GCP scale by adding AlloyDB clusters as Foreman shards.

Azure. Azure Database for PostgreSQL Flexible Server with zone-redundant HA is the default per shard. Azure does not have a high-throughput-rewrite tier equivalent to Aurora or AlloyDB; deployments stay on Flexible Server until they need horizontal sharding. However, Azure Cosmos DB for PostgreSQL is managed Citus (Microsoft acquired Citus in 2019), and Citus has been production-tested for horizontal sharding since 2016, so Azure’s sharded tier is the most mature of the three clouds. On Azure, Cosmos DB for PostgreSQL is a viable “skip app-level sharding” option today with relatively low risk.

Choosing the configuration shape

Rough guidance, calibrate to your own benchmarks:

The choice is rarely about pure cost: it is about how many operational units (clusters, runbooks, dashboards) your team can comfortably operate. Managed offerings compress that count substantially compared to self-hosted Postgres, but only the horizontally-sharded tier eliminates the Foreman’s sharding model.

Multiple Foreman Replicas

The Foreman is a normal Microbus microservice and runs with as many Foreman replicas as you want. Every Foreman replica connects to every shard — the shard set is a property of the Foreman service, not of an individual Foreman replica — and each Foreman replica’s worker pool draws steps from all shards. Steps are leased at claim time so only one worker across all Foreman replicas executes a given step, and an expired lease is recovered automatically if a Foreman replica crashes mid-step. No Foreman replica is a single point of failure for dispatch.

Total worker capacity is the per-replica worker ceiling multiplied by the replica count. Scaling out is just adding Foreman replicas — no shard rebalancing, no coordinator election, no client-side changes. The Foreman replicas are interchangeable.

# Same config on every Foreman replica.
foreman.core:
  Shards: '[{"index":1,"dsn":"foreman:secret@tcp(db1.internal:3306)/flows","virtualCPUs":16},
            {"index":2,"dsn":"foreman:secret@tcp(db2.internal:3306)/flows","virtualCPUs":16},
            {"index":3,"dsn":"foreman:secret@tcp(db3.internal:3306)/flows","virtualCPUs":32},
            {"index":4,"dsn":"foreman:secret@tcp(db4.internal:3306)/flows","virtualCPUs":32}]'

The shard list must be byte-for-byte agreed on across the fleet: a flow created on a shard a peer does not know about is unroutable there.

A note on database connections: each Foreman process opens one connection pool per shard, and workers reuse those connections rather than each holding one. Because the engine derives each pool from the shard’s virtualCPUs and splits that budget across the replicas registered against the shard, total load on a shard is bounded by the shard’s own capacity rather than growing with the fleet — adding replicas divides the same budget instead of multiplying it. Performance Tuning below covers the symptoms that indicate the derivation needs overriding.

Run as many Foreman replicas of this config as your throughput budget needs. The framework’s scheduling rules — Priority and Fairness — still apply, but they are evaluated per Foreman replica rather than globally. With N Foreman replicas each draining independently there is no global ordering across them, only within each; that’s an accepted trade-off for the horizontal scaling.

Performance Tuning

Once the structural backbone is in place, throughput is shaped by a small number of configurable levers and the relationships among them. Most production deployments leave most of these at defaults. Knowing which symptom indicates which lever is the difference between profiling a real bottleneck and over-provisioning blind.

The levers

The connection budget

The budget belongs to the shard, not to the replica. The engine caps a shard’s pool at the knee for its declared hardware — about 12 connections per vCPU at 32 vCPUs or more, 6 per vCPU below that — and then divides that cap among the replicas registered against the shard. So a 16-vCPU shard tops out near 96 connections whether one replica or eight are drawing on it, and adding replicas narrows each one’s share rather than growing the total.

That inverts the arithmetic operators used to do by hand: the number to check against the database’s max_connections is the per-shard cap, and it does not move when the fleet scales. What does move it is virtualCPUs, so an entry that understates its hardware is the usual cause of a pool that feels too small.

MaxOpenConns overrides the derivation for every shard at once, which makes it the wrong tool for a fleet of unequal shards — it pins the 32-vCPU shard to the same number as the 2-vCPU one. Reach for it only when something outside the engine’s view owns the budget: a database shared with another application, an external pooler such as PgBouncer sitting in front of the shards, or a deliberate global cap imposed by the database’s own connection limit.

Symptom-to-lever map

Read this table alongside the metrics in What to Monitor.

Why the defaults are what they are

Workers and MaxOpenConns default to deriving because the two numbers are a function of facts the engine can measure — each shard’s connection budget and the round-trip time it actually observes — and a hand-picked constant goes stale the moment the hardware or the workload moves. Both remain settable as expert overrides, and neither should be the first thing reached for.

One value of Workers is not a derivation hint but a distinct shape: Workers: 0 stands up a replica that creates flows, awaits them, and serves reads, but never executes a task. That is why the “derive” sentinel is -1 rather than 0 — the config has to stay able to express a dispatch-free replica.

Each worker is a goroutine and a socket, both cheap, and a worker blocked on a slow or parked dispatch is just an idle goroutine, not a held thread. Backing off an overloaded or rate-limited downstream is the task’s own job via flow.Retry, which parks the step rather than tying up a worker, so the derived ceiling can be generous without the engine needing to throttle it.

A single shard is the right starting point for single-database deployments. Declaring extra shards ahead of demand is cheap and saves a future fleet restart, but starting at one avoids coordination a single-shard deployment does not need.

TimeBudget: 2m matches a typical HTTP dispatch ceiling, comfortably longer than most tasks but short enough that a wedged task replica is recovered without hours of stall. Tasks that genuinely need longer should declare it on the endpoint via sub.TimeBudget rather than raising the global ceiling.

DefaultPriority: 5 sits in the middle of the integer space so callers can express both higher-priority work (priority < 5) and lower-priority work (priority > 5) without re-tuning the default. Strict priority means a steady stream of high-priority flows can starve lower bands, by design; if multiple workload classes need to coexist, set each class’s priority explicitly at Create.

Isolate Foreman Replicas

The Foreman mints actor access tokens before dispatching each step so the task runs under the original caller’s identity. That mint happens through the access-token service on the trust-root tier (:666), which means the Foreman holds a publish grant to :666 — a privilege that only a small, named set of microservices in any deployment ever has. Co-bundling the Foreman with ordinary application services would extend that privilege to every sibling in the bundle’s address space: a compromise of any of them gives the attacker the Foreman’s in-memory credentials and, transitively, the ability to call the trust-root tier.

Run Foreman replicas as their own application bundle, separate from the bundles that hold your business microservices. This is the same isolation rule that already applies to the trust-root services themselves, one tier removed — the Foreman is a trust-elevated caller and should be treated accordingly. Other :666 callers in the deployment (for example a control-plane microservice that orchestrates token issuance) belong in their own bundle for the same reason.

The cost is modest: an extra deployment artifact and an extra process per Foreman replica. The benefit is that a compromised business microservice has no in-process path to the Foreman’s privileged credentials, and the blast radius of any single bundle compromise stays bounded by the broker’s ACLs instead of by process memory.

What to Monitor

Operating in production means knowing which dial to turn before users notice. The Foreman exposes a ShardInfo endpoint that reports per-shard latency, row counts, and last error. The scheduling metrics come from the embedded engine and carry a dwarf_ prefix; the Foreman contributes one microbus_-prefixed series of its own.

Two of these are computed by querying the shared shard databases, so every replica reports the same number and they must aggregate with max, not sum — summing multiplies the backlog by the replica count. They are called out below. Every other engine gauge is per-replica and sums normally.

• dwarf_steps_pending (labelled by priority, aggregate with max) — a sustained backlog at a given priority means worker capacity is below the sustained inbound rate. Add Foreman replicas.
• dwarf_steps_queue_depth — current size of each replica’s in-memory candidate cache. Plot it on the same panel as the pending series: when pending stays high but queue depth stays at zero, the refiller is the bottleneck (database shard latency, not worker capacity).
• dwarf_steps_oldest_pending_age_seconds (labelled by priority, aggregate with max) — a steady increase signals lower-priority work being starved, often paired with a high backlog at higher priorities.
• dwarf_peer_replicas (per replica, per shard) — how many replicas this one currently sees holding connections to that shard, which is the divisor its pool is sized by. Replicas should agree, so a spread across the fleet is itself the signal: one replica reading 3 while its peers read 4 is sizing its pool for a fleet that does not exist. It is deliberately slow to fall, so a drop lags a real departure by a reading or two.
• dwarf_peer_blind_seconds (per replica, per shard) — how long since that shard’s peer registry was last read successfully; zero on a healthy replica. Past two read cadences the replica is blind on that shard: it holds its last known fleet and stops partitioning that shard’s candidates. Check it first whenever a shard’s counts look frozen.
• dwarf_state_in_flight_bytes, over dwarf_state_in_flight_steps — flow state this replica is holding across in-flight task dispatches. The ratio is the mean state a task carries: a large mean says state size is what loads the replica, a small mean against a large count says fan-out width is. This is the series that catches a workflow carrying large blobs forward, so pair it with the housekeeping guidance in State.
• microbus_foreman_timeout_requests_total (labelled by task_url and outcome) — 404 ack-timeout dispatches, i.e. a task whose hosting microservice did not answer. outcome="retry" counts re-probes during a brief absence; outcome="giveup" counts steps failed because the microservice stayed absent past the step’s time budget — alert on this series. These dispatches are the engine’s one engine-level retry for missing microservices.
• dwarf_steps_executed_total{status="retried"} (labelled by task_name) — retry-dispatch churn: every re-dispatch a flow.Retry armed, whether task-owned backoff or an ack-timeout re-probe. A sustained climb means something is backing off repeatedly; pair it with the timeout series to tell a missing microservice from a task riding out a rate limit.

Wire these into your dashboards alongside Foreman-replica CPU and database connection counts. The combination tells you whether your bottleneck is dispatch capacity, database write throughput, the task endpoints themselves, or a single hot tenant.

Failure Modes

A quick reference for what each failure looks like in production:

Putting It Together

A typical production deployment combines all the layers:

• N database shards sized for your write throughput and storage targets, each declared in Shards with its own DSN and its real virtualCPUs.
• Each shard backed by a primary + standby with failover routed through the DSN endpoint. This is non-optional at any shard count — any unreachable shard halts Foreman scheduling deployment-wide.
• M Foreman replicas, every one of them carrying an identical Shards list, providing M times the per-replica worker ceiling in total dispatch capacity and surviving any single Foreman-replica restart.
• Foreman replicas isolated in their own application bundle, separate from business microservices, so the Foreman’s :666 privilege does not leak through process memory.

A small but resilient starting point might be two declared shards, two databases each with a standby, and two Foreman replicas. From there you scale shards for throughput and Foreman replicas for dispatch capacity, independently.

Operational Notes

• Engine upgrades are a maintenance window, not a rolling deploy. Schema migrations are forward-only and run at startup, so the first replica on a new release migrates the databases that every still-running old replica is reading, and there is no downgrade path. Back up every shard, drain the fleet, bring up a single replica and confirm it comes up clean, then start the rest. Flows survive it untouched: pending steps stay pending and interrupted flows stay parked. Allow extra time on that first restart — it migrates every shard before it serves.
• Flow lifecycle is operator-managed. The Foreman does not auto-purge flows — every row remains potentially resurrectable (Resume an interrupted flow, Continue into a completed thread, Fork a failed/cancelled flow), and no single retention policy fits both short batch jobs and long human-approval workflows. Plan archival, partitioning, or manual cleanup at the database layer, per shard. Retention discipline matters more on MySQL/MariaDB than on PostgreSQL or SQL Server because those two engines confine the scheduling indexes to non-terminal steps via partial/filtered indexes, while MySQL and MariaDB index every row regardless of status — a difference the engine choice turns on.
• History queries hit the shard the flow lives on. Snapshot, History, and HistoryMermaid look up the flow’s shard by the shardNum-flowID-flowToken composite key embedded in the flow key, so a query for a specific flow only touches one shard. List enumerates across all shards and merges in reverse chronological order; the shards are queried in parallel so latency stays roughly flat, but aggregate database load grows linearly with shard count.
• You can pre-provision empty shards. Declaring four shards on day one when you only need two-shard throughput is fine — the extra ones open empty and start receiving flows immediately. Since any change to Shards costs a coordinated fleet restart, provisioning ahead of demand is how you avoid paying for one later.
• Subgraphs and forks share the parent’s shard. Cross-shard joins never appear in normal operation because every flow tree is shard-local by construction.

Further Reading

• Foreman — full configuration reference, including Workers and TimeBudget.
• Priority and Fairness — per-flow scheduling controls that work alongside this scaling model.
• Application Bundling — how to package microservices into the bundles this page recommends.
• Deploy to Production — broader production-deployment checklist for a Microbus solution, including NATS HA and other deployment-wide concerns.