Reliability and Recovery

The Foreman dispatches every workflow step to a task endpoint over the bus. When a dispatch fails — the task errored, a downstream it called is down, or its hosting microservice is absent — something has to decide whether to retry and wait or fail the step. Microbus puts almost all of that decision in the layer that actually owns the failing resource: the task itself. The engine stays out of backpressure.

The Engine Never Reads Status Codes

The orchestration engine treats any error a task returns as terminal for that attempt. It does not inspect the HTTP status or the error text. If the graph declares an onError (or onTimeout) transition from the step, the error routes there; otherwise the step — and the flow — fails. There is no rate-limit or unavailability signal the engine wraps an error with, and no engine-level controller that paces or parks dispatches.

This is deliberate. The engine’s only vantage is the task URL, but the resources that actually get scarce are finer: an LLM provider’s per-account token quota is shared across many task URLs and invisible to the engine, and a downstream service a task calls is a downstream of the task, not of the engine. The engine cannot be the resource-accurate controller for either, so backpressure belongs to the layer that holds the resource identity.

Task-Owned Backoff

A task that wants to ride out a transient failure — a 429 from a rate-limited provider, a 503 from a downstream that is momentarily unavailable — detects that condition itself and arms flow.Retry. The retry bound is wall-clock: giveUpAfter is the elapsed time since the step was first created, so it includes execution time and survives parks. The engine consumes only the backoff shape to compute the next re-dispatch delay; the give-up decision is made in-task, so the returned bool tracks return nil versus return err.

err := callDownstream(ctx)
switch {
case isRateLimited(err), isUnavailable(err): // e.g. HTTP 429 / 503
    if flow.Retry(100*time.Millisecond, 2.0, 10*time.Second, time.Hour) {
        return nil // re-run after wall-clock-bounded backoff
    }
    return err // horizon exceeded: onError or fail
default:
    return err // ordinary failure: onError or fail
}

Because the task owns the retry, it also owns the policy. Rate limits are a known-reset condition, so a constant interval at the provider’s Retry-After is the principled wait (multiplier 1.0, no per-interval cap). An unknown recovery — a downstream that may be back in seconds or minutes — wants exponential backoff. The LLM core services already do exactly this: a provider attaches a retryAfter to a genuine throttle, and the ChatLoop task backs itself off against its own time budget.

A present microservice whose own downstream is down never reaches the Foreman as a failure to retry — the Foreman calls it fine, it runs, and it owns its retry against its downstream. A 503 or 529 from such a microservice is an ordinary step failure: it flows through the graph’s error and timeout transitions and is not retried by the engine. The microservice, not the orchestrator, decides how to handle its own degradation.

The One Engine-Level Retry: Missing Microservices

There is exactly one case the task cannot own, because the task body never runs: its hosting microservice is momentarily absent. When no replica acknowledges a dispatch within the ack timeout, the connector fails fast with a 404 carrying an ack timeout: prefix — the framework’s “no responder answered” signal. This retry is designed for the short, intermittent gaps a healthy deployment produces — a microservice being rolled, briefly scaled to zero, or mid-restart — so a routine deploy doesn’t fail in-flight flows. There is no task code present to own a policy, so the Foreman owns it as a genuine last resort.

On a 404 ack-timeout — and only that, never a 404 returned by a task that actually ran — the Foreman arms flow.Retry on the carrier and returns nil instead of failing the step. It re-probes with exponential backoff (the step’s time budget divided into a handful of probes), because the recovery time of an absent microservice is unknown. The horizon is the step’s own time budget: flow.Retry gives up the moment the next probe would overshoot it. So a brief absence heals transparently, but a 404 that persists past the horizon becomes an ordinary failure — the step fails with the original ack timeout: error, exactly as it should when the microservice is gone for good.

This is the only engine-level retry in the system; every other backoff is task-owned.

Observing It

Two signals track the ack-timeout retry:

microbus_foreman_timeout_requests_total — labeled {task_url, outcome}. outcome="retry" is emitted on each ack-timeout re-probe (the “a microservice is briefly missing” signal); outcome="giveup" is emitted once the budget horizon is spent and the step is failed. The giveup series is the alertable one — a microservice absent long enough to drop work is something a human must see. Wire an alert on it.
dwarf_steps_executed_total{task_name, status="retried"} — retry-dispatch churn, counting every re-dispatch a flow.Retry armed (task-owned or ack-timeout alike).

Detecting Failed Flows

When a step fails — an ack-timeout give-up, a horizon-exceeded retry, or an ordinary task error with no onError handler — the flow lands in failed status with outcome.Error set. The Foreman never auto-purges, so failed flows stay queryable and recoverable until you act on them.

Find them with List, which takes a workflow.Query and returns newest-first with an opaque pagination cursor:

flows, nextCursor, err := client.List(ctx, workflow.Query{
    Status:       workflow.StatusFailed,
    WorkflowName: "credit-approval",      // optional: one workflow
    TaskName:     "VerifyCredit",         // optional: failed at a specific task
    NewerThan:    24 * time.Hour,         // optional: only the last day
    Search:       "ack timeout",          // optional: substring over the error column
    Limit:        100,
})

Each result is a workflow.FlowSummary carrying FlowKey, Status, WorkflowName, TaskName, timestamps, and — for a failed flow — the Error string that explains why it failed. Search is a case-insensitive substring match over the flow’s workflow_url, workflow_name, current task_name, error, and cancel_reason columns, so Search: "ack timeout" is exactly how you isolate the missing-microservice give-ups from ordinary failures. Page by passing the returned nextCursor back as Query.Cursor on the next call.

This same query shape drives operator dashboards and archival scripts.

Recovering Failed Flows

Once you have read the cause and fixed it — the microservice is back, the quota is raised, the bad input is corrected — re-run the work with Fork.

A terminal flow is immutable: the engine never re-runs it in place. Instead, Fork(stepKey, stateOverrides) clones the flow’s prefix up to a chosen step into a new, self-contained running flow and re-executes from that step, leaving the original untouched. The step key comes from History; stateOverrides are applied to that forked step, so target the shared upstream input fields that let it succeed. The fork point may be any recorded step, including one inside a subgraph — the clone re-runs from there and bubbles back up to the root. A partially-failed fan-out is recovered by forking one failed branch at a time.

The operator loop is List → read Error → fix → Fork:

flows, _, _ := client.List(ctx, workflow.Query{
    Status:    workflow.StatusFailed,
    Search:    "ack timeout",
    NewerThan: 24 * time.Hour,
})
for _, f := range flows {
    // locate the failed step, then fork from it into a fresh running flow
    steps, _ := client.History(ctx, f.FlowKey)
    for _, s := range steps {
        if s.Status != workflow.StatusFailed {
            continue
        }
        newFlowKey, err := client.Fork(ctx, s.StepKey, nil)
        if err != nil {
            log.Printf("fork %s: %v", f.FlowKey, err)
            continue
        }
        log.Printf("recovered %s as %s", f.FlowKey, newFlowKey)
    }
}

There is no bulk Fork — an operator loops it over the List results, which keeps each recovery’s overrides and error handling explicit, and the original failed flows remain as an audit trail until Purged. (Purge covers bulk deletion of flows you do not intend to recover.)