Graceful Shutdown

Microservices are designed to run on cloud hardware that can shut down at unpredictable times. When a microservice receives the signal to terminate, the framework stops accepting new work, lets in-flight work finish, then tears down the transport and observability exporters in a fixed order so nothing on the network or disk is left half-flushed.

Shutdown Phases

Connector.Shutdown(ctx) runs the following phases in order. Phase 6 is the only one whose budget is non-negotiable - it runs on context.WithoutCancel(ctx) so an expired caller deadline cannot skip an OTel flush or the dlru peer offload.

  1. Stop accepting work. Tickers are disabled; auto-activated subscriptions are deactivated. Manual subscriptions (e.g. dlru’s :888/dcache/...) stay on-bus.
  2. OnShutdown(ctx). Runs before any drain, while the lifetime context is still valid, dlru is still up, the transport is still connected, outbound calls work, and svc.Go / svc.Parallel can still launch goroutines. User code that owns long-lived workers - a worker pool, a refiller, an ML executor - should drain them here. Bounded by the deadline on ctx.
  3. Soft drain. Cooperatively polls in-flight operations (incoming requests, ticker invocations, goroutines launched via Go / Parallel) until the counter reaches zero or the soft-drain budget elapses. The lifetime context is still valid during this phase, so goroutines that haven’t observed shutdown yet keep running.
  4. Cancel the lifetime context (svc.Lifetime()). This is the escalation step: goroutines launched on the lifetime context observe ctx.Done() and should exit promptly.
  5. Hard drain. A short tail for cancellation-aware goroutines to exit after observing the cancel.
  6. Mandatory teardown. Close the distributed cache (with its peer offload), unsubscribe the response sub, disconnect the transport, flush OTel traces and metrics. Runs on context.WithoutCancel(ctx) wrapped in a 2-second budget so the caller’s deadline expiring cannot skip data flushes.

Drain Budget Partitioning

The soft and hard drain budgets are derived from the deadline on the ctx passed to Shutdown:

  • With a deadline (the normal case when called by application.Run): the framework reserves the 2-second mandatory-teardown budget out of the remaining time, then partitions the rest. Hard drain takes min(remaining/4, 2s), floored at 100ms. Soft drain takes whatever is left.
  • Without a deadline: fixed defaults of 8 seconds soft and 2 seconds hard.

application.Run reads MICROBUS_SHUTDOWN_TIME_BUDGET (default 24s) and applies it as the deadline on the ctx it passes to Shutdown, so the partition above runs against that envelope. MICROBUS_SHUTDOWN_TIME_BUDGET is one of the framework’s environment variables for timing.

The Lifetime Context

svc.Lifetime() returns a context that becomes valid before OnStartup runs and is cancelled in phase 4 above. Pass it as the parent context for any long-lived goroutine launched from OnStartup:

func (svc *Service) OnStartup(ctx context.Context) error {
    go svc.refillLoop(svc.Lifetime())   // outlives the OnStartup request, exits on Shutdown
    return nil
}

For goroutines launched in response to a request, prefer svc.Go(ctx, fn) and svc.Parallel(ctx, fns...) - both parent on the lifetime context internally, copy the frame and trace span from the request ctx, and account against the in-flight counter that phases 3 and 5 wait on. Plain go is invisible to that counter and will be killed mid-flight by the process exit if it outruns the drains.

Backward Embedded Resources Internationalization Forward