Operating Securely

This guide walks through setting up a Microbus application for production with security in mind. It covers broker bring-up, credential generation, ingress hardening, access-token issuer configuration, observability for security events, secrets management, and an operations runbook.

The guide is opinionated where the framework has a clear recommended path, and explicit about choices the operator has to make.

Overview

A production Microbus deployment has these moving pieces, and each section of this guide covers one or more of them:

• A NATS broker running in operator mode, with operator and account JWTs preloaded.
• One application binary per deployment unit, listing its microservices via app.Add(...) in main/main.go.
• A per-microservice .creds file in each binary’s working directory, generated by gencreds from the application’s source code.
• An HTTP ingress proxy configured to terminate TLS and enforce port-block rules.
• An access token service and bearer token service deployed somewhere in the application bundle.
• Operator secrets (the operator NKey and the account NKey) held by ops, never deployed.

Prerequisites

• Go at the version pinned in the framework’s go.mod.
• The nsc CLI from NATS for the one-time operator and account setup. Install via brew install nats-io/nats-tools/nsc or download from nats-io/nsc releases.
• A NATS server you can configure. Options include the upstream nats-server binary, a managed Synadia hosted cluster, or self-hosted on Kubernetes.
• A production-suitable secrets store for the operator and account NKeys. A filesystem-only setup works for early-stage deployments. Treat the seeds as you would TLS private keys.

Setting Up the NATS Broker

The broker is the trust root for interservice ACL. It enforces the per-microservice .creds permissions. Setting it up correctly is a prerequisite for everything else in the model.

Step 1: One-Time Operator and Account Setup

You’ll do this once, ever. The operator NKey is the trust root for every microservice identity in every environment you operate.

# Create an isolated nsc workspace.
mkdir -p ~/.microbus-ops && cd ~/.microbus-ops
export NSC_HOME=~/.microbus-ops/nsc
nsc init --name MICROBUS_OPERATOR

# Create the user account that will sign per-microservice user JWTs.
nsc add account --name MICROBUS

# Optional: Allow account-managed JWT lookups via the NATS resolver.
nsc edit account --name MICROBUS --sk generate

What nsc init does: generates the operator NKey, creates a system account (SYS), and produces the operator JWT that the NATS server will need.

What nsc add account does: generates the user account NKey. This is the key gencreds uses to sign per-microservice .creds.

Locate the seeds you’ll need later:

# Operator seed: never share, never deploy.
nsc list keys --operator --show-seeds | grep -A1 MICROBUS_OPERATOR

# Account seed: secret, but you'll feed it to gencreds at deploy time.
nsc list keys --account MICROBUS --show-seeds | grep -A1 MICROBUS

Copy the account seed (the line starting SA...) into a file you’ll guard like a private key, for example ~/.microbus-ops/secrets/microbus-account.nk. This is what gencreds --signing-key consumes.

Generate the account JWT once and keep it for the NATS server:

nsc describe account --name MICROBUS --raw > ~/.microbus-ops/secrets/microbus-account.jwt
nsc describe account --name SYS      --raw > ~/.microbus-ops/secrets/sys-account.jwt

Step 2: Configure Your NATS Server

Operator-mode NATS expects an operator JWT and an account resolver. A minimal nats-server.conf:

listen: 0.0.0.0:4222

operator: "/path/to/MICROBUS_OPERATOR.jwt"
system_account: "<SYS account public NKey, AB...>"

resolver: MEMORY
resolver_preload: {
  "<SYS account public NKey>":      "<contents of sys-account.jwt>",
  "<MICROBUS account public NKey>": "<contents of microbus-account.jwt>"
}

Get the operator JWT path and account public keys via:

nsc describe operator --raw > ~/.microbus-ops/secrets/MICROBUS_OPERATOR.jwt
nsc list accounts --show-keys

Boot the server and confirm it’s healthy:

nats-server -c nats-server.conf
# In another terminal:
nats account info --server nats://localhost:4222 --creds <(nats sys creds emit)

For production, swap MEMORY for the URL resolver so account JWTs can be rotated without restarting the server.

TLS for NATS Connections

For any deployment where NATS traffic crosses a network you do not fully control, terminate the connection over TLS. Configure nats-server.conf with tls { cert_file, key_file, ca_file } and ship the CA certificate to every microservice’s host. The framework picks up the standard system trust store; for a private CA, place ca.pem (or per-microservice {hostname}_ca.pem) alongside the .creds file in CWD, as described in NATS Connection Settings.

Network Isolation

By default, NATS should listen only on a network reachable to the application bundles. A private VPC subnet, an overlay network, or 127.0.0.1 for single-host deployments all work. For typical HTTP-fronted applications, the HTTP ingress is the only externally reachable component.

Some deployments deliberately expose NATS externally, for example to edge clients (browsers via NATS WebSocket, IoT devices, native NATS SDKs). In those cases, terminate TLS at the broker and issue edge clients their own narrow accounts and ACLs, distinct from the per-microservice .creds issued by gencreds. Treat the broker as an internet-facing service.

Connection Limits and Slow-Consumer Policies

NATS has connection-level knobs that complement the framework’s per-microservice ACL enforcement. Configure these per account or per user in nats-server.conf, or per-user inside the JWT permissions:

• max_payload caps the size of a single NATS message. Microbus carries the HTTP envelope plus the body in each message, so set this comfortably above your largest legitimate request or response. The NATS default of 1 MiB is fine for most workloads. A misbehaving caller cannot exceed this cap regardless of its ACL grants.
• max_subscriptions caps the number of active subscriptions per connection. Bound it to the framework’s expected count (typically a few dozen per microservice) to limit memory exposure if a connection is compromised.
• Slow-consumer policy (max_pending, the broker’s slow-consumer disconnect threshold) controls when the broker disconnects a subscriber whose pending queue is growing faster than it can drain. Set the threshold tight enough that a stuck consumer is dropped before it consumes broker memory. The framework treats disconnects as transient errors with reconnect-and-retry.

These are NATS-side knobs, not framework knobs. They apply uniformly whether a connection holds a per-microservice .creds or a shared one. They are operator-side backstops; spoof-immune in the same sense as the per-user ACLs, since the broker enforces them per connection.

Setting Up the HTTP Ingress

The HTTP ingress is the only bridge between external HTTP traffic and the bus. It is the perimeter.

TLS Termination and Edge Concerns

Terminate TLS at the ingress (or at a load balancer in front of it) so internal traffic does not need to renegotiate. Configure the ingress for the standard web-edge concerns: rate limiting, IP allowlists for sensitive paths, and any WAF integration your environment requires.

PROD Deployment Mode

Set MICROBUS_DEPLOYMENT=PROD for production builds. Under PROD:

• The ingress blocks inbound external requests from reaching internal ports (:1 to :1023), except for :80 and :443.
• The framework refuses to accept actor JWTs with alg=none.
• Various hardening and assertion paths in the framework are enabled.

PROD is the strictest deployment mode. LAB is similar but skips a few hardening checks that don’t matter outside production. TESTING bypasses the configurator service and is intended only for go test runs.

Port-Block Rules

The ingress always blocks external traffic from reaching internal ports :666 (trust-root) and :888 (control), regardless of deployment mode. PROD additionally blocks the rest of the internal port range. Make sure any endpoint you intend to expose externally listens on :80 or :443.

Generating Per-Microservice Credentials

For every deployment of every Microbus app, run gencreds against your bundle’s main.go. The output is one .creds file per microservice, each containing a user NKey seed and a user JWT signed by your account NKey.

Basic Invocation

cd /path/to/microbus-app

go run github.com/microbus-io/fabric/cmd/gencreds \
  --bundle main/main.go \
  --signing-key ~/.microbus-ops/secrets/microbus-account.nk \
  --plane prod \
  --out ./deploy/creds/

What this does:

1. Parses main/main.go for app.Add(...) calls and resolves each to a microservice package via go list.
2. For each microservice, scans the source code to derive the exact NATS subjects the microservice publishes to and subscribes from.
3. Generates a fresh user NKey per microservice, signs the derived permissions with your account NKey, and writes ./deploy/creds/<hostname>_nats.creds.

You’ll see one file per microservice:

deploy/creds/
├── access.token.core_nats.creds
├── bearer.token.core_nats.creds
├── configurator.core_nats.creds
├── foreman.core_nats.creds
├── http.egress.core_nats.creds
├── ...
└── yellowpages.example_nats.creds

Plane Substitution

The --plane flag ties the issued .creds to a specific plane. The plane is the isolation prefix on every NATS subject Microbus emits, so two deployments on different planes share no traffic on the bus even when they share the broker. Two common uses:

• Per environment. Distinct planes for prod, staging, dev, etc., when each environment has its own deployment of the same application.
• Per application. Distinct planes for separate applications that share NATS infrastructure but have no business seeing each other’s traffic.

Re-run gencreds once per <environment, application> combination with the matching --plane. The default is microbus, which matches the framework’s local-development default.

Credential Expiration

By default, gencreds does not set an exp claim on the emitted .creds. The broker accepts a leaked .creds indefinitely until its public key is revoked. To time-bound issued credentials, pass --expiration:

go run github.com/microbus-io/fabric/cmd/gencreds \
  --bundle main/main.go \
  --signing-key ~/.microbus-ops/secrets/microbus-account.nk \
  --plane prod \
  --expiration 720h \
  --out ./deploy/creds/

--expiration sets the exp claim on every emitted .creds. Once it elapses, the broker rejects the credentials at CONNECT and on every subsequent publish. Pick a value comfortably longer than your deploy cadence so rolling restarts do not race the expiry.

NKey Rotation

By default, gencreds generates fresh user NKeys on every run. Rotation does not retroactively invalidate older creds (see “Operations” below for the actual revocation mechanism), but it limits exposure prospectively. A leak from one build cannot be used to predict or impersonate the identity of any other build.

If you need stable user public keys (for example, to allowlist them at the NATS server beyond the JWT chain):

go run github.com/microbus-io/fabric/cmd/gencreds \
  --bundle main/main.go \
  --signing-key ~/.microbus-ops/secrets/microbus-account.nk \
  --plane prod \
  --persist-user-nkeys ~/.microbus-ops/secrets/user-nkeys/prod/ \
  --out ./deploy/creds/

gencreds reads existing <hostname>_user.nk files from the persist directory, or writes new ones for microservices without one. The persist directory is itself sensitive; guard it like the account NKey.

Bundle Alternative

If your deployment doesn’t have a single main.go (you’re packaging a custom subset of microservices), use --manifests instead:

go run github.com/microbus-io/fabric/cmd/gencreds \
  --manifests ./coreservices/configurator,./coreservices/httpegress,./examples/hello \
  --signing-key ... \
  --plane prod \
  --out ./deploy/creds/

Running an Application

Each microservice resolves auth artifacts from the current working directory at startup, looking for <hostname>_nats.creds first and falling back to nats.creds. So either:

• Per-microservice identity (recommended). Place every <hostname>_nats.creds in the binary’s working directory.

  cd /path/to/deploy
  cp ~/.microbus-ops/built-bundles/myapp ./myapp
  cp ./creds/*.creds ./
  MICROBUS_NATS=nats://nats.internal:4222 \
  MICROBUS_DEPLOYMENT=PROD \
  MICROBUS_PLANE=prod \
  ./myapp

  Each microservice in the bundle opens its own NATS connection with its own narrow .creds.

• Shared identity (for development or early-stage production). All microservices in the bundle present the same nats.creds. gencreds does not emit a shared .creds, because merging every microservice’s per-endpoint ACL into a single JWT would exceed the NATS 4 KB max_control_line budget for any non-trivial bundle. The shared .creds has to be minted manually using nsc add user, with a deliberately broad allow-list such as <plane>.> (full access on the plane) or <plane>.safe.> (everything except the :666 trust-root tier). Once you have the file, drop it into the binary’s working directory:

  cd /path/to/deploy
  cp shared.creds ./nats.creds
  ./myapp

  The broker cannot tell bundled microservices apart, per-endpoint ACL granularity is gone, and the framework’s per-microservice capability isolation is lost. Suitable when operational simplicity outweighs the security loss.

Each microservice logs its NATS connection at startup:

Connected to NATS  service=foreman.core  url=nats://nats.internal:4222  server=NCYDG3...

Setting Up the Access-Token Issuer

The access token service mints actor JWTs that propagate through the bus. The bearer token service mints longer-lived bearer tokens for use at the edge. Both must be deployed in the application bundle for any production posture that relies on actor claims.

Deployment

Add accesstoken.NewService() and bearertoken.NewService() to one of your application’s main.go files. The framework pins JWKS lookups to these microservices; if either microservice is missing, every actor-JWT verification fails closed.

The recommended posture is to deploy each of these as its own application (a separate OS process). See section “Hardening the trust-root tier” below.

Bearer Token Signing Key

Bearer tokens are long-lived and must verify across replicas and restarts. Unlike the access token service, which generates and rotates ephemeral Ed25519 keys automatically, the bearer token service requires the operator to provision the signing key it uses.

Generate an Ed25519 private key:

openssl genpkey -algorithm ed25519 -out bearer-token-private.pem

Set the contents in your application’s config under the bearer token service’s PrivateKey property:

bearer.token.core:
  PrivateKey: |
    -----BEGIN PRIVATE KEY-----
    ...
    -----END PRIVATE KEY-----

The property accepts PEM or raw base64. Hold the key in a secrets manager and load it through the production config pipeline; never commit it to source. The bearer token service refuses to mint until PrivateKey is configured.

For zero-downtime rotation, set AltPrivateKey to the previous key when promoting a new one to PrivateKey. The bearer token service publishes both public keys via its :888/jwks endpoint, so verifiers continue to accept tokens signed with either key while in-flight tokens from the old key drain naturally.

Replacing the Standard Issuers

Token issuers are pinned to the access token service and bearer token service hostnames. The framework will not accept tokens from any other issuer. If you need to customize how tokens are minted, for example to source claims from a different identity backend or apply a custom TTL policy, build a replacement microservice that satisfies the same interface and registers under the same hostname, then add it to the application instead of the standard implementation. Verifying microservices keep pinning JWKS to the same hostname and do not need to know which implementation is behind it.

TTL Configuration

Actor JWTs are deliberately short-lived. The shorter the TTL, the smaller the replay window if a token leaks.

For access tokens, the lifetime is derived from the issuing request’s remaining time budget: an actor JWT minted for a request with a 10-second budget expires in 10 seconds. The access token service exposes two configuration properties as guardrails. DefaultTokenLifetime is the lifetime used when the request has no time budget (default 20s). MaxTokenLifetime caps the lifetime regardless of the request’s budget (default 15m). Both are bounded to a hard ceiling of 15 minutes.

Bearer tokens have a single direct TTL: AuthTokenTTL on the bearer token service (default 720 hours, 30 days). Bearer tokens live only at the edge and are refreshed less frequently than access tokens.

JWKS Reachability

Every microservice in the application must be able to fetch JWKS from the access token service and the bearer token service to verify actor JWTs. Verify reachability after deployment by inspecting the application’s startup logs and confirming that the first verification of an actor JWT succeeds.

Configuring Authentication

Actor Identity Propagation

External HTTP requests typically arrive at the HTTP ingress with a bearer token. An ingress middleware exchanges that bearer token for a short-lived actor JWT minted by the access token service, and stamps the JWT into the Microbus-Actor header on the inbound call to the bus. From there, every downstream microservice-to-microservice call propagates the same actor JWT automatically, so any microservice in the chain can authorize against the user’s claims. Application code does not have to copy headers manually.

Per-Endpoint requiredClaims

Endpoints declare authorization requirements via requiredClaims boolean expressions in their manifest.yaml. Examples:

functions:
  AdminAction:
    requiredClaims: roles.admin
  PrivilegedAction:
    requiredClaims: level>5 && !guest
  CorpOnly:
    requiredClaims: idp=~"corp\\.example\\.com$"

The framework verifies the JWT signature (via JWKS pinning), evaluates the requiredClaims expression, and rejects the request if it evaluates false.

External IDP Integration

Tokens signed by external IDPs (Auth0, Okta, Google, enterprise SSO) cannot be consumed directly by Microbus microservices. Their iss is not in the pinned list, so verification fails.

The integration pattern is the wrapper microservice. The ingress, or a dedicated IDP-bridge microservice, validates the external token against its IDP and calls the access token service’s Mint endpoint to obtain an internal access token with claims derived from the external assertion. Downstream microservices see only the internal token.

Hardening the Trust-Root Tier

Port :666 is the trust-root tier of Microbus’s security model. Compromise of any :666 endpoint undermines the framework’s security guarantees. Deploy with extra care.

Isolate Trust-Root Microservices

The recommended posture is to deploy each trust-root microservice as its own application, in its own OS process, with its own working directory and nats.creds. The two foundational trust-root microservices in any Microbus deployment are:

• The access token service, which mints actor JWTs.
• The bearer token service, which mints bearer tokens.

Apply the same isolation posture to any other microservice that exposes :666 endpoints, such as the optional shell service or any application-defined privileged-write microservice.

A compromise in any other application cannot reach memory inside an isolated trust-root microservice. A compromise of the trust-root microservice itself is bounded to its own permission set.

Audit :666 Grants

A microservice gets a :666 PUB permission only if its source code calls a :666 endpoint. To enumerate every active trust-root grant for a microservice:

grep "\.danger\." ./deploy/creds/<hostname>_nats.creds

For an entire deployment:

grep -l "\.danger\." ./deploy/creds/*.creds

This lists every microservice whose .creds contains a :666 PUB rule.

Review PRs That Introduce a :666 Call Site

A PR that adds a call site like accesstokenapi.NewClient(svc).Mint(...) from a microservice that did not previously have that capability ships a corresponding PUB <plane>.danger.666.<src>.<dest>.*.<method>.<path> permission in that microservice’s next .creds. Review such PRs with the same scrutiny as new infrastructure access.

Bundle Composition Decisions

The same set of microservices can be deployed as one large bundle, several smaller bundles, or one application per microservice. The choice has security and performance implications.

• Bundle aggressively for cost and latency. Fewer processes, lower memory and connection overhead, easier deploys. Co-located microservices share an address space and dispatch through the in-process short-circuit, skipping the NATS round-trip.
• Split aggressively for isolation. Every cross-microservice call traverses NATS; every microservice has its own connection and .creds; every compromise is bounded to a single OS process.
• Split where it matters. Bundle most microservices for cost and latency, but isolate the ones whose compromise has outsized blast radius: trust-root microservices, microservices with broad ACL surface (such as the HTTP ingress proxy and Foreman), microservices holding sensitive data, and microservices subject to compliance audit.

Security and latency can pull in opposite directions. The clearest example is the access token service: the ingress mints an actor JWT for every inbound request, so the ingress-to-issuer round-trip sits on the hot path of every request. Bundling the ingress with the access and bearer token services keeps those mints in-process and removes the NATS round-trip, at the cost of the isolation benefit. The right call depends on the deployment’s latency budget and threat model.

Plane Discipline

The <plane> segment of every NATS subject isolates one deployment from another on the same NATS cluster. Plane discipline is what keeps prod traffic from leaking into staging, and what keeps two unrelated applications sharing the broker from seeing each other’s traffic.

• One plane per <environment, application> combination. Pick a short alphanumeric string.
• Pass the plane to gencreds via --plane <name> at credential generation time.
• Set MICROBUS_PLANE=<name> in the runtime environment.
• The two must match. A drift means generated rules cover one plane while the runtime publishes to another, and every interservice call silently fails the broker’s ACL check.
• Plane wildcards in ACLs (such as *.safe.*.<src>.<dest>.>) are an anti-pattern. They break the isolation the plane segment exists to provide.

MICROBUS_PLANE is read at startup. Restart the binary after changing it.

Observability for Security Events

Several signals are worth monitoring in production.

Logs

Microbus logs auth failures, ACL rejections, and actor-JWT verification failures at WARN or ERROR level. Pipe these logs into your log aggregator and alert on:

• Repeated Permissions Violation errors from NATS, which indicate a microservice is attempting subjects outside its allow-list.
• Repeated JWT verification failures, which indicate forged or stale actor tokens.
• Unexpected CONNECT failures, which indicate broker-side configuration drift.

Metrics

Microbus’s built-in metrics include per-endpoint request counts and latencies. Filter by source hostname (a label is attached automatically) to spot anomalous caller patterns. A microservice suddenly making calls it didn’t make yesterday is worth investigating, even if those calls succeed.

Traces

Every cross-microservice call participates in a distributed trace. The verified caller identity is included as a trace attribute. Use this to audit the full call graph of a suspicious request.

Secrets Management

A short list of what’s secret and how to handle it.

Verification

After deploying, verify the security posture is what you expect.

Decode a Microservice’s .creds

awk '/-----BEGIN NATS USER JWT-----/{flag=1;next}/-----END/{flag=0}flag' \
  ./foreman.core_nats.creds | nsc tool decode

Look for the nats.pub.allow and nats.sub.allow arrays. Every entry is one PUB or SUB pattern the broker will allow this microservice to use. If you see a subject the microservice should not be able to publish or subscribe to, audit the source code that produced the rule.

Confirm Broker-Side Enforcement

nats context add --server nats://nats.internal:4222 --creds ./<hostname>_nats.creds prod-svc
nats account info --context prod-svc

If the connection succeeds and account info returns, the broker accepted the JWT chain.

Verify Perimeter Blocks

From an external network, confirm that direct connections to the NATS real port (:4222) and any internal Microbus port (:444, :888, etc.) are refused. Only internal Microbus ports :80 and :443 should be reachable, and only via the HTTP ingress.

Operations Runbook

Rotating User NKeys

Re-run gencreds and redeploy. By default the user NKeys are fresh, so the previous deployment’s .creds files become orphaned. Their user public keys are not pre-registered with the broker, but the broker accepts any .creds signed by the account, so the previous .creds continue to work until they expire or are revoked.

To time-bound issued creds, pass --expiration (see “Generating per-microservice credentials” above). For immediate active revocation, use the account’s revocation list (next).

Adding a New Microservice

A new microservice added to main.go needs nothing operator-side beyond re-running gencreds. The new <newservice>_nats.creds file appears in the output and gets shipped with the next deployment.

Granting a Microservice :666 Access

Trust-root access is detected automatically. A new call site in code (for example, accesstokenapi.NewClient(svc).Mint(...)) becomes a PUB <plane>.danger.666.<self>.access_token_core.*.POST.mint permission in that microservice’s next .creds.

For PR review, look for new nats.pub.allow entries containing .danger. after running gencreds against the proposed code.

Revoking a Leaked .creds

Three paths, in increasing order of disruption:

1. Re-run gencreds, redeploy. The new deployment uses fresh user NKeys; the old .creds become useless once the deployment is rolled out, if you used --expiration. Without an exp, the leaked creds remain valid until path 2 or 3 happens.

2. Add a revocation in nsc for the compromised user public key. This invalidates that specific .creds immediately, even if old binaries are still running. Useful when you cannot redeploy fast enough.

   nsc revocations add-user --account MICROBUS --user <USER_PUBKEY> --at $(date +%s)
   nsc push --account MICROBUS

3. Rotate the account NKey. Invalidates every .creds signed by that account at once, not just the leaked one. Overkill for a single leak, but the right escalation when broader compromise is suspected (multiple suspected leaks, the account NKey itself may be exposed, the operator’s CD pipeline may be compromised). See the next section for the full procedure.

Compromised Account NKey

A compromised account NKey is a serious incident. The attacker can mint .creds for any microservice identity in any environment that trusts that account.

Recovery:

1. Generate a new account NKey: nsc edit account --name MICROBUS --rotate-keys. Push the new account JWT to the broker’s resolver.
2. Re-run gencreds with the new account NKey to issue fresh .creds for every microservice.
3. Redeploy every application bundle.
4. Add the old account public key to the operator’s deny list, or rebuild from a fresh operator if compromise of the operator NKey is also suspected.

Switching Environments

Each environment gets its own --plane value and its own MICROBUS_PLANE env var. Run gencreds once per environment with the appropriate plane string.

If you also rotate account NKeys per environment (highly recommended), maintain one <env>-account.nk per environment and pass the matching one as --signing-key.

Pre-Production Checklist

Before a deployment goes into production, confirm each item:

• NATS broker is in operator mode with operator and account JWTs preloaded.
• NATS port is not externally reachable.
• HTTP ingress terminates TLS and is the only externally reachable component.
• MICROBUS_DEPLOYMENT=PROD is set in the runtime environment.
• MICROBUS_PLANE matches the --plane used to generate the .creds.
• Per-microservice <hostname>_nats.creds files are present in each binary’s working directory.
• The access token service and bearer token service are deployed and reachable.
• Foundational trust-root microservices (the access token service and the bearer token service) are isolated in their own applications, along with any optional :666 microservice such as the shell service if deployed.
• Account NKey is held in a vault, not on a developer machine or in CI logs.
• Operator NKey is held offline or in an HSM.
• *.creds, *.nk, *.local.yaml are all in .gitignore.
• Logs and metrics are aggregated and security alerts are wired up.
• An incident-response runbook exists for leaked creds and compromised account NKey.

Local Development

The simplest local setup needs no broker at all. When all microservices in your application live in one process, they dispatch through the in-process short-circuit transport and NATS is never involved. No broker, no .creds, no ACL enforcement. This is the default LOCAL deployment mode.

If your local setup does run a broker (typically a vanilla unauthenticated nats-server on 127.0.0.1:4222), the application connects to it in the clear. There are no ACLs in effect, since gencreds is a deploy-time tool that’s not run during development. Set MICROBUS_NATS=nats://127.0.0.1:4222 (or rely on the default) and don’t put any .creds in CWD.

To exercise the full operator-mode pipeline against a local broker:

1. Run a local nats-server in operator mode (use the same Step 2 config above, or start with the simpler nats-server -auth_callout-style setup for prototyping).
2. Generate .creds for your local app via gencreds --plane local --signing-key ./local-account.nk.
3. Place the resulting .creds in your binary’s CWD.

Troubleshooting

“Authorization Violation” on CONNECT

The microservice’s .creds is signed by an account the server does not trust. Confirm the account JWT is preloaded in the server’s resolver (see “Setting up the NATS broker”) and that the operator JWT in the server config matches the operator that signed the account.

“Permissions Violation” on a Publish

The microservice’s .creds does not include the subject pattern the microservice tried to publish on. Two common causes:

1. The microservice’s source code added a new call but gencreds was not re-run before deploy. Regenerate and redeploy.
2. The plane mismatch: gencreds --plane prod was used, but the microservice is publishing under a different plane (MICROBUS_PLANE env var or default). Make them match.

Decode the current .creds (see “Verification” above) and check what subjects are actually allowed.

MICROBUS_PLANE Change Not Picked Up

MICROBUS_PLANE is read at startup. Restart the binary after changing it.

Actor JWT Verification Fails

Symptoms: every cross-microservice call is rejected with a JWT signature error. Common causes:

• The access token service is not deployed in the application bundle or is unreachable from other microservices.
• The token’s iss does not match the pinned hostname for its type.
• The signing key has been rotated but the JWKS endpoint has not been refreshed. Restart the verifying microservices to clear cached keys.

.creds Files Committed to Git

They contain a private NKey seed. Treat them as you would treat a TLS private key. Never commit, never share, rotate if leaked. The gencreds output directory should be .gitignore’d.