llm

The LLM microservice bridges LLM tool-calling protocols with Microbus endpoint invocations. Callers pass a conversation, a provider, a model, and a list of endpoint URLs they want to expose as tools. The service drives the tool-calling loop, dispatches tool calls over the bus, and returns the completed conversation along with token usage.

Key capabilities:

• Caller-selected provider and model - a Chat call either names a concrete provider hostname (e.g. claude.llm.core) and model, or passes llmapi.ProviderAny (or empty) together with a capability-tier alias (llmapi.ModelFast, llmapi.ModelDefault, llmapi.ModelSmart) and lets llm.core resolve a configured provider at runtime. There is no ProviderHostname config and providers do not carry a Model config. Hiding the model behind operator config is dangerous - a smart-tier model is far costlier than a fast-tier one - so the tier or model is always visible at the call site.
• OpenAPI-derived tool schemas - callers identify tools by their canonical Microbus URL. At chat time the LLM service fetches each host’s :888/openapi.json document in parallel (the connector’s built-in handler) and reflects the matching operation’s request-body schema into a JSON Schema for the LLM. Authorization flows through automatically: the OpenAPI handler filters by the caller’s actor claims, so the LLM only sees tools the actor is authorized to invoke.
• Tool execution over the bus - invokes Microbus endpoints directly when the LLM requests a tool call, with security context propagated through the request frame.
• Token usage tracking - each turn returns an llmapi.Usage carrying input, output, cache-read and cache-write tokens plus the resolved model identifier. Usage.ReasoningTokens reports the subset of output tokens spent on internal reasoning (zero when the provider exposes no breakdown). Chat aggregates per-turn usage and reports the totals. The microbus_llm_tokens_total counter (labeled by provider, model, direction) feeds the LLM Grafana dashboard.
• Normalized stop reasons - every provider’s Turn returns a stopReason from a common vocabulary (llmapi.StopReasonEndTurn, StopReasonToolUse, StopReasonMaxTokens, StopReasonStopSequence, StopReasonRefusal, StopReasonPauseTurn, StopReasonUnknown), so the chat loop can branch on why a turn ended regardless of provider — continue on tool_use, return on a completion, and fail loud on a truncated max_tokens rather than ship a partial response.
• Anthropic prompt caching - the claudellm provider sets two cache_control breakpoints on requests so Anthropic’s prompt cache can be reused across turns. Cached input is reflected in Usage.CacheReadTokens/CacheWriteTokens.
• Multi-turn workflow - the built-in ChatLoop workflow orchestrates conversations that exceed a single request’s time budget, with durability, human-in-the-loop support via flow.Interrupt(), and natural continuation via foremanapi.Continue.

Chat

The Chat functional endpoint sends messages to an LLM with optional tools and returns the updated conversation along with aggregated token usage. It handles the tool-calling loop internally, up to MaxToolRounds rounds:

import (
    "github.com/microbus-io/fabric/coreservices/llm/llmapi"
)

items := []llmapi.Item{llmapi.NewMessage("user", "What is 3 + 5?").AsItem()}
toolURLs := []string{calculatorapi.Arithmetic.URL()}
itemsOut, usage, resolvedProvider, err := llmapi.NewClient(svc).Chat(
    ctx,
    llmapi.ProviderAny,           // provider
    llmapi.ModelDefault,          // model
    items,
    toolURLs,
    nil,                          // *llmapi.ChatOptions, optional
)

provider is the hostname of a provider microservice (claude.llm.core, chatgpt.llm.core, gemini.llm.core, lite.llm.core, or any custom provider that implements the Turn contract), or llmapi.ProviderAny (or empty) to let llm.core resolve a configured provider at runtime. lite.llm.core is the LiteLLM provider — it implements Turn against a LiteLLM proxy using the OpenAI Responses API wire format, so any model the proxy routes to is reachable without a per-vendor provider microservice. A model is named one of three ways: a capability-tier alias (llmapi.ModelFast, llmapi.ModelDefault, llmapi.ModelSmart), a provider-family alias (e.g. opus), or a concrete vendor-prefixed model-name string. A single API key of any one brand is enough to run these examples.

Each entry in toolURLs is the URL of a downstream microservice’s endpoint. Only function, web, and workflow endpoints can be exposed as tools - tasks and outbound events are silently skipped by the connector’s OpenAPI handler. When two endpoints share the same operation name, the first keeps the bare name and subsequent ones get _2, _3, … suffixes in argument order.

itemsOut contains the full conversation as an ordered []llmapi.Item log, including the new items produced by the LLM. usage is the aggregated llmapi.Usage across all turns. resolvedProvider reports which provider actually served the request; pass it back as the provider on a follow-up call to pin the same one.

ChatOptions

Pass *llmapi.ChatOptions to override defaults for a single call:

opts := &llmapi.ChatOptions{
    MaxToolRounds: 5,        // overrides the MaxToolRounds config for this call
    MaxTokens:     1024,     // caps response length per turn
    Temperature:   0.2,      // sampling randomness
    Effort:        "medium", // reasoning-effort level forwarded to the provider
}

MaxToolRounds remains as a service-level config (operational guardrail); ChatOptions.MaxToolRounds is an optional per-call override.

ChatLoop Workflow

For conversations that may require many tool rounds or human interaction, the ChatLoop workflow orchestrates the same flow across multiple durable steps. Inputs are provider, model, items, toolURLs, options; outputs are items and usage:

flowID, _ := foremanapi.NewClient(svc).Create(ctx, llmapi.ChatLoop.URL(), map[string]any{
    "provider": llmapi.ProviderAny,
    "model":    llmapi.ModelDefault,
    "items":    items,
    "toolURLs": toolURLs,
}, nil)
status, state, _ := foremanapi.NewClient(svc).Await(ctx, flowID)

ChatLoop returns the full items conversation, and per-turn usage accumulates into the usage state field. The flowID returned by Create doubles as a threadKey - pass it (or any flowKey in the thread) to foremanapi.Continue, which adds another user turn to the same thread and carries the conversation forward:

newFlowID, _ := foremanapi.NewClient(svc).Continue(ctx, flowID, map[string]any{
    "items": []llmapi.Item{llmapi.NewMessage("user", "Follow-up question").AsItem()},
})

Configuration

The llm.core service holds only one config. Provider-specific settings (the provider’s endpoint URL config, APIKey) live on the provider microservice.

llm.core

Provider Service (claudellm, chatgptllm, geminillm, litellm)

Each provider names its endpoint URL config after the API it speaks. claudellm calls the Anthropic Messages API; chatgptllm and litellm both speak the OpenAI Responses API; geminillm calls the native Gemini API.

The APIKey is a secret and should be set in config.local.yaml, scoped by the provider’s hostname:

claude.llm.core:
  APIKey: sk-ant-your-key-here
chatgpt.llm.core:
  APIKey: sk-your-openai-key
gemini.llm.core:
  APIKey: your-gemini-key

To use a different provider for a call, simply pass that provider’s hostname and model. There is no global “active provider” config to flip.

Mocking

To mock the Chat endpoint of llm.core:

llmMock := llm.NewMock()
llmMock.MockChat(func(ctx context.Context, provider string, model string, items []llmapi.Item, toolURLs []string, options *llmapi.ChatOptions) (itemsOut []llmapi.Item, usage llmapi.Usage, resolvedProvider string, err error) {
    return []llmapi.Item{llmapi.NewMessage("assistant", "Mocked response").AsItem()}, llmapi.Usage{Turns: 1}, provider, nil
})

To exercise the real llm.core against a mocked provider, mock the provider’s Turn instead:

claudeMock := claudellm.NewMock()
claudeMock.MockTurn(func(ctx context.Context, model string, items []llmapi.Item, tools []llmapi.Tool, options *llmapi.TurnOptions) (itemsOut []llmapi.Item, stopReason string, usage llmapi.Usage, err error) {
    return []llmapi.Item{llmapi.NewMessage("assistant", "Hello from mock!").AsItem()}, llmapi.StopReasonEndTurn, llmapi.Usage{Model: model, Turns: 1}, nil
})

Mocking at the provider boundary exercises the full tool-calling loop, schema resolution, and bus dispatch in llm.core while keeping the test offline.

Turn on llm.core

The Turn endpoint is part of the contract that provider microservices implement. A turn returns its response as an ordered []llmapi.Item (a message item plus any tool-call items), a normalized stopReason, and usage:

func (svc *Service) Turn(ctx context.Context, model string, items []llmapi.Item, tools []llmapi.Tool, options *llmapi.TurnOptions) (itemsOut []llmapi.Item, stopReason string, usage llmapi.Usage, err error)

Each provider maps its native finish reason into the llmapi.StopReason* vocabulary; a custom provider must return one of those constants. Pure consumers of Chat need no change — the stop reason is handled inside the chat loop. llm.core itself is not a provider - calling its Turn endpoint returns 501. Use llmapi.NewClient(svc).ForHost(<providerHost>).Turn(...) to invoke a specific provider directly, or use Chat for the conversation loop.

The endpoint stub is registered (rather than removed) because llmapi.Turn.URL() is referenced as the canonical form of the contract.

The Chat endpoint listens on internal Microbus port :444 rather than :443, and the task and workflow endpoints use port :428. These ports are not accessible from outside the bus via the HTTP ingress proxy.

See Also

• Building an LLM Workflow — end-to-end howto: add llm.NewService() + a provider to your app, make a first Chat call, author an endpoint as a tool, then promote to a durable ChatLoop step inside a parent workflow.
• Chatbox example — a worked custom provider implementing the Turn contract, plus an interactive demo wired into this service’s Chat endpoint.
• LLM integration — the call-site API surface (Chat, ChatLoop, ChatOptions, provider switching, mocking).
• LLM tooling — the conceptual model: OpenAPI as the tool description, internal vs. external delivery paths, end-to-end authorization.