LLM Integration

The LLM core microservice bridges LLM tool-calling protocols with Microbus endpoint invocations. Callers supply a list of canonical endpoint URLs as tools, an LLM provider hostname, and a model identifier. The service drives the tool-calling loop - invoking each selected endpoint over the bus and feeding results back to the LLM until it produces a final response. It supports Claude, ChatGPT, Gemini, and LiteLLM providers out of the box, plus any custom microservice that implements the Turn contract. The LiteLLM provider fronts a LiteLLM proxy over the OpenAI Responses API wire format, reaching any model the proxy routes to.

Configuration

Provider-specific settings (the endpoint URL and APIKey) live on the provider microservice. The endpoint URL config is named per provider - MessagesURL on claudellm, ResponsesURL on chatgptllm and litellm, and ModelsURL on geminillm - and each defaults to the provider’s public API. The APIKey is a secret and belongs in config.local.yaml:

# config.local.yaml (git-ignored)
claude.llm.core:
  APIKey: sk-ant-your-key-here
chatgpt.llm.core:
  APIKey: sk-your-openai-key
gemini.llm.core:
  APIKey: your-gemini-key

Provider and model are chosen per call - there is no global “active provider” or “active model” config. A caller may instead pass llmapi.ProviderAny (or an empty string) together with a capability-tier alias (llmapi.ModelFast, llmapi.ModelDefault, or llmapi.ModelSmart) and let llm.core resolve a configured provider at runtime. Choosing the tier explicitly keeps cost visible at the call site, which matters because models can differ in cost by 100x or more.

llm.core itself has a single optional config:

# config.yaml
llm.core:
  MaxToolRounds: 10           # max tool call round-trips per Chat invocation

Single Chat Request

The simplest usage is a synchronous Chat call. Pass provider, model, the conversation items, and (optionally) tools and options:

import (
    "github.com/microbus-io/fabric/coreservices/llm/llmapi"
)

// Simple text conversation, no tools
items := []llmapi.Item{
    llmapi.NewMessage("user", "What is the capital of France?").AsItem(),
}
itemsOut, usage, resolvedProvider, err := llmapi.NewClient(svc).Chat(
    ctx,
    llmapi.ProviderAny,     // resolve a configured provider at runtime
    llmapi.ModelDefault,    // capability-tier alias
    items,
    nil,                    // no tools
    nil,                    // no options
)
// itemsOut is the full conversation, ending with the assistant's reply.
// usage carries aggregated token counts across all turns.
// resolvedProvider reports which provider actually served the request.

A model is named in one of three ways: a capability-tier alias (llmapi.ModelFast, llmapi.ModelDefault, llmapi.ModelSmart), a provider-family alias (such as opus), or a concrete vendor-prefixed model-name string. Combined with llmapi.ProviderAny, a tier alias lets the same code run against whichever brand is configured, so a single API key of any one brand (Claude, ChatGPT, or Gemini) is enough.

With Tools

Tools are passed as a []string of canonical Microbus endpoint URLs. Each Def value in a downstream service’s *api package has a URL() helper that returns its canonical form:

import (
    "github.com/microbus-io/fabric/coreservices/llm/llmapi"
    "github.com/mycompany/myproject/calculator/calculatorapi"
    "github.com/mycompany/myproject/weather/weatherapi"
)

items := []llmapi.Item{
    llmapi.NewMessage("user", "What is 42 * 17, and what's the weather in Paris?").AsItem(),
}
toolURLs := []string{
    calculatorapi.Arithmetic.URL(),
    weatherapi.Forecast.URL(),
}
itemsOut, usage, resolvedProvider, err := llmapi.NewClient(svc).Chat(
    ctx,
    llmapi.ProviderAny,
    llmapi.ModelDefault,
    items,
    toolURLs,
    nil,
)

The Chat endpoint runs the tool-calling loop internally: it fetches each host’s OpenAPI document, reflects the matching operation’s request-body schema into a JSON Schema, and exposes the tool to the LLM. If the LLM requests a tool call, the service invokes the Microbus endpoint over the bus, feeds the result back to the LLM, and repeats until the LLM produces a final text response or MaxToolRounds is reached (configured on llm.core, default 10; can be overridden per call via ChatOptions.MaxToolRounds).

Only functional, web, and workflow endpoints are exposed - tasks and outbound events are silently skipped. When two endpoints share the same operation name, the first keeps the bare name and subsequent ones get _2, _3, … suffixes in argument order, so URLs from multiple services can be concatenated without collision.

Chat returns itemsOut (the full conversation, including the original items and every item the LLM produced), usage (aggregated token counts), and resolvedProvider (the provider that served the request). To continue the conversation, append a new user item to itemsOut and call Chat again; pass resolvedProvider back as the provider to pin the same one across turns.

ChatOptions

opts := &llmapi.ChatOptions{
    MaxToolRounds: 5,          // overrides the MaxToolRounds config for this call
    MaxTokens:     1024,       // caps response length per turn
    Temperature:   0.2,        // sampling randomness
    Effort:        "medium",   // reasoning-effort level forwarded to the provider
}
itemsOut, usage, resolvedProvider, err := llmapi.NewClient(svc).Chat(ctx, provider, model, items, toolURLs, opts)

Switching Providers

To pin a specific provider, pass its hostname and a model for that provider. The calling code does not need any other change:

import "github.com/microbus-io/fabric/coreservices/chatgptllm/chatgptllmapi"

// Same call, pinned to ChatGPT
itemsOut, usage, resolvedProvider, err := llmapi.NewClient(svc).Chat(
    ctx,
    chatgptllmapi.Hostname,   // "chatgpt.llm.core"
    llmapi.ModelDefault,
    items,
    toolURLs,
    nil,
)

Alternatively, leave the provider open with llmapi.ProviderAny plus a tier alias and let llm.core pick whichever brand is configured - the same call site then works against any provider without change.

ChatLoop Workflow

For conversations that may involve many tool rounds or need durability, use the ChatLoop workflow. It performs the same logic as Chat but as a series of durable workflow steps orchestrated by the Foreman. Inputs match Chat; outputs are items and usage.

The graph (source: CHATLOOP.mmd):

%%{init: {'themeVariables': {'lineColor': '#32a7c1', 'edgeLabelBackground': '#e5f4f3', 'textColor': '#434343', 'titleColor': '#32a7c1', 'clusterTextColor': '#32a7c1'}}}%%
graph LR
    classDef task fill:#32a7c1,color:#f4f2ef,stroke:#32a7c1
    classDef sub fill:#ed2e92,color:#f4f2ef,stroke:#ed2e92
    classDef term fill:#e5f4f3,color:#434343,stroke:#32a7c1

    _title{{"chat-loop"}}:::term --> _start
    _start(( )):::term --> t0
    t0["initChat"]:::task
    t1["firstLLM"]:::task
    t2["nextLLM"]:::task
    t3["processResponse"]:::task
    subgraph fo_t3 ["for each in pendingToolCalls"]
        direction LR
        t4["executeTool"]:::task
    end
    t0 --> t1
    t1 --> t3
    t3 -->|"goto"| _end(( )):::term
    t3 -->|"fan out"| t4
    t4 -->|"fan in"| t2
    t2 --> t3
    style fo_t3 fill:#32a7c1,fill-opacity:0.15,stroke:#434343,stroke-dasharray:4 2

initChat normalizes the inputs; firstLLM invokes the provider’s Turn once; processResponse looks at what the LLM returned. If there are no tool calls, it gotos the end and the workflow completes. If there are, it fans out one executeTool branch per pending call (the forEach over pendingToolCalls), fans the results back in at nextLLM, and loops back to processResponse. Every state transition is durably persisted by the Foreman, so a crash mid-loop resumes from the last completed step on restart.

Single Turn

import (
    "github.com/microbus-io/fabric/coreservices/foreman/foremanapi"
    "github.com/microbus-io/fabric/coreservices/llm/llmapi"
)

outcome, err := foremanapi.NewClient(svc).Run(ctx, llmapi.ChatLoop.URL(), map[string]any{
    "provider": llmapi.ProviderAny,
    "model":    llmapi.ModelDefault,
    "items": []llmapi.Item{
        llmapi.NewMessage("user", "What's the weather in San Francisco?").AsItem(),
    },
    "toolURLs": []string{weatherapi.Forecast.URL()},
}, nil)
// outcome.State["items"] contains the full conversation
// outcome.State["usage"] contains the aggregated llmapi.Usage

The trailing nil is the optional *workflow.FlowOptions. Pass &workflow.FlowOptions{Priority: 1} (or a fairness key) to schedule this conversation ahead of background work, following the rules in Priority and Fairness.

Multi-Turn via Continue

ChatLoop returns the full items conversation as its output. To carry a conversation forward, Continue starts a new flow in the same thread from the completed flow’s final state and adds another user turn:

flowID, _ := foremanapi.NewClient(svc).Create(ctx, llmapi.ChatLoop.URL(), map[string]any{
    "provider": llmapi.ProviderAny,
    "model":    llmapi.ModelDefault,
    "items":    []llmapi.Item{llmapi.NewMessage("user", "What's the weather in San Francisco?").AsItem()},
    "toolURLs": []string{weatherapi.Forecast.URL()},
}, nil)
status, state, _ := foremanapi.NewClient(svc).Await(ctx, flowID)
// Present state["items"] to the user...

// Second turn - Continue finds the latest completed flow in the thread and adds the new user turn.
// flowID doubles as the threadKey (any flowKey in the thread works).
newFlowID, _ := foremanapi.NewClient(svc).Continue(ctx, flowID, map[string]any{
    "items": []llmapi.Item{llmapi.NewMessage("user", "What about tomorrow?").AsItem()},
})
status, state, _ = foremanapi.NewClient(svc).Await(ctx, newFlowID)

Each Continue creates a new flow in the same thread, starting from the final state of the latest completed flow and continuing the conversation with the new user turn. The caller can pass any flowKey from the thread - Continue automatically finds the latest one.

When to Use Chat vs ChatLoop

Use Chat for simple, quick interactions. Use ChatLoop when the conversation may involve many tool rounds, when you need durability against failures, when you need to pause for a human in the loop, or when you want the Foreman’s debugging and continuation capabilities.

Token Usage and Metrics

Every Chat call returns an llmapi.Usage aggregating token consumption across all turns:

type Usage struct {
    InputTokens      int    // prompt tokens charged
    OutputTokens     int    // completion tokens generated
    ReasoningTokens  int    // subset of OutputTokens spent on internal reasoning (0 when the provider exposes no breakdown)
    CacheReadTokens  int    // tokens served from the provider's prompt cache
    CacheWriteTokens int    // tokens written to the provider's prompt cache
    Model            string // provider's model identifier that produced this completion
    Turns            int    // number of LLM turns aggregated
}

The claudellm provider sets two cache_control breakpoints on requests so Anthropic’s prompt cache can be reused across turns. Cached input is reflected in CacheReadTokens / CacheWriteTokens.

Per-turn token consumption is also emitted as the microbus_llm_tokens_total counter metric, labeled by provider, model, and direction (input, output, cacheRead, cacheWrite). The bundled LLM Grafana dashboard charts tokens by direction/provider/model and the cache hit ratio.

Testing with Mocks

Mock the LLM service in tests to avoid needing a real API key:

llmMock := llm.NewMock()
llmMock.MockChat(func(ctx context.Context, provider string, model string, items []llmapi.Item, toolURLs []string, options *llmapi.ChatOptions) (itemsOut []llmapi.Item, usage llmapi.Usage, resolvedProvider string, err error) {
    return []llmapi.Item{llmapi.NewMessage("assistant", "Mocked response").AsItem()}, llmapi.Usage{Turns: 1}, provider, nil
})

app := application.New()
app.Add(svc, llmMock, tester)
app.RunInTest(t)

To test against the real llm.core without calling a live LLM API, mock the provider service (claudellm, chatgptllm, or geminillm) at the Turn boundary:

claudeMock := claudellm.NewMock()
claudeMock.MockTurn(func(ctx context.Context, model string, items []llmapi.Item, tools []llmapi.Tool, options *llmapi.TurnOptions) (itemsOut []llmapi.Item, stopReason string, usage llmapi.Usage, err error) {
    return []llmapi.Item{llmapi.NewMessage("assistant", "Hello from mock!").AsItem()}, llmapi.StopReasonEndTurn, llmapi.Usage{Model: model, Turns: 1}, nil
})

app := application.New()
app.Add(svc, llm.NewService(), claudeMock, tester)
app.RunInTest(t)

Mocking at the provider boundary exercises the full tool-calling loop, schema resolution, and bus dispatch in llm.core while keeping the test offline.