Process a batch of independent documents¶

A ChatClient is a conversation by default. Every Chat, Ask and StreamChat call appends the exchange to the client's history, and the next call re-sends all of it.

That is right for a conversation and wrong for batch work: classifying, extracting, summarising or judging a set of documents that have nothing to do with one another. Set Config.Stateless for that.

The trap¶

Holding one client across N independent calls looks like the obvious way to avoid rebuilding a client per document. It is also the expensive way:

// Don't do this for independent documents.
client, _ := chat.New(ctx, chat.Settings{Config: cfg})

for _, doc := range docs {
    var out Extraction
    _ = client.Ask(ctx, prompt(doc), &out) // carries every previous doc
}

Call 1 sends document 1. Call 2 sends documents 1 and 2 plus the first answer. Call 18 sends all seventeen documents before it. Eighteen documents cost 18×19/2 = 171 document-equivalents rather than 18, a 9.5× multiplier, and it grows with the square of the batch.

Nothing warns you. The calls succeed, the answers look plausible, and the cost shows up on a bill. Two things make it worse than it sounds:

• Rate limits trip for the wrong-looking reason. Per-minute input-token limits are breached by growth within a run, which reads like a pacing problem and invites a between-run cooldown that cannot help.
• The answers are contaminated. Call N is answered in the light of calls 1..N-1. An extractor that can see the previous documents may re-report their entities; a judge that can see its own earlier verdicts anchors on them. For anything you intend to measure, that is a correctness bug, not just a cost one.

Do this instead¶

cfg := chat.Config{
    Provider:       chat.ProviderGemini,
    ResponseSchema: chat.GenerateSchema[Extraction](),
    Stateless:      true, // each call is a one-shot
}

client, err := chat.New(ctx, chat.Settings{Config: cfg})
if err != nil {
    return err
}

for _, doc := range docs {
    var out Extraction
    if err := client.Ask(ctx, prompt(doc), &out); err != nil {
        return err
    }
    // handle out
}

One client, N calls, each carrying only its own document. No per-call construction cost, and Usage() still accumulates across the whole batch so you can account for the run as a whole.

What stateless does and does not change¶

Tool calling is unaffected. Stateless means "no history across calls", not "no messages within one". The ReAct loop keeps the assistant and tool turns it needs to make progress and discards them when the call returns.

Add becomes a one-shot preamble. Buffered turns go out with the next call and are then cleared, so Add is still the way to assemble a multi-part request:

_ = client.Add(ctx, "Use British spelling throughout.")
_ = client.Ask(ctx, prompt(doc), &out) // carries the preamble
_ = client.Ask(ctx, prompt(next), &out) // does not

For instructions that apply to every document, use Config.SystemPrompt instead, because it is sent with every call and costs nothing to repeat.

Run the batch across workers¶

Stateless provider clients are safe for concurrent use, so one client can back a worker pool:

var wg sync.WaitGroup
sem := make(chan struct{}, 8)

for _, doc := range docs {
    wg.Add(1)

    go func() {
        defer wg.Done()

        sem <- struct{}{}
        defer func() { <-sem }()

        var out Extraction
        _ = client.Ask(ctx, prompt(doc), &out)
    }()
}

wg.Wait()

Conversational clients from the current provider modules are safe to share too, but they serve one call at a time, so a pool gets no parallelism from one; see Concurrent use.

A fallback composite from NewFallback is safe to share when every member is, from chat v0.29.0.

Several models on one endpoint¶

A client is built for one model. To compare models served by one endpoint, such as a base model and its LoRA adapters on one vLLM server, build one stateless client per model from the same Config, and share one HTTP client between them:

base := chat.Config{
    Provider:  chat.ProviderOpenAICompatible,
    BaseURL:   "https://vllm.internal:8000/v1",
    Stateless: true,
}
base.HTTPClient = chat.ChatHTTPClient(base) // one connection pool for every model

clients := map[string]chat.ChatClient{}
for _, model := range []string{"base", "adapter-a", "adapter-b"} {
    cfg := base
    cfg.Model = model

    client, err := chat.New(ctx, chat.Settings{Config: cfg})
    if err != nil {
        return err
    }

    clients[model] = client
}

• Share the HTTP client. Without one, each client builds its own transport and so its own connection pool to the same host.
• Build it with chat.ChatHTTPClient. A client you pass in is used as it is, and Config.RequestTimeout no longer applies, so a plain http.Client{} would have no timeout at all.
• Each client keeps its own usage, so cost per model comes for free, and the map key is the model that answered.
• Each client is stateless, so all of them are safe to share across the workers above.

One client that addresses every model was considered and declined; see spec 0033.

Providers that predate the flag¶

Config.Stateless is honoured by claude, openai, openai-compatible, gemini and claude-local, each of which implements chat.StatelessCapable.

If you pin a provider module older than the flag, chat.New returns an error rather than building a client that ignores it:

provider gemini does not support Config.Stateless

That is deliberate. A silently-ignored flag would bill you for the accumulated history you asked not to send, the exact failure the flag exists to remove. Fix it by upgrading the provider module, not by unsetting the flag.