Process a batch of independent documents¶

A ChatClient is a conversation by default. Every Chat, Ask and StreamChat call appends the exchange to the client's history, and the next call re-sends all of it.

That is right for a conversation and wrong for batch work: classifying, extracting, summarising or judging a set of documents that have nothing to do with one another. Set Config.Stateless for that.

The trap¶

Holding one client across N independent calls looks like the obvious way to avoid rebuilding a client per document. It is also the expensive way:

// Don't do this for independent documents.
client, _ := chat.New(ctx, chat.Settings{Config: cfg})

for _, doc := range docs {
    var out Extraction
    _ = client.Ask(ctx, prompt(doc), &out) // carries every previous doc
}

Call 1 sends document 1. Call 2 sends documents 1 and 2 plus the first answer. Call 18 sends all seventeen documents before it. Eighteen documents cost 18×19/2 = 171 document-equivalents rather than 18, a 9.5× multiplier, and it grows with the square of the batch.

Nothing warns you. The calls succeed, the answers look plausible, and the cost shows up on a bill. Two things make it worse than it sounds:

• Rate limits trip for the wrong-looking reason. Per-minute input-token limits are breached by growth within a run, which reads like a pacing problem and invites a between-run cooldown that cannot help.
• The answers are contaminated. Call N is answered in the light of calls 1..N-1. An extractor that can see the previous documents may re-report their entities; a judge that can see its own earlier verdicts anchors on them. For anything you intend to measure, that is a correctness bug, not just a cost one.

Do this instead¶

cfg := chat.Config{
    Provider:       chat.ProviderGemini,
    ResponseSchema: chat.GenerateSchema[Extraction](),
    Stateless:      true, // each call is a one-shot
}

client, err := chat.New(ctx, chat.Settings{Config: cfg})
if err != nil {
    return err
}

for _, doc := range docs {
    var out Extraction
    if err := client.Ask(ctx, prompt(doc), &out); err != nil {
        return err
    }
    // handle out
}

One client, N calls, each carrying only its own document. No per-call construction cost, and Usage() still accumulates across the whole batch so you can account for the run as a whole.

What stateless does and does not change¶

Tool calling is unaffected. Stateless means "no history across calls", not "no messages within one". The ReAct loop keeps the assistant and tool turns it needs to make progress and discards them when the call returns.

Add becomes a one-shot preamble. Buffered turns go out with the next call and are then cleared, so Add is still the way to assemble a multi-part request:

_ = client.Add(ctx, "Use British spelling throughout.")
_ = client.Ask(ctx, prompt(doc), &out) // carries the preamble
_ = client.Ask(ctx, prompt(next), &out) // does not

For instructions that apply to every document, use Config.SystemPrompt instead, because it is sent with every call and costs nothing to repeat.

Run the batch across workers¶

Stateless provider clients are safe for concurrent use, so one client can back a worker pool:

var wg sync.WaitGroup
sem := make(chan struct{}, 8)

for _, doc := range docs {
    wg.Add(1)

    go func() {
        defer wg.Done()

        sem <- struct{}{}
        defer func() { <-sem }()

        var out Extraction
        _ = client.Ask(ctx, prompt(doc), &out)
    }()
}

wg.Wait()

Two exceptions:

• Conversational clients are not safe to share. The guarantee applies only when Stateless is set.
• The fallback composite is never safe to share, stateless or not. It advances its active provider mid-call, so concurrent calls would disagree about which provider is live. Give each worker its own composite from NewFallback.

Providers that predate the flag¶

Config.Stateless is honoured by claude, openai, openai-compatible, gemini and claude-local, each of which implements chat.StatelessCapable.

If you pin a provider module older than the flag, chat.New returns an error rather than building a client that ignores it:

provider gemini does not support Config.Stateless

That is deliberate. A silently-ignored flag would bill you for the accumulated history you asked not to send, the exact failure the flag exists to remove. Fix it by upgrading the provider module, not by unsetting the flag.