Skip to content

Bound a long conversation

Every provider appends each exchange to its conversation and re-sends the whole of it on the next call. For a genuine conversation that is the point. For a long-running agent loop it is a slow leak: the transcript grows, every call costs more than the last, and eventually the provider returns a context-overflow error that the failover policy classifies as fatal, correctly, since retrying a too-large request against another provider does not make it smaller.

If your calls were never a conversation, you want Config.Stateless instead. This page is for the conversation that legitimately accumulates.

See the growth first

ChatClient.History() reports what a client is carrying:

h := client.History()
log.Info("conversation size",
    "turns", h.Turns,
    "last_input_tokens", h.LastInputTokens,
)

LastInputTokens is the provider's own count for everything the last call sent (system prompt, retained history and the new turn together). It is a measurement, not an estimate, and stale by exactly one call.

The core cannot do better than one call stale, and that is worth understanding before you build on it. Counting locally would need a tokenizer per model family, and there is no universal one: Anthropic publishes none, Gemini counts server-side, and only OpenAI can be tokenised offline. So the number you get is the number the provider billed you for.

Check Known before trusting Turns

History.Known is false when a provider cannot count its own transcript. claude-local is the case in the core: the conversation lives in the claude CLI's own session, resumed by ID, so the provider can see what it has buffered locally but not what the far side will replay. LastInputTokens is still reported whenever the provider reports usage at all.

Bound it

A policy is applied by the provider, so the provider module has to have adopted it

Core v0.10.1 shipped HistoryPolicy, TruncateOldest and the invariant checks, and no provider called any of them, so a policy set on that release has no effect at all (issue 13).

The core now provides the application path (chat.BoundConversation) and a conformance case, history_policy_applied, that fails any provider module which does not run it. A policy therefore takes effect on any provider whose release notes record adopting it, and the conformance case is what makes that claim checkable rather than a promise. See Verify provider conformance.

claude-local is the deliberate exception and will not gain it: its conversation lives in the claude CLI's own session, so it cannot enumerate the turns a policy would bound. Setting a policy there warns once at construction and is otherwise ignored. For a long-running loop on claude-local, start a fresh client, or use Config.Stateless if the calls were never a conversation.

Set Config.HistoryPolicy. It is nil by default, which is the behaviour every release before v0.10.0 had:

cfg := chat.Config{
    Provider:      chat.ProviderClaude,
    HistoryPolicy: chat.TruncateOldest(40),
}

TruncateOldest keeps the most recent turns and drops the oldest. Two rules constrain it, and both are enforced by the core rather than by the policy, so they hold for any policy written later.

Cached turns are never dropped. A turn added with AddCached is billed as a matching prefix, so dropping one does not merely lose context. It destroys every subsequent cache hit. The failure would show up as a larger invoice and nothing else.

A tool request and its results stay together. Chat's ReAct loop commits the model's tool call and the result it produced as a pair, and providers reject a request that carries one without the other. A policy that cut between them would produce an API error rather than a shorter conversation.

Turns are kept as a contiguous recent run rather than cherry-picked to fill the budget exactly. Skipping a large tool exchange to fit two smaller older turns would leave the model reading a conversation that never happened.

Caching and truncation together

They work together, but not for free, and construction says so:

WARN Config.HistoryPolicy and Config.CacheTTL are both set; turns added with
     AddCached are pinned and will not be truncated, so the policy can only
     bound the conversation that follows them

That is a warning rather than a refusal because the combination is legitimate: pinning is exactly what makes it safe. What it tells you is where your truncation budget actually went: a conversation with a large cached prefix has correspondingly less that the policy is allowed to trim, and a policy that looks generous may free almost nothing.

If you need both and the budget is not landing where you want it, the lever is the size of the cached prefix, not the policy.

Compact instead of forgetting

TruncateOldest drops turns, so a long-running agent forgets what it did. Compaction summarises the older region and substitutes the summary, which keeps the thread while still bounding the window:

cfg := chat.Config{
    Provider:      chat.ProviderClaude,
    HistoryPolicy: chat.CompactOldest(),
}

That is the whole of the simple case. Every knob has a default that works: the summary comes from the model the conversation is already using, compaction runs above 200 retained turns or 30% of the model's reported input limit, whichever is reached first, and a failure fails the call.

It costs a round-trip, so it runs rarely

Compaction calls the model. That is the difference from truncation, and it is why the default budgets are much larger than a truncating policy's would be: dropping turns is free and summarising them is not.

Deciding it has nothing to do is free. A conversation inside both budgets is passed through without any call, so the per-request policy pass costs nothing until it is actually needed.

The token budget deserves a word. A model advertising a very large input window is not usefully attentive across all of it. Recall degrades well before the hard limit, in a conversation that still fits. Compacting at a share of the limit rather than at the limit is a quality decision, not a capacity one:

chat.CompactOldest(
    chat.WithTurnBudget(200),
    chat.WithTokenBudget(250_000),
)

The token count compared against that budget is the provider's own figure for the previous call, so it lags by one call and is absent before the first. The turn budget is the backstop for both cases rather than a second opinion.

Summarise with a cheaper model

Summarising is not the hard task, so a smaller model against a larger conversation is the obvious economy:

chat.CompactOldest(chat.WithSummarizer(chat.Config{
    Provider: chat.ProviderClaude,
    Model:    "claude-haiku-4-5-20251001",
}))

Whatever the summariser is built from, three things are cleared and one is set. Each has a failure behind it: a ResponseSchema would force the summary into your domain type, registered tools would give the summariser side effects, a history policy would let a compaction trigger a compaction, and Config.Stateless is set so the summarising client keeps no history of its own. Supply a whole client with chat.WithSummarizerClient and those guarantees become yours to keep.

When it fails

A failed compaction fails the call. The conversation is left as it was.

That is deliberate rather than convenient: silently truncating instead would give you forgetting when you asked for summarisation, and you would learn it from the model's behaviour rather than from an error. If a bounded conversation matters more to you than which way it was bounded, say so:

chat.CompactOldest(chat.WithCompactionFallback(chat.TruncateOldest(200)))

Worth knowing where this can bite: the policy runs before every provider round-trip, so a compaction can fail at step four of a tool-using exchange whose first three tool handlers have already run. Their side effects have happened. The alternative (sending the request anyway) is the context overflow the policy existed to prevent.

Keeping what matters to you

The house prompt asks for decisions, facts, identifiers and anything outstanding. Where a domain has terms a general summary would discard, say so:

chat.CompactOldest(chat.WithSummaryPrompt(
    "Summarise the exchange, preserving every order ID and every decision taken."))

The transcript is appended as delimited data, and the framing tells the model to treat instructions inside it as material to report rather than commands to follow. That framing survives your override: the wording is yours, the delimiting is not.

Compacting on demand

A policy runs when the client is about to send something. To compact because somebody asked (a /compact in an interactive session) there is no request to attach to:

keep, replace, changed, err := chat.Compact(ctx, cfg.HistoryPolicy, turns)

It ignores the budgets, folds up everything but the most recent exchange, and reports whether it changed anything so you can say "nothing to compact" honestly. A policy that is not a compacting one is bounded normally, so you need not check which one is configured.

What compaction will not touch

Two regions are off limits, and chat.ApplyHistoryEdit refuses an edit that breaks either:

  • Cached turns. A turn added with AddCached is neither dropped nor rewritten. Caching bills a matching prefix, so rewriting any of it invalidates every later cache hit, a cost that shows up only on the invoice.
  • Tool exchanges. A tool request and its results are folded in together or not at all. Summarising away a result while leaving the request produces an API error rather than a shorter conversation.

Specified in spec 0014.

Writing your own policy

A policy sees a description of the conversation, never the provider's own message objects. The core treats a transcript as opaque, and a neutral turn type would lose an Anthropic tool-use block, a Gemini session and an OpenAI params struct in three different ways.

type HistoryPolicy interface {
    Bound(ctx context.Context, turns []TurnInfo) (HistoryEdit, error)
}

Each TurnInfo carries the turn's index, role, a lossy text rendering, whether it is Pinned, and its ToolGroup. Return a HistoryEdit naming the indices to Keep, and optionally Replace text for turns you are rewriting.

You do not have to enforce pinning or tool grouping yourself. The provider runs your edit through ApplyHistoryEdit, which refuses one that drops a pinned turn, splits a tool group, names a turn that does not exist, or replaces a turn it did not keep. Getting it wrong fails loudly rather than corrupting a conversation.