Cache a large stable prompt¶

If you send the same large block of text on every call (a reference corpus, a document you ask many questions about, a fixed set of examples) you can ask the provider to cache it and be billed at a much lower rate for the repeats.

cc, ok := client.(chat.CachingChatClient)
if !ok {
    // this provider has no caching; carry on without it
}

_ = cc.AddCached(ctx, referenceCorpus)   // cached, billed once at full rate

for _, q := range questions {
    var out Answer
    _ = cc.Ask(ctx, q, &out)             // corpus billed at the cached rate
}

AddCached is Add with a request to cache. Everything else about the call is unchanged.

Check that it worked¶

This is the important part. Caching is a hint, and every provider silently declines it in some circumstances. The only honest confirmation is the number:

fmt.Println(client.Usage().CachedTokens)   // zero means nothing was cached

If that stays zero, caching is not happening, and the most likely reason is that your content is below the provider's minimum.

Minimum sizes¶

Every provider refuses to cache small content, and most do so without an error. The thresholds are per model and are not discoverable through any API.

Claude's varies more than you might expect, and not in version order:

So a prefix that caches on one model may quietly stop caching when you change model. Usage.CachedTokens is how you find out.

What each provider does without being asked¶

Some caching happens whether or not you use AddCached, and how much you can rely on it differs sharply.

This is why AddCached is worth using even on Gemini, where something happens anyway: explicit caching covered ~100% of the same prefix on every call, against implicit's varying ~40%.

Choosing a TTL¶

cfg := chat.Config{CacheTTL: time.Hour}

Providers offer discrete choices and pick the nearest they support. Claude offers 5 minutes or 1 hour. Zero leaves the provider default, which is 5 minutes on Claude.

A cache write costs more than an uncached read. So if your content expires before you reuse it, caching costs you more than not caching. Roughly:

• A burst of calls within minutes. The default TTL is fine.
• Calls spread over an hour. The longer TTL pays, though writes cost more.
• A handful of calls a day. Every call finds a cold cache. Do not cache.

There is no way for the module to know your call rate, which is why caching is off unless you ask.

Things that will surprise you¶

Changing Config.Effort can invalidate the cache on Claude, depending on model. If you cache a corpus and then raise the effort level, expect the next call to pay the write again.

Claude allows at most 4 cached blocks per request. A fifth AddCached returns an error rather than dropping one, because which four to keep is not a decision this module should make for you.

Changing the content makes a new cache. That is correct, not a bug: a cache is keyed by exactly what went into it. But a corpus with a timestamp in it is different content on every call, and will never be reused.

Order matters on Claude. Caching covers everything up to and including the marked block, so put stable content first and per-call content after it. Calling AddCached with content that varies wastes the write premium on something that can never be read back.

Provider support¶

A provider that cannot cache does not implement the interface, so the type assertion tells you. Nothing is silently ignored.

Related¶

• Configuration fields: what CacheTTL defaults to and when it is a construction error.
• Control sampling & reasoning effort: Effort interacts with caching.
• Process a batch of documents: stateless clients, which share a cache across a worker pool.