Skip to content

Verify a provider against the conformance suite

ChatClient promises that providers are interchangeable. Nothing tested that promise, so each provider drifted into its own reading of the contract. Ask returning raw text on one and failing on another, RequestTimeout bounding every HTTP provider but not the subprocess one, a stream leaking on the documented cancellation path. None of it failed anything until someone swapped providers.

The conformance package is the shared test suite that closes that gap. A provider module runs it against its own client, and a divergence fails CI in the module that introduced it rather than in someone else's application.

It depends on the standard library testing package and nothing else (no testify) so running it adds no assertion-library dependency to your module.

Write the test

One test per provider module:

func TestConformance(t *testing.T) {
    conformance.Run(t, conformance.Suite{
        Provider: chat.ProviderOpenAI,
        NewClient: func(t *testing.T, cfg chat.Config, b conformance.Behaviour) (chat.ChatClient, conformance.Control) {
            stub := newStubServer(b)        // an httptest server shaped by b
            cfg.HTTPClient = stub.Client()
            return newOpenAIClient(t, cfg), stub
        },
    })
}

NewClient is called once per case, so every case gets a clean conversation. Register teardown with t.Cleanup inside the factory. The suite tears nothing down for you.

Why your module supplies the backend

The suite asserts behaviour that is identical across providers, but the wire format producing that behaviour is not. A hanging backend is an httptest server that never writes for one provider and a sleeping subprocess for another.

So the suite says what the backend should do and asks what it saw, and your module bridges the two:

  • Behaviour is the instruction. Your stub reads it and responds accordingly. The zero value is a backend returning empty text successfully.
  • Control is the observation. The suite asks it what actually arrived, rather than inferring it from the client's own return values, which is precisely what a diverging provider gets wrong.

Behaviour carries Text, Hang (block until the context is cancelled, never responding), FailStatus (fail with an HTTP status, exercising your registered status extractor), and ToolCall/ToolInput (request a tool on the first step and return Text on the second, the minimum ReAct exchange).

What gets checked

Run executes one named subtest per contract, so a failing provider sees exactly which one it breaks:

Subtest What it proves
ask_semantics Ask returns raw text when no schema is set, and never offers registered tools to the model
history_observable A fresh client reports zero turns; a provider claiming History.Known grows them across a Chat, and reports LastInputTokens whenever it reports usage
history_policy_applied A Config.HistoryPolicy actually bounds what reaches the backend; skipped for a provider reporting History.Known == false, which cannot enumerate its turns
timeout_honoured Config.RequestTimeout bounds a call against a backend that never responds, even when the caller supplies no deadline
stream_lifecycle A callback error is returned to the caller and closes the response body, rather than leaking a connection per cancel
error_chain_preserved A provider HTTP error stays legible to chat.ProviderHTTPStatus, so the failover policy can still classify it

Capability gating is structural

Cases needing an optional contract (StreamingChatClient, CachingChatClient, PersistentChatClient, StatelessCapable) run only when the client your factory returns satisfies it. That is the same structural test chat.New already applies, so a provider that gains an interface starts running its cases without anyone remembering to switch them on.

The same applies to observation. Control has two optional halves:

  • ToolObserver reports whether a request advertised tool definitions. Every HTTP provider can implement this, since tools are a field in the request body.
  • StreamObserver reports whether the streamed response body was closed. Without it the stream case can prove the callback's error reaches the caller, but not that the connection was released, which is the half that actually leaks. An httptest-based stub implements it by wrapping the response body.

A Control that cannot make an observation simply omits the method, and the case skips with a line naming what was missed rather than passing vacuously. A run never reports coverage it did not have.

Read the skips

A green run with three skips is a weaker result than a green run with none. Because skips are the mechanism that keeps the suite honest, they are worth reading on every run. A case skipping for a reason you did not intend is the suite telling you your Control is thinner than you thought.