Skip to content

Verify a provider against the conformance suite

ChatClient promises that providers are interchangeable. Nothing tested that promise, so each provider drifted into its own reading of the contract. Ask returning raw text on one and failing on another, RequestTimeout bounding every HTTP provider but not the subprocess one, a stream leaking on the documented cancellation path. None of it failed anything until someone swapped providers.

The conformance package is the shared test suite that closes that gap. A provider module runs it against its own client, and a divergence fails CI in the module that introduced it rather than in someone else's application.

It depends on the standard library testing package and nothing else (no testify) so running it adds no assertion-library dependency to your module.

Write the test

One test per provider module:

func TestConformance(t *testing.T) {
    conformance.Run(t, conformance.Suite{
        Provider: chat.ProviderOpenAI,
        NewClient: func(t *testing.T, cfg chat.Config, b conformance.Behaviour) (chat.ChatClient, conformance.Control) {
            stub := newStubServer(b)        // an httptest server shaped by b
            cfg.HTTPClient = stub.Client()
            return newOpenAIClient(t, cfg), stub
        },
    })
}

NewClient is called once per case, so every case gets a clean conversation. Register teardown with t.Cleanup inside the factory. The suite tears nothing down for you.

Why your module supplies the backend

The suite asserts behaviour that is identical across providers, but the wire format producing that behaviour is not. A hanging backend is an httptest server that never writes for one provider and a sleeping subprocess for another.

So the suite says what the backend should do and asks what it saw, and your module bridges the two:

  • Behaviour is the instruction. Your stub reads it and responds accordingly. The zero value is a backend returning empty text successfully.
  • Control is the observation. The suite asks it what actually arrived, rather than inferring it from the client's own return values, which is precisely what a diverging provider gets wrong.

Behaviour carries Text, Hang (block until the context is cancelled, never responding), FailStatus (fail with an HTTP status, exercising your registered status extractor), ToolCall/ToolInput (request a tool on the first step and return Text on the second, the minimum ReAct exchange), and Refuse (answer with a refusal, in whatever shape your provider's API marks one).

What gets checked

Run executes one named subtest per contract, so a failing provider sees exactly which one it breaks:

Subtest What it proves
capabilities_match_the_client Every capability the provider's table claims as SupportYes that has an interface behind it (stateless, persistence, streaming, caching, logprobs) is implemented by the client, and a client implementing ToolCallProposer does not report CapTools as unsupported. Skipped when the provider registers no capabilities
ask_semantics Ask returns raw text when no schema is set, and never offers registered tools to the model, with or without a ResponseSchema. A schema travels in the provider's structured-output field, never in a tool, and registering tools does not discard it
history_observable A fresh client reports zero turns; a provider claiming History.Known grows them across a Chat, and reports LastInputTokens whenever it reports usage
history_policy_applied A Config.HistoryPolicy actually bounds what reaches the backend; skipped for a provider reporting History.Known == false, which cannot enumerate its turns
timeout_honoured Config.RequestTimeout bounds a call against a backend that never responds, even when the caller supplies no deadline
stream_lifecycle A callback error is returned to the caller and closes the response body, rather than leaking a connection per cancel
error_chain_preserved A provider HTTP error stays legible to chat.ProviderHTTPStatus, so the failover policy can still classify it
refusal_reported A refusal reaches the caller as chat.ErrRefused on Chat, Ask and StreamChat, with a *chat.RefusedError naming the reason, and the refused exchange is not recorded. Skipped without Refuser
concurrent_use Calls queued behind a running call share one request, an Ask runs alone, each edit rule (wait, refuse, merge) behaves as documented, and Restore and Add wait their turn, so nothing a running call commits can undo them. Skipped without Holder
logprobs_one_shot ChatWithLogprobs refuses a top outside 0 to 20 before any request, and neither records a turn, replaces History().LastInputTokens, nor sends the conversation. Skipped without LogprobsCapable
proposals_one_shot ProposeToolCalls refuses no tools, or a tool with a Handler, before any request; returns a call by name with its arguments, in one request, without running it; and leaves the conversation as logprobs_one_shot does. Skipped without ToolCallProposer

Capability gating is structural

Cases needing an optional contract (StreamingChatClient, CachingChatClient, PersistentChatClient, StatelessCapable, LogprobsCapable, ToolCallProposer) run only when the client your factory returns satisfies it. That is the same structural test chat.New already applies, so a provider that gains an interface starts running its cases without anyone remembering to switch them on.

The same applies to observation. Control has four optional halves:

  • ToolObserver reports whether a request advertised tool definitions. Every HTTP provider can implement this, since tools are a field in the request body.
  • SchemaOffered, the SchemaObserver half, reports whether a request carried a response schema in the provider's structured-output field (output_config.format, response_format and so on). Without it the suite cannot tell a provider that sends the schema without a tool from one that dropped the schema, because both offer no tools.
  • Holder holds the next request in flight until released, which is how the concurrent_use cases act while a call is running. An httptest stub implements it by blocking its handler on a channel.
  • Refuser says your stub honours Behaviour.Refuse. Unlike the halves above it declares what the stub can send rather than what it saw: a stub that ignored Refuse would answer normally, and the case couldn't tell that from an adapter that missed the refusal. Choose a shape every path can receive; for OpenAI that is finish_reason: content_filter, since refusal reaches only a schema-bound Ask.
  • StreamObserver reports whether the streamed response body was closed. Without it the stream case can prove the callback's error reaches the caller, but not that the connection was released, which is the half that actually leaks. An httptest-based stub implements it by wrapping the response body.

A Control that cannot make an observation simply omits the method, and the case skips with a line naming what was missed rather than passing vacuously. A run never reports coverage it did not have.

Read the skips

A green run with three skips is a weaker result than a green run with none. Because skips are the mechanism that keeps the suite honest, they are worth reading on every run. A case skipping for a reason you did not intend is the suite telling you your Control is thinner than you thought.