Regenerating capability data¶

tools/capabilitygen turns models.dev's catalogue into a capability matrix for one provider adapter. It is maintainer tooling: CI runs it on release, and you run it by hand when you want to see what would change.

This page is for people working on the family. If you are using the library and want to know what a capability answer means, read Providers.

What problem it solves¶

Capabilities() has to answer questions about the model on the other end of the wire, and that is not knowable from our code. For a while the adapters answered anyway, reporting adapter-level facts where endpoint-level ones belonged, which is how openai-compatible came to claim it accepted images while the core refused every attachment sent to it.

models.dev is a community-maintained, MIT-licensed catalogue of what models actually support. LangChain generates its own model profiles from it. It covers every provider this family ships, and its fields map almost one-to-one onto our capability vocabulary.

Running it¶

The justfile target is the short way. It writes into a sibling checkout, because the matrix belongs to the adapter rather than to the core:

just capabilities bedrock chatbedrock ../chat-bedrock/models_catalogue.go ../chat-bedrock/models.go

The paths are relative to this repository's root. The recipe changes directory into the nested module to run the tool, so a path relative to where you are standing would resolve from the wrong place.

The fourth argument is the adapter's measured table, and is optional. Give it wherever the adapter has one, because that is what makes the run a cross-check rather than a download; see Comparing with what was measured.

Directly, if you want the output on stdout to eyeball first:

cd tools/capabilitygen
go run . -provider bedrock -package chatbedrock

-input exists so tests and offline builds do not depend on a third party being reachable, and so a CI job can pin a downloaded copy rather than fetching mid-release.

Comparing with what was measured¶

Two adapters measure capabilities themselves: chat-openai probes the API from internal/genmodels and writes models_generated.go, and chat-bedrock carries a hand-measured modelTable in models.go, assembled by sending real requests. Both are kept after models.dev arrived, as a cross-check on it, and -probe is where the check happens.

Point it at the file holding the measured table. The tool parses the Go source and reads every string-keyed map of struct literals holding chat.Support values, which is the shape every measured table in the family has, so there is no second data format to keep in step. Field names are matched by meaning (tools, Temperature, structured, multimodal, reasoning and so on); fields models.dev cannot answer, such as TopP and caching, are skipped.

For every model both sources know, each capability where both give a confident answer and the answers differ is a disagreement. It is written to stderr and into the header of the generated file:

// Measured table models_generated.go: 23 models compared with models.dev, 1 disagreements (spec 0020 D6).
//   gpt-5.2-pro: Sampling measured Yes, models.dev No

Putting it in the file rather than only on a terminal is deliberate. A regeneration lands as a diff in a merge request, and the header is the first thing in that diff, so the conflict is seen by whoever reviews it rather than lost in a CI log nobody opens.

The tool does not decide who is right, and does not fail the run. The adapter's resolution order decides which answer a caller sees, and the record exists so that choice is made knowingly. The three disagreements that motivated this turned out to be our probe fabricating a Yes it had never measured, which is the kind of thing you only find by looking.

Unknown on either side is not a disagreement. It is the absence of an answer, and the adapters already fall through it to whichever source has one. Reporting it would bury the real conflicts under every model the probe could not reach.

The report also says how many models were compared. A run reporting no disagreements across no models means the ids did not line up, not that the two sources agree, and the count is what tells those apart.

Which providers it knows¶

openai-compatible is deliberately absent, and a test asserts it stays that way. A compatible endpoint is whatever a caller pointed BaseURL at, so there is nothing to key a lookup on. models.dev's provider-agnostic dataset is keyed vendor/model, so an endpoint reporting mixtral-8x7b matches nothing without a normalisation step nobody has designed yet.

What it will not tell you¶

Caching. models.dev carries no caching field, so CapCaching stays measured where it is measured today. We know only the Nova family caches on Bedrock because somebody sent requests and watched; no registry would have said so.

Streaming, stateless and persistence. These are adapter-level: properties of our own code, known with certainty, and not models.dev's business. That it has no field for them is the distinction being confirmed by an independent party rather than a gap.

Absent means unknown, never no¶

The one rule worth internalising. A field missing from models.dev becomes chat.SupportUnknown, not chat.SupportNo.

models.dev not knowing about a capability is not the same as a model not having it. SupportUnknown proceeds untouched, so it refuses nothing; a manufactured SupportNo would refuse a request that would have worked. The same reasoning applies to the probes, which used to infer support from the absence of a rejection and now report unknown instead.

Where the attachment flag and the declared modalities disagree, modalities wins: a model whose declared inputs are text only cannot take an attachment whatever the flag says.

When it runs¶

CI runs it on release, per provider, so a matrix is refreshed as part of cutting a version rather than drifting until somebody notices. Running it by hand and committing the diff is fine too, and is how you see what changed before a release does it for you.

Adding a provider¶

Add the mapping to providerIDs in catalogue.go, then run it. If models.dev does not carry the provider you will get an error naming it rather than an empty table, which is deliberate: a generator that silently writes nothing looks like a generator that worked.

When it breaks¶

The output is guarded by a golden file, so an upstream format change fails the build rather than quietly producing a wrong matrix. If TestGenerate_MatchesGolden fails, read the diff before regenerating: a change there changes shipped capability data.

cd tools/capabilitygen
UPDATE_GOLDEN=1 go test ./...

An empty catalogue and a provider with no models are both errors for the same reason.

Why it is a nested module¶

tools/capabilitygen has its own go.mod. A nested module's dependencies never reach the core's, so an HTTP client and a four-megabyte JSON catalogue stay out of the dependency graph of everything that imports chat. go list ./... in the core returns none of it.

The consequence is that go test ./... in the core does not reach it either, so the justfile runs its tests and lint explicitly. just ci covers both.

The decisions behind all of this are in spec 0020.