90 Days of Model Drift: 40 Retirements, 18 New Models, and Two Months of Our Own Red CI

Our integration tests went red on 18 August and stayed red every day until 19 October. Nothing in our code was broken. Providers had retired model IDs that were still listed in our model picker, and every test that called one of them failed.

When we finally audited it, the count was 33 model IDs already gone across nine providers, six more from OpenAI with a shutdown date of 23 October, and one more scheduled for April 2027. Forty retirement entries in all, against 18 new models we added in the same sweep. This post is the inventory, and what we changed so it does not happen again.

We build a product whose job is to tell you when models change under you. We did not notice for two months that they had changed under us. That is the story.

Why nobody noticed

A test suite that fails every day stops being a signal. Ours went red three days after the last human commit of the summer, 15 August, and nothing forced anyone to look until our 90-day audit on 19 October.

The root cause, in our audit's words: "Not code. Providers retired models that are still in" the model registry. The dead IDs were not only in the picker. Our consensus-model map and the MCP server's evaluation-model map pointed at dead models for Groq, xAI, DeepSeek, Cerebras and Fireworks, and seven API documentation pages used retired IDs in their examples.

What retired

Every ID below was in our registry and had stopped serving by the time we checked each provider's live /models endpoint on 2026-10-19. For these 33, the date is the day we noticed, not the day the provider pulled them. We do not know the real dates, which is part of the problem. Together and Fireworks IDs are shown without their path prefixes.

mistral-large-latest is on the list. An alias with "latest" in the name is not a promise that it will keep resolving.

Together retires models without taking them off the list. Qwen3-235B-A22B-Instruct-2507-FP8 is still returned by Together's /models endpoint, but a serverless request to it returns HTTP 400, model_not_available, with a message to create a dedicated endpoint. Any check that compares your registry to /models says it is fine. We only caught it because our benchmark battery made a real call and got 15 out of 15 errors. Five days later the same thing happened to most of the rest of our Together lineup. On 2026-10-24 Llama-3.3-70B-Instruct-Turbo, Meta-Llama-3.1-8B-Instruct-Turbo, Llama-4-Maverick-17B-128E-Instruct-FP4, DeepSeek-V3.1 and Qwen3-235B-A22B-Instruct-2507-tput all answered model_not_available while still listed. We then probed all 176 chat entries in Together's /models with one token each: 20 answered a serverless call, and no Llama model was among them. Nothing in the listing marks which models are serverless; a uuid field looked like a marker until we found 59 entries that had one and still failed. We retired all five, pointed their replacements at openai/gpt-oss-120b (DeepSeek V3.1 at DeepSeek-V4.1-Flash), and moved our consensus and evaluation calls off Llama.

The OpenAI cohort

OpenAI is the only provider in our schedule with a published shutdown date. Six models were due to stop serving on 2026-10-23:

For gpt-4.1-nano we deliberately point at gpt-6-luna rather than the replacement OpenAI names, gpt-5.4-nano, because gpt-5.4-nano is itself scheduled to retire on 2027-04-01. Migrating to a model with a six-month shelf life means migrating twice. The swap table post has the price arithmetic for each line.

What shipped

In the same sweep we added 18 model IDs to the picker, every one confirmed present in the provider's live /models list on 2026-10-19:

Three more were renames rather than new models: Together's DeepSeek-V4-Pro became DeepSeek-V4-Pro-0813, its Llama 4 Maverick moved from an FP8 to an FP4 build (which then went dedicated-only on 2026-10-24, see above), and Mistral's dated Magistral Medium became magistral-medium-latest.

Some IDs providers now serve we chose not to add yet, including grok-4.20, gemini-3.7-flash, gpt-5.4-nano (see above) and gpt-5.5-pro, which is available only on OpenAI's Responses API rather than chat completions.

The 90-day ledger: roughly two models out for every one in.

What we changed

1. A nightly registry check. A script now compares every model ID we reference, in the picker, the consensus and evaluation maps, and the integration-test config, against each provider's live /models list. A dead ID fails the job and opens or updates a single "model registry drift" issue, which closes itself once the registry is clean. New IDs are reported but never fail the build. A provider whose key is missing or whose request fails is reported as unchecked rather than as drift, so a bad key cannot hide behind a false alarm. Our pricing table is deliberately not scanned: it keeps retired IDs so old saved tests still show a cost.

2. One retirement schedule. Every entry in the tables above now lives in a single file with a provider, a model, a shutdown date, an optional replacement and a source. That one list drives three things:

• A "Retires <date>" badge in the model picker for upcoming shutdowns.
• An additive warnings array on /v1/prompts/test, /v1/prompts/compare and the MCP test_prompt tool, so a script calling a retiring model hears about it in the response it already reads:

{
  "code": "model_retiring",
  "provider": "openai",
  "model": "gpt-5.4-nano",
  "shutdown_date": "2027-04-01",
  "replacement": "gpt-6-luna"
}

• A retirement watch job that finds schedules, saved tests and recent API calls that use a retired or retiring model and notifies their owners in-app, naming the replacement. It runs daily.

The nightly check also prints a "retiring within 30 days" section from the same file, so a scheduled shutdown shows up a month ahead rather than as a red build on the day.

3. Real calls, not just lists. The Together case shows the limit of comparing against /models. A list check catches removals; only a real request catches "listed but not available to you". Since 2026-10-24 the nightly check sends every Together ID we reference a one-token request instead of trusting the listing, the way it already did for Perplexity, which has no /models at all. The benchmark battery stays part of the process for the same reason.

What we would tell anyone running a model registry

Model IDs are data with an expiry date. Store the date next to the ID. If the provider has not published one, you will learn it from a failed request, so make that request before your users do.

Do not let a red build become normal. One issue that stays open and keeps updating is harder to ignore than a daily red tick.

Pin versions, and still watch them. Pinning protects you from silent behaviour changes, not from the pinned ID disappearing, which is what happened to 33 of ours. Scheduled drift detection covers the first; the registry check covers the second.

Keep old prices. Deleting a price because a model is gone corrupts the history you compare against.

For how the new models compare, see the open-weight vs closed-source benchmark and October's cost-per-quality table.

Try it yourself

• Ask a model why teams pin versions — the three-bullet task from our battery, on the topic of this post

No account, no API key. To run your own prompts against a pinned model on a schedule and get told when its output or its availability changes, that's Promptster.

Retirement and addition lists are from our 90-day audit (2026-07-21 to 2026-10-19) and our model retirement schedule, checked against each provider's live /models endpoint on 2026-10-19 with our test keys. For the 33 IDs marked as observed missing, 2026-10-19 is the date we saw them gone, not the provider's shutdown date. OpenAI shutdown dates are from OpenAI's deprecations page; prices are list rates from our pricing table. The first Together result is from our 2026-10-19 benchmark battery, where all 15 calls to that model returned HTTP 400; the wider Together retirement was observed on 2026-10-24 by sending each listed chat model a one-token request.