The on-call with no deploy
It is 2:14 on a Tuesday afternoon and the pager goes off. Support escalations are up 3x. The refund bot is telling people it can "go ahead and process that for you" and then doing nothing. The compliance disclosure it used to append to every financial answer is gone about half the time.
You pull up the deploy log. Nothing. No merges since Thursday. No config changes. No feature flags flipped. Your prompt is byte-for-byte what it was last week, the week before, and the month before that. Git blame is a flat line.
So you do the thing everyone does next: you assume you are wrong. You diff the prompt against staging three times. You check the retrieval index. You re-read your own code convinced you missed something. You did not miss anything. Somewhere in a changelog you never subscribed to, a vendor rotated the model behind your latest alias, or quietly sunset the snapshot you thought you had pinned, and your "stable" system started running on a slightly different brain.
Welcome to a class of incident most teams still do not have a name for. Let's give it one: behavioral supply-chain risk.
What actually happened this summer
All three major providers have spent mid-2026 on the same cadence: rotate the snapshots, publish the deprecation notice, retire the old pin on a timeline. OpenAI, Anthropic, and Google have each pushed "model version sunset" notices this season, and the clustering is not a coincidence. Dated snapshots are expensive to keep serving. Aliases like latest are the whole point of aliases: they move.
Here is the part that trips people up. Pinning was supposed to be the safe move. But a pin is a lease, not a deed. When the snapshot behind gpt-4o-2024-xx-xx or an equivalent dated tag reaches end of life, the pin does not freeze your behavior forever; it just buys you a window before you are forced onto the next thing. And when your window closes, the migration is not "same behavior, new name." Refusal boundaries shift. Formatting defaults drift. Tool-call verbosity changes. The model that used to reliably emit JSON now wraps it in a friendly sentence 4% of the time. None of that shows up as an error. It shows up as your customers having a worse experience while every dashboard you own stays green.
The mental model most teams have is wrong
Ask an engineer who owns an AI feature "what controls how this behaves?" and you will hear: the system prompt, the temperature, the retrieval context, maybe the fine-tune. All internal. All in your repo. All things you change on purpose through a pull request.
That inventory is missing the largest input. A huge fraction of your production behavior is set by a weight file you have never seen, hosted by a company that can replace it on their schedule without asking you. You do not control it, you do not version it, and until this summer most teams did not even monitor it.
We have written before, in 5 Reasons Why AI Agents Fail (And How to Prevent Them), about failure modes like hallucination and edge-case collapse. Every one of those posts carried an unstated assumption: the failure originates from something you built or something a user did. Model swaps break that assumption. This is the first common failure mode where the trigger is a change you did not make, cannot see, and were not consulted on. It is a regression that arrives through your dependency graph, not your diff.
That reframing matters because it tells you where the fix lives. You already have a mature discipline for code you do not control that can change under you: dependency management. Lockfiles, Dependabot alerts, scheduled npm audit, renovate bots. Nobody would ship a service that silently pulled latest for every npm package on every request. Yet that is exactly the posture most AI systems have toward the single most important dependency they have.
Why your PR gate does not catch this
Most quality automation, including the kind we generally advocate for, is triggered by your activity. A pull request opens, checks run, the gate blocks the merge if quality drops. That is genuinely valuable, and it catches regressions you caused.
But a PR-triggered check has a fatal blind spot for this problem: there is no PR. The vendor's rotation does not open a pull request in your repo. It does not touch your branch. If the only time you evaluate behavior is when you ship, then during the exact window when an external change is silently degrading you, your checks never fire. You could go weeks. You will find out from your customers, which is the most expensive place to find out anything.
The lesson is not "pin harder." Pins expire, and expiry forces the very migration that changes behavior. The lesson is that behavioral evaluation has to run on a schedule, independent of your commit activity, against a standing baseline you captured when things were known-good. A nightly or hourly run that replays a fixed battery of realistic conversations and compares today's verdicts against last week's is the only thing positioned to see a vendor-side drift the moment it lands, instead of the moment it goes viral.
What to do differently on Monday
You do not need a new platform to start. You need a change in posture:
- Inventory your external behavioral dependencies. For every AI feature, write down the exact model, the alias vs. pinned snapshot, and the published deprecation date. Treat that date like a certificate expiry, because it is one.
- Capture a baseline while it is green. Record verdicts for a representative set of conversations today. This is your "known-good." Without it, you cannot prove a regression, you can only argue about vibes. This is the automation extension of the mystery evaluation methodology we described earlier: a fixed panel, run repeatedly, judged consistently.
- Move evaluation off the PR trigger. Add a scheduled run. Cron, GitHub Actions
schedule:, whatever you already use. The trigger is the whole point: it fires when the vendor changes something, not when you do. - Alert on delta, not just absolute score. A 6-point drop overnight with zero deploys is a louder signal than a low-but-stable score. Watch the derivative.
- Diff before you migrate. When a sunset forces you onto a new snapshot, run your baseline against both the old and new model before cutting over. Now the migration is a measured decision instead of a surprise.
The category worth naming
The industry spent years building muscle for catching regressions we caused. The regressions that will hurt most this year are the ones we did not. "Behavioral supply-chain risk" is not a scare phrase; it is just the honest name for a dependency you have always had and rarely tracked. The teams who get through the next round of deprecation notices calmly are the ones who already know what their AI is supposed to do, on a schedule, on a baseline, whether or not they shipped this week.
A scheduled behavioral baseline is exactly what UndercoverAgent runs against your live target on a cron, so a vendor's release notes never reach your customers before they reach you. If your only quality gate fires on your own pull requests, add a standing run this week; capture your known-good baseline while it is still green.