Your main branch has not moved in ten days. The last merged pull request was a copy tweak. The on-call rotation is already being rewritten around Labor Day, Monday, September 7. In stand-up, someone called the stack stable.
That sentence is false.
This is the week of August 24, 2026. US campuses are opening. Parents are buying. Telcos are standing up student plans. Retail and consumer SaaS are taking a cohort that did not exist in your July numbers. The people talking to your LLM product this week are not the people who produced your summer baseline. Floating aliases (*-latest, Azure OpenAI auto-upgrade deployments, Bedrock on-demand routing) kept moving on vendor cadence while the repo went quiet. Nobody merged a release. You shipped one anyway.
Git is quiet. The product is not.
Most quality programs still treat the application repository as the system of record. If Git is green, the product is frozen. That was a reasonable lie for deterministic services. It is a dangerous one for anything whose answers depend on four independent clocks: application code (Git), the resolved model identity behind the alias you call, the RAG corpus and tool schemas the model can see, and the live user mix.
Only the first clock shows up in git log. This week, the other three moved.
We have written before about why LLM products fail in production, including 5 Reasons Why AI Agents Fail (And How to Prevent Them). Those modes are still real. They are not what changed between early August and today. What changed is the denominator. You are judging a different population against a stack you may not have pinned.
If your last recorded score describes July traffic against a July alias, it does not describe this product. It describes a product that no longer exists.
What actually moved this week
Back-to-school is not a marketing season for an LLM app. It is a distribution shift.
New students and parents arrive with vocabulary your summer conversations barely contained: add/drop deadlines, orientation holds, meal equivalency, Parent PLUS, residence hall move-in, family-plan ports, dorm shipping. Return windows and student discounts land in the same thread as identity proofing. Fraud follows the money: parent-funded wallets, gift cards, ship-to-campus, account takeover on inboxes that were quiet all summer.
Your intent histogram will not look like July. A lull-season mix will look fluent on the old head and blind on the new one.
The stack moved without a pull request:
- Floating aliases.
gpt-4o-latest,gpt-4.1-latest, and Azure deployment names are pointers, not versions. Providers update them on their calendar. Bedrock on-demand routing can change which physical model serves a request with no deploy of yours. - RAG refresh. Nightly indexers, SharePoint connectors, and Zendesk crawls land documents with no app diff. A policy page edited by ops on Monday is in answers on Tuesday.
Ask for one artifact: the resolved model version ID from a live call this morning, next to the document-set hash from the last successful index. If neither appears in the summer baseline notes, you have your release.
The summer baseline is a different product
Launch week is loud. You expect surprise. Silent-change week is the opposite. Git is calm. Coverage is thinning into the long weekend. The last recorded score still sits in the risk register because nothing "shipped." The user mix that produced that score has been replaced. The alias that produced it may have been replaced too.
A baseline is a claim about a joint distribution: this population, this model pointer, this corpus, this week. Drop any term and the claim is void. Teams keep the number and drop the terms. That is how a July 82 becomes an unexamined September assumption.
We already argued, in The Secret Shopper Methodology for AI Testing, that internal audits miss live behavior. The gap this week is cruder. Last month's live behavior is the wrong object. You would not quote a Sunday capacity run against Cyber Monday traffic. LLM apps get a pass because the miss does not look like a 500. It looks like a confident wrong refund policy, a missed fraud pattern, a refusal that was correct in July and rude in week one of term.
The highest-risk window is not launch week. It is this week: cohort in motion, aliases in motion, on-call coverage going thin, and a green repo used as a sleeping pill.
Bring a mix-shift ticket to Friday's risk review
Between now and Labor Day someone will ask whether coverage can come down. The honest answer depends on evidence younger than the mix-shift, not on Git history.
Do not bring a green build to Friday. Bring a one-page RFC that treats the week of August 24 as a production release you did not author.
Title: Mix-shift release, week of 2026-08-24. No application diff.
| Field | What to attach |
|---|---|
| Traffic-mix delta | Intent histogram this week vs the summer baseline week |
| Resolved model identity | Azure model version, OpenAI snapshot ID, or Bedrock inference profile. Not the alias. |
| Corpus version | Last index job time, document count, content hash |
| Evidence timestamp | Last observed sample taken after the mix-shift, against that resolved identity |
| Coverage | Named on-call through 2026-09-07 who can interpret a mix-shift incident |
| Decision | Accept as-is, pin the alias, freeze the indexer, or staff a watch |
If the evidence timestamp is older than the cohort change, you are not accepting a release. You are asserting that last month's product still exists. Make that assertion explicit and signed.
Pinning stops clock 2 while clocks 3 and 4 are already in motion. If you will not pin, collect new evidence whenever the alias resolves to a new ID, a RAG refresh lands policy content, or top intents swing by 20 points. Those are releases. Write them down.
git log --since='2026-08-01' --oneline -- prompts/ src/
curl -s "$AZURE_OPENAI/openai/deployments/chat" \
-H "api-key: $KEY" | jq '{model, version}'
If the first command is empty and the second has moved since the summer baseline, stop calling the system unchanged. Population drift has a campus calendar, not a status page. Put it on the CAB agenda anyway.
Evidence freshness is the buying question
Platform comparisons this week should stop treating a green pull-request check as a stability signal. That measures clock 1. The product is clocks 1 through 4.
Ask how old the newest observed-behavior evidence is relative to the current traffic mix, whether it was collected against the currently resolved vendor alias or merely the alias name, and whether a corpus change without a Git event produces a new evidence record.
If a vendor can only wake up when someone opens a pull request, they will be asleep for the most expensive week on the calendar.
UndercoverAgent is relevant here only if it can show evidence freshness after a mix-shift and an alias move, not because a pipeline went green in July. If you cannot put a sample on the table that is newer than this week's cohort, you are governing a product that has already left the building.
Put the mix-shift RFC on Friday's agenda. Pin or re-sample before Labor Day. The repo can stay quiet. The release already shipped.