Your Model Card Is Not Your Conformity Artifact

Two weeks later, the file is already lying

On 2 August 2026, Articles 6 and Annex III of Regulation (EU) 2024/1689 started applying to high-risk AI systems. Article 15 (accuracy and robustness), Article 14 (human oversight), and Article 72 (post-market monitoring) now bind providers of in-scope systems.

Today is 19 August, the second full work week after the deadline. Legal and GRC teams are closing technical-file checklists. Platform teams still cannot show that the live conversational system matches those documents. Customer DDQs arrived the same week.

The popular answer is a folder: a vendor system card, a 2025 GPAI Code of Practice note, a model card from OpenAI, Anthropic, Google, or Mistral, plus last quarter's Annex IV draft. That folder is not conformity. It is a description of a component you do not operate.

The Act attached the duty to you, not the model vendor

Most coverage treated 2 August as a legal milestone. Policies. Vendor paperwork. That coverage is wrong about the engineering object.

GPAI provider duties hit in August 2025. High-risk system duties hit in August 2026. They do not share an addressee. A system card from a frontier lab describes a foundation model under lab conditions. Your customer talks to a composed product: your system prompt, your tools, your retrieval corpus, your routing logic, and the provider's silent safety defaults.

If you place that product on the market for an Annex III use (creditworthiness, employment, education, access to essential services), you are the provider of the high-risk system. Article 25 on the AI value chain does not let you hand that status to whoever sold you tokens.

We already mapped how composed conversational systems fail in production in 5 Reasons Why AI Agents Fail (And How to Prevent Them). Invented policy. Oversight handoff that never fires. Drift nobody logged. The Act does not invent those failure modes. It makes you produce evidence that they are under control on the system that is live this week, not on the model that was measured last year.

A system card is a component spec

Pin the gap to a real stack. You call gpt-4o on Azure OpenAI, Claude on Amazon Bedrock, or Gemini on Vertex AI. The vendor published a system card. Your technical file cites it. None of the following appear in that card:

• The system prompt your PM changed in a feature flag last Tuesday
• The RAG index ops re-ingested from Confluence
• Tool schemas that can write to the case-management API
• Temperature, max tokens, and rewrite settings the SDK defaulted because you never set them
• The provider safety classifier that moved when they shipped a new filter version behind the same model name

If you did not pin a dated snapshot (the gpt-4o-2024-08-06 pattern, or a dated Bedrock model ID), the alias can move without a commit in your repo. Git is clean. The live system is not the system in the file.

That is the structural failure. Conformity claims were written against a frozen narrative. The product is a moving composition.

What you actually have to prove

Article 15 does not ask whether the base model scored well on a public benchmark. It asks whether your system is accurate and robust for its intended purpose. A credit assistant that pastes an MMLU number from a model card has answered the wrong question.

Article 14 does not ask whether the lab has a human-oversight slide. It asks whether a competent person can understand, override, and interrupt your composed product in the real workflow: the UI, the confidence display, the escalation path, and the model's habit of sounding certain when it is guessing.

Article 72 expires last month's PDF. Post-market monitoring is a lifetime duty on the system as it runs. A technical file you cannot regenerate against the running product is already stale. Two weeks is enough for a prompt change, a corpus refresh, and a provider default to land.

Independent observation of the live experience, which we argued for in Why Your Chatbot Needs a Secret Shopper, is how you show the file still describes what customers talk to.

The false sense of conformity

Here is the pattern in board threads this week:

1. We only use a frontier model that published a 2025 GPAI pack.
2. Legal signed the vendor DPA and filed the system card.
3. Therefore we made 2 August.

Step 3 does not follow. The GPAI pack is the vendor's homework. Your homework is the high-risk system you assembled. Filing a component spec as if it were a system spec is how you manufacture a false sense of conformity.

Auditors will not stay on the PDF. The first serious customer DDQ already asks you to show accuracy and robustness on the deployed experience, the oversight path as a user would hit it, and that you still know what is running. "We use a frontier model with a safety report" used to close the ticket. It will not close it now.

Put living evidence on the delivery path

Do not start another policy. Start with the running product.

1. Name the composition. Model alias, snapshot ID, prompt version, tool list, retrieval source and last ingest, safety-filter settings, SDK defaults you never overrode. If you cannot list these on one page, you cannot sign Annex IV honestly.
2. Treat silent provider movement as a release. Alias updates, filter version bumps, and default-parameter changes are production events. Log them. If the vendor will not notify you, assume drift on a calendar.
3. Map Articles 14, 15, and 72 to observed behavior. For each claim in the technical file, keep a dated record of what the live conversational system actually did: disclosure, limitation language, escalation, refusal, answers against your policy corpus.
4. Regenerate the pack when the product moves. Prompt, retrieval, and tool changes should not ship unless evidence can be rebuilt against the same endpoint customers hit. Conformity in SharePoint is a snapshot of intent. Conformity next to the release is a control.

UndercoverAgent keeps a standing record of observed behavior on the live conversational system, including EU AI Act scenarios for disclosure, limitations, oversight, and data transparency, and attaches it to the pipeline that ships the change.

Ask your platform lead one question this week: can we reproduce the technical-file claims against the system customers are using today? If the answer is no, you did not make 2 August. You made a document. The document is already out of date.