You already know the ritual. Open the chatbot. Try a couple of questions. Screenshot the weird one. Drop it in Slack. Wait for a customer to report the next one.
That is watching. It is not a weekly brief.
The job is a written brief
An operator who owns a live AI agent — support bot, sales chatbot, MCP-connected agent — needs a recurring answer to one question: what did this agent do this week, and what should we change.
Last month's eval does not answer that. The model, the tools, the traffic mix, and the policy pages moved. The score from July describes a product that is no longer in production.
Ad-hoc secret shopping has the same gap. You pose as a customer, you find a crack, you move on. The finding lives in a screenshot. Nobody writes it down. Next week you do it again, or you don't.
We already argued, in Why Your Chatbot Needs a Secret Shopper, that conversational systems need mystery shopping, not a login-form test. The next step is the artifact. The secret shopper visit is the fieldwork. The weekly brief is the job.
What a weekly brief is
The Weekly Brief Retainer is one written intelligence brief on one live agent, every week. Undercover tests run against the target. The brief reports security, quality, and compliance highlights with evidence. Recommendations are transcript-backed. It is not a certification, and it is not a guarantee that the agent is safe.
That is the whole offer. $500/week. Cancel from the receipt Stripe emails you.
Watching the chat does not produce that document. Neither does treating last month's eval as this week's product.
DIY watching vs a brief
DIY watching looks like this:
- Spot-check a handful of chats when you have time
- Screenshot a failure and hope someone files a ticket
- Wait for a customer to report a bad refund, a leaked prompt, or a wrong policy
- Quote last month's score in this week's standup
A weekly brief looks like this:
- The same secret-shopper method, on a calendar
- Findings written down, with the transcript attached
- A recommendation you can act on before next week's traffic
- One live target, one brief, every week
The first is a habit. The second is an artifact someone else can read.
If you are already doing the fieldwork yourself, start the weekly brief and stop treating screenshots as the record.
Key Takeaways
- Watching the chat is not a weekly intelligence brief
- Last month's eval is not this week's product
- The secret shopper visit is fieldwork; the brief is the job
- Findings need transcripts, not screenshots
<CtaBanner title="The weekly brief is the job" buttonText="Start weekly brief" href="/weekly-brief"
One weekly intelligence brief on your live AI agent. Transcript-backed findings, not a certification.