The Undercover
Agent Files
Guides, research, and field reports on AI agent testing, adversarial security, and conversational AI quality. Findings are test evidence, not a certification.
Watching the Chat Is Not a Weekly Brief
Spot-checking chats, screenshots, and last month's eval are not this week's product. The job is a written weekly brief with transcript-backed findings.
AI Testing & QA
97Watching the Chat Is Not a Weekly Brief
Spot-checking chats, screenshots, and last month's eval are not this week's product. The job is a written weekly brief with transcript-backed findings.
Read moreYour Users Changed This Week. Your Diff Didn't.
A green Git repo is not a stable LLM product. Back-to-school traffic and floating model aliases changed what you shipped this week, with no pull request.
Read moreYour Code Freeze Doesn't Freeze the Model
A code freeze locks Git before Labor Day. Vendor defaults and routing weights keep moving. Your last observed behavior is already a stale baseline.
Read moreYour Git History Is Lying About What Shipped
A quiet repo is not proof production is unchanged. Post-Black Hat MCP write-ups show capability-graph drift with zero commits.
Read moreSecurity
11Your AI Dependency Changed. Did Your Gate Notice?
The LiteLLM breach exposed more than credentials. It showed why AI reliability controls must detect behavior changes beyond application code diffs.
Read moreThe Gym Hack: Permissions Set the Blast Radius
A Claude agent exploited a gym booking workflow. The real lesson is that permissions, tools, and defaults now shape product reliability.
Read moreIntegrating Compliance and Security in the Era of GPT-4
Explore how GPT-4's new features impact compliance and security in AI development, and how to adapt your workflows accordingly.
Read moreWhy Ignoring Cybersecurity in AI Development Could Cost You Everything
Rising cybercrime demands that organizations integrate security into AI development, not as an afterthought but as a core component.
Read moreCI/CD & DevOps
27Your Model Pin Will Not Survive September 1
Labor Day week Git stays frozen while the September 1 vendor control plane still ships. A pinned model ID and green main are not the same product.
Read moreYour AI's Biggest Release May Skip Git
Anthropic's Claude Code default change exposes a control gap: AI behavior can shift without a commit, while CI keeps checking yesterday's assumptions.
Read moreAI in CI/CD: Balancing Speed and Quality Control
The rush to integrate AI in CI/CD pipelines is real, but are we sacrificing quality for speed? Here’s how to maintain standards amidst innovation.
Read moreThe AI Operations Debt Crisis: When Speed Kills Sustainability
Companies are accumulating AI operations debt faster than they realize. The rush to deploy is creating infrastructure complexity that traditional DevOps can't handle.
Read moreIndustry & Insights
3The AI Code Quality Cliff: Why Enterprise Development Just Crossed a Dangerous Threshold
GitHub's latest Copilot Enterprise features signal a tipping point where AI generates more enterprise code than humans write, but QA infrastructure remains dangerously outdated.
Read moreWhy Your AI Budget Is About to Break (And It's Not Your Fault)
Microsoft's latest AI pricing changes expose the uncomfortable truth: enterprise AI costs don't scale like traditional software, and CFOs are demanding answers.
Read moreMystery Shopper Testing for Enterprise AI: Making the Business Case
How to quantify the ROI of adversarial AI testing and convince your leadership that proactive chatbot QA saves money.
Read more