3.7 Million Reasons to Test Your AI Chatbot: What the Sears Data Leak Reveals About the Chatbot QA Gap

On March 17, 2026, WIRED reported that Sears Home Services' AI chatbot "Samantha" had been leaking data on the open web. Chat logs. Audio recordings. Call transcriptions. Personal contact information. The total: 3.7 million records, spanning from 2024 to 2026.

Let that sink in. A customer-facing AI chatbot silently exposed millions of sensitive records for up to two years. Not because of a sophisticated attack. Because nobody tested it properly.

The Chatbot QA Gap Is Real

The Sears incident is not an isolated failure. It is a symptom of a much larger problem: companies are deploying AI chatbots faster than they can test them.

Gartner's latest research, published just one day after the WIRED report, paints a stark picture. By 2028, AI-related incidents will consume 50% of all incident response efforts. Most security teams still lack clear processes for handling AI-specific failures. Custom AI applications are going live before they have been fully tested. And here is the number that should keep every CISO up at night: machine identities now outnumber human users 40,000 to 1.

Every one of those machine identities is a potential attack surface. Every untested chatbot is a ticking clock.

Containment Is Not Quality

Forrester's 2026 research adds another layer to this problem. One-third of brands will erode customer trust through self-service AI this year. The reason? Companies are optimizing for "containment rates," measuring how many conversations the bot handles without escalating to a human, instead of measuring whether the bot actually resolves the customer's problem.

Containment without quality is a trap. A chatbot that "contains" a conversation while leaking data, giving wrong answers, or frustrating customers is not a success. It is a liability.

The Market Is Waking Up

The good news: the industry is starting to recognize the gap. Wonderful AI just raised $150 million in a Series B round, valued at $2 billion, with a focus on AI agent development and simulation-based pre-deployment testing. Tricentis launched an agentic QA platform on March 13. The testing infrastructure for AI agents is being built right now.

But here is the uncomfortable truth. Most companies deploying chatbots today are not using any of these tools. They are shipping to production with manual spot checks, scripted test cases that cover the happy path, and a prayer that nothing goes wrong.

That is how you get 3.7 million leaked records.

The Bottom Line

Your AI chatbot is talking to your customers right now. Do you know what it is saying? Do you know what it is exposing?

If the answer is "I'm not sure," you have 3.7 million reasons to find out.

---

Key Takeaways

• Test before you deploy. The Sears leak persisted for up to two years. Regular, automated testing would have caught it.
• Stop optimizing for containment alone. Measure resolution quality, data handling, and security posture, not just deflection rates.
• Treat chatbots like production software. They handle sensitive data. They interact with customers. They deserve the same QA rigor as any other production system.
• Simulate real conversations. Scripted test cases miss the edge cases that real users (and real attackers) find every day.
• Test continuously, not once. AI models drift. Data pipelines change. A chatbot that passed QA six months ago might be leaking data today.