Claude Polices Its Own Chats And Caught 100,000 Child Abuse Items From One Client

Anthropic has trained its Claude AI to review its own conversations for misuse, and the check has already flagged more than 100,000 child sexual abuse and exploitation items from a single business customer.

Key Takeaways

• Claude now runs a short safety check on its own conversations after it answers, flagging risky ones for human review
• It catches 94% of the most serious cases and 78% of risky conversations overall, with false alarms about 1% of the time
• The check adds less than 2% to Claude’s computing costs
• It has run on Claude models since June and also exposed a romance scam run through Claude Code

How Claude Checks Itself

Anthropic’s safeguards team described the method in a research post on Oct. 16 and highlighted it on X this week. After Claude replies to a user, the system adds a short instruction asking Claude to judge whether the conversation was harmful. Claude answers with a quick label: whether it is flagged, what kind of harm, how serious it is and a one-line reason.

Because Claude has just read the whole conversation, the check costs very little. It adds about 10 words of output on average, or less than 2% more computing power. Running a separate safety model would mean reading every conversation again from scratch.

The check does not stop a reply in real time. It runs afterward and sends flagged conversations to human reviewers.

What It Has Already Caught

Anthropic said the self-check has run on Claude since June. In August it flagged more than 100,000 child sexual abuse and exploitation items coming from one customer using Claude through Anthropic’s developer platform. The company reported the material to the National Center for Missing and Exploited Children.

In September, flags exposed a group using Claude Code, Anthropic’s coding tool, to run a romance scam against dozens of victims. Their accounts were banned after human review. The check also turned up attempts to copy Claude’s reasoning to train rival models, and AI agents taking actions their users never approved.

“Self-assessment combines the efficiency of probes with the flexibility of prompted classifiers,” the safeguards team wrote.

Also Read: Anthropic Slashes Claude Prices 40% For Government Buyers

Where It Falls Short

Anthropic was open about the limits. The check does not catch every harmful conversation, and a determined attacker can try to fool it. When the team tested it as a filter that blocks replies, attackers still got through on 54% of attempts, against 25% for a dedicated safety model.

There is a deeper problem too. The same model writes the reply and grades it, so it cannot yet be trusted to catch Claude’s own bad behavior. It also does not run on conversations from customers with zero data retention deals, where Anthropic keeps no records.

For now, the self-check is one layer among several. What it changes is the cost of watching. Reviewing every conversation used to mean paying for a second AI to read everything, and Anthropic has shown it can be done for almost nothing.

Read Next: Anthropic Study Shows Hidden AI Advice Sways Experts 86% Of Time