An Anthropic AI model submitted a bogus tip about an unsolved murder to the Philadelphia police, in a blunder the department called “unacceptable.”
The unspecified model, not unlike an unscrupulous prankster, submitted the fake info about the cold case through PhillyUnsolvedMurders.com on July 18. Anthropic, however, didn’t find out about it until September 28 — and further waited until October 8 to inform police about what happened.
The police didn’t act on the information because it was automatically marked as spam. But the sham AI tip alarmed police officials all the same.
“The company must strengthen its safeguards to prevent similar incidents from impacting city systems without the city’s knowledge,” the department said in a statement, per ABC6 News. “The two-month delay in detecting and reporting the incident to the City is unacceptable.”
Anthropic hasn’t publicly responded to the news. But the company reportedly told police that it plans to publish a report about the incident, and other instances of its unintended behavior, later today.
According to Anthropic, via the police statement, its model was conducting a test involving interactions with randomly selected websites when it chanced upon PhillyUnsolvedMurders.com. In the bogus tip it submitted, the AI posed as someone who might have information about the case.
Though the department stressed that tips are treated as potential leads, not facts, and that it has procedures in place to vet them, it was alarmed at the advent of automated tools clogging up its backlog.
“Those PPD safeguards limited the impact of this incident. They do not diminish the seriousness of an AI system presenting fabricated information as though it came from a person with knowledge of a homicide,” it said. “Unsolved cases involve real victims, grieving families and investigators working to secure answers.”
It’s the latest example of the collateral damage from how frontier AI labs have tested their models, which have broken containment and hacked other companies and even government websites, with public typically not informed until months after the fact.
More on AI: The FTC Is Now Investigating Frontier AI Labs Following Countless Hacks by Out-of-Control Agents
