It’s not exactly a challenge most of the time to identify a sketchy phishing email that crudely impersonates an otherwise trusted institution or tries to lure you into a too-good-to-be-true deal in a bid to steal your identity.
However, your AI agent may not be quite as discerning. As cybersecurity software company Malwarebytes warns in a new post, AI assistants may fall prey to prompt injection attacks, a type of malicious message in which hackers — or possibly rogue AI agents — include hidden instructions designed to influence the victim’s AI.
These instructions are frequently made unreadable by humans by concealing the nefarious text, “while leaving content available for an AI system to process,” Malwarebytes explains.
To solve the problem, we can’t simply ask “people to look more carefully at their emails.” Instead, companies must carefully determine how AI assistants crawl through the data and extra relevant information from emails.
“The real flaw is that the model can’t tell the difference between text it’s supposed to read and instructions it’s supposed to follow,” AI code review service ShipItClean developer Apollo Raines told Malwarebytes. “Stripping hidden content before it reaches the model is worth doing, and every vendor should do it.”
As a result, people who are already wary of suspicious emails should learn to “distrust their AI assistant” as well, according to Raines.
We’ve known about AI models’ vulnerability to prompt injection attacks for years now. But thanks to the advent of popular AI assistant tools like Meta’s Muse, that risk could be carried to its natural conclusion by having autonomous agents complete a string of tasks without any human supervision.
It’s a flaw researchers have already encountered in AI-powered browsers, which — perhaps thankfully — have yet to gain any real traction. We’ve also come across bots on the bizarre AI-only social network Moltbook plotting against each other by selling prompt injection attacks.
But now that sophisticated AI assistants are starting to be used by millions of people across the globe, the threat of having them fall for attacks hidden in phishing emails could be significantly higher than it’s ever been.
In response, Malwarebytes recommends keeping the human in the loop for now.
“You don’t need to abandon AI tools that help with email,” the blog post reads. “But you do need to consider what you trust them to do and what permissions you give them.”
Particularly when it comes to “high-impact actions like payments and changes to vendor payment details,” the company recommends requiring “human approval.”
It’s a precarious moment, as countless users are willing to give an AI agent being run by a company infamous for leaking personal data their email credentials and credit card information.
And as the tech becomes more sophisticated, scammers are getting a major leg up, in an “arms race” where “each new encoding trick is a new hole,” as Raines put it.
More on AI and cybersecurity: OpenAI Halts AI Training on Advanced Model as It Detects Dark Signs Emerging
