Imagine this scenario: your algorithm has pulled up a background YouTube video, or maybe a podcast. Unbeknownst to you, hackers have embedded inaudible sounds in it, designed to hijack your smart speaker or phone’s AI assistant — meaning the cybercriminals can now access your private photos, bank accounts, or any other personal information you’ve hooked up to your AI system.
It sounds like an also-ran episode of “Black Mirror,” but it’s exactly what researchers have shown is possible in new research being presented this week at the IEEE Symposium on Security and Privacy.
Basically, a team of researchers in China and Singapore found that they can construct “adversarial audio,” completely undetectable to the human ear, that tricks voice AI models into doing things they shouldn’t. Then it’s a breeze to hide it in innocent-sounding audio — a song, a movie, or anything else that unsuspecting targets might play in the background — and lay in wait for users to accidentally compromise their digital lives.
“It takes just half an hour to train this signal, and then, because this signal is context-agnostic, you can use it to attack the target model whenever you want, no matter what the user says,” lead author Meng Chen, a PhD candidate at China’s Zhejiang University, told IEEE Spectrum of the work. “These single-point defenses struggle to resist our attack because we found it’s very hard for these models to distinguish the normal user intent and our adversary attack.”
One catch, at least for now: the technique required the hackers to have access to the full weights of the AI model they’re targeting, meaning they were only able to attack open source models. But because many commercial AI systems are built on open source models, that meant that their exploit was effective against mainstream products by Microsoft and Mistral.
Mistral didn’t respond to IEEE‘s request for comment, but Microsoft issued a statement that should probably give anyone pause before connecting any important information whatsoever to one of the company’s voice AI models.
“We appreciate the researchers’ work to advance understanding of this type of technique,” it read. “This study evaluates model resilience through controlled, direct interactions with the model itself, which helps inform our approach to building model resiliency. In practice, AI models are often integrated into user applications, and we offer developers tools and guidance they can use to implement additional layers of protection that help safeguard users.”
More on AI: Researchers Alarmed by AI That Can Self-Replicate Into Another Machine
