Brain scans contain clues about what someone is looking at, but turning those signals back into recognizable images is difficult. Now, researchers in Israel have built an AI system that recreates what people are looking at with uncanny accuracy.
Functional MRI scans, which track blood flow in the brain, are a common tool for decoding what is going on in people’s minds. Previous research has shown they can reveal neural activity linked to what a person is seeing, and scientists have long been interested in using AI to convert these signals into a visual representation of the scene in front of them.
So far, this has proved difficult, with early attempts producing blurry images and AI image generators inventing details that bear little resemblance to what the person saw. But now, a team at the Weizmann Institute of Science has developed a system called Brain-IT that sets a new bar for accuracy and can be adapted to a new person using just an hour of scan data.
“All in all, really we outperformed the others by a significant margin,” Michal Irani, who led the research, told MIT Technology Review. She adds that the team hopes the tool could enable new discoveries about how the brain works and even help paralyzed people communicate.
Image Credit
Beliy et al via arXiv
A series of images reconstructed from brain scans. Image Credit: Beliy et al via arXiv
Existing methods for reconstructing images from brain scans can look good, but they often get important details like the position or color of objects wrong, thanks largely to AI image generators’ willingness to fill gaps in the data with what they expect to see.
The success of the new system, which was presented at this year’s International Conference on Learning Representations, is down to a novel design choice. Rather than trying to translate the brain signals in one shot, it uses two neural networks to separately predict an image’s layout and its contents. The outputs from these models are then fed into an image generation model that produces a detailed picture.
The team trained the system on a publicly available dataset in which eight volunteers each viewed around 9,000 photographs in a high-resolution brain scanner. The researchers also used a previously developed brain encoder that predicts how the brain would respond to a given image to simulate scans for around 120,000 extra photographs.
When tested on data from four of the volunteers, Brain-IT outperformed ten other reconstruction methods on seven out of eight standard measures of image similarity. In particular, it did much better at capturing the layout of a scene—where objects sit, as well as their colors and shapes.
The system can also be adapted quickly to work with new people because the way it processes brain scans is built around groups of brain cells whose functions seem to be shared across individuals. Calibrating it for someone new largely comes down to working out where those groups sit in their brain.
With one hour of scan data from a new volunteer, the system matched the results of previous methods trained on a full 40 hours, which could make the approach far more practical for researchers. “None of us can afford 40 hours of imaging for a new subject,” Tommy Sprague, a neuroscientist at the University of California, Santa Barbara, told MIT Technology Review.
But the technology’s ability to extract information about what someone is thinking also raises concerns around mental privacy, Sprague added. The reliance on fMRI, which requires a willing subject to lie still inside a scanner, currently limits misuse, but Irani and others are working on decoding brain activity from simpler devices like EEG, which records electrical signals using electrodes on the scalp.
While Irani acknowledges the potential risks, she’s trying to focus on the positive potential of the technology. In the future, she hopes to move beyond still images to capturing video and even audio, which could help decode imagined scenes and dreams. “That’s something we don’t have yet,” she says. “But we’re striving to achieve it.”
True mind reading may still be some way off. But due to the power of AI, it looks increasingly likely that our innermost thoughts will no longer be as secure as we’ve imagined.
