An AI “mind-reading” tool can reconstruct what you’re looking at based on a brain scan. That is the claim from scientists at the Weizmann Institute of Science in Rehovot, Israel, who say their system can take a brain scan and produce a recognizable image of what a person was seeing.
The system works in both directions. It can guess what a person is looking at from a brain scan, and it can also predict a person’s brain activity from an image. The team, led by computer scientist Michal Irani, says the method could eventually be used to help people with locked-in syndrome communicate or to recreate the content of dreams.
The research was presented at the Cognitive Computational Neuroscience conference in New York last month.
How the Scanner Reads the Mind
Functional magnetic resonance imaging, or fMRI, tracks the flow of oxygenated blood through the brain. Areas that light up on an fMRI scan are thought to be active at that moment. The resolution is limited: each highlighted “voxel” covers around three cubic millimeters, containing some 16,000 neurons.
Irani and her colleagues used newer datasets with a higher resolution, where each voxel covers around one cubic millimeter. Those scans showed how volunteers’ brains reacted to viewing various images.
Previous attempts produced blurry, hard-to-make-sense-of reconstructions. The new system aims to do better.
Training the Model
The team started with publicly available brain scan data from volunteers who had been shown hundreds of images while lying in fMRI scanners. Irani and her colleagues trained an AI model on data from eight people who each saw around 9,000 images during a high-resolution fMRI session.
The model has two branches. One predicts the structure of an image — where the colors are. The other predicts the content — a bunch of bananas on a plate, for example. A diffusion model, which cleans up noisy pixel data to produce clearer images, combines the predictions into a final reconstruction.
The early results were not promising. Show a person a banana, and the model might generate an image of a banana, but the shape and position would be wrong. Irani says the reconstructed image would not have the same structure or the same position as the original.
The Encoder Trick
To improve the models, the team needed more data than was actually available. So they trained another model in the opposite direction — an encoder that predicts brain activity from an image.
They then used the encoder and decoder together to refine both tools. The process starts with a new image, say of a leopard. The encoder predicts what an fMRI scan would look like if a person saw that picture. The decoder reconstructs the image from the predicted scan.
At first, the reconstructed image probably won’t look much like a leopard. But repeated training eventually leads to dramatic improvements.
What the Brain Shows
By combining data from multiple studies, the team identified brain regions that seem to share functions across all individuals. One region responded to images of food. Another responded to images of sports.
Irani, a computer scientist, says she is now working with neuroscientists to see if the tools can reveal new things about the brain.
The resulting “universal brain encoder” can work on a scan from a new person with minimal calibration. Previous tools typically require about 40 hours of fMRI data on a new person before they can predict what they are seeing.
Irani’s decoder needs only one hour of data.
The Ethical Warning
Not everyone is cheering. Some scientists warn that a similar approach could be used to reveal a person’s inner thoughts and mental imagery without their consent.
Tommy Sprague, a neuroscientist at the University of California Santa Barbara, called the results “very impressive.” But he added a caution: “If there’s a way to surreptitiously extract information about what you’re thinking about, then…150 years of sci-fi can come true anytime, and that’s worrisome in a lot of ways.”
Judy Illes, a neuroethicist and professor of neurology at the University of British Columbia, described the work as “magnificent.” She says the idea of using this approach to help people with neurological conditions “therapeutically is tremendously exciting.”
Why This Matters
The system could speed up neuroscience research. Sprague noted that none of the researchers can afford 40 hours of imaging for a new subject. It’s something like $600 to $1,000 an hour.
The new tool could cut that cost dramatically.
The technology is still in development, and the team’s findings were presented at a conference rather than published in a peer-reviewed journal. That means independent verification has not yet happened.
Key Facts Box
- Model trained on: Data from eight people, each shown around 9,000 images
- New encoder training: Around 70% of images not originally paired with scans
- Previous tool calibration: About 40 hours of fMRI data per new person
- New decoder calibration: Only one hour of data
- Conference: Cognitive Computational Neuroscience, New York
- Cost of imaging: Something like $600 to $1,000 per hour
Schedule Table
| Stage | Details |
|---|---|
| Data collection | Volunteers shown hundreds of images in fMRI scanners |
| Initial training | Model trained on eight people, 9,000 images each |
| Encoder training | Model learns to predict brain activity from images |
| Decoder refinement | Encoder and decoder trained together, leopard example |
| Universal encoder | Works on new person with minimal calibration |
| Conference presentation | Cognitive Computational Neuroscience, New York |
The technology raises serious questions about privacy. If a mind-reading tool can reconstruct a person’s thoughts from a brain scan, who controls that information? How is it stored? Who gets access?
Illes’s excitement about therapeutic applications is tempered by Sprague’s warning. The ethical framework for this kind of technology is still being built.
For now, the science is still being tested. The decoder has been shown at a conference. The question is what happens next.
Source material: “An AI “mind-reading” tool can reconstruct what you’re looking at based on a brain scan,” MIT Technology Review.
Get the Notebook.
The day's best stories and every fresh verdict, in plain English, in your inbox by seven. One email a day, no more.