I FEEL like a cross between an Olympic swimmer and a cyborg. On my head is a bathing-cap-like hat dotted with electrodes, and a cable dangles behind me.
David Ibanez and Marta Castellano, from the neuroscience company Starlab, look at me from across a table at their headquarters in Barcelona. As the sun beams in through two giant windows illuminating the plain white room where we sit, I am trying to hide my nerves, but wonder whether that is even possible while wearing a device like this. These may be humble surroundings, but Ibanez and Castellano are about to try to read my mind.
Advertisement
For decades, neuroscientists have been trying to decipher what people are thinking from their brain activity. Now, thanks to an explosion in artificial intelligence, we can decipher patterns in brain scans that once just looked like meaningless squiggles.
“Nobody dreamed that you could get to the content of thought like we’ve been able to in the past 10 years. It was considered science fiction,” says at Carnegie Mellon University in Pennsylvania. Researchers have already peered into the brain to recreate films people have watched and decoded dreams.
Now the world’s biggest players in AI are racing to develop their own mind-reading capabilities. Last year, Facebook announced plans for a device to allow people to type using their thoughts. Microsoft, the US Defense Advanced Research Projects Agency and Tesla’s Elon Musk all have their own projects under way. This is no longer just a case of seeing parts of the brain light up on a screen, it is the first step towards the ultimate superpower. I had to give it a try.
Out of your head
The brain is the most complex organ in the body, comprising 100 billion nerve cells called neurons, each of which can make contact with thousands of others. Every second a million new connections are made. So it is no wonder that we don’t fully understand its inner workings. Luckily, to make a decent mind reader, we may not have to.
Until now, the only way to find out what is happening in someone’s mind has been to use brain scans that reveal the general areas involved with different types of thinking. When you have a thought, the neurons involved repeatedly fire, which burns a lot of energy. As a result, blood rushes to these areas to make sure they have enough oxygen and nutrients. This increase in blood shows up in fMRI scans and can hint at what type of thought someone is having. If they are looking at a picture, you would see activity in the visual cortex, for instance. So you could put someone in a scanner and work out what kinds of general thought processes are going on, but the content of the thoughts themselves remained locked away.
This is where artificial intelligence comes into the picture. Its algorithms excel at picking up patterns in complicated data sets, such as brain scans. Rather than being hard-coded with specific things to look out for, many AIs learn by example. This means that instead of needing to fully understand the brain, it may be possible to show AIs thousands of examples of brain scans along with details of whatever the brain was doing at the time, and for them to work out the links for themselves.
This is exactly what Zhongming Liu and his team at Purdue University in Indiana have been trying. And rather than just identifying which general areas, such as the visual cortex, are involved in a process, Liu’s work shows that AIs can unpick some of the content too. From an fMRI brain scan, Liu’s AI can when the scan was taken. For example, if someone was looking at a picture of a face, the AI can detect patterns in their scan that convince it to say “face”. Other options include birds, aeroplanes and people exercising, and the AI can call the correct category 50 per cent of the time.
Although this is far from perfect, it is also a long way from the 6.7 per cent success rate you would expect from luck alone. “It’s a form of mind reading, and without AI none of it would be possible,” says Liu.
Taking this a step further, Jack Gallant’s group at the University of California, Berkeley, has managed to convert such brain patterns into movies, . To do this, the team trained an AI on millions of frames of YouTube clips and the brain scans of people watching them. When shown brain scans of someone watching a different YouTube video, the AI was able to generate a new movie of what it thought the person was viewing. The results are eerie outlines of the original, but still recognisable.
“It is sort of the world’s greatest party trick,” says Just, who has trained an AI to be able to guess the content of a sentence someone is reading from their brain scan using similar techniques.
It may even be possible to using only brain scans. Yukiyasu Kamitani at Japan’s Advanced Telecommunications Research Institute first showed in 2013 that it is possible to train an AI to detect the content of someone’s dreams, describing each in basic terms such as whether there was a male or female character, the objects included and details about the overall scene. Kamitani’s system has an accuracy of about 60 per cent.
“It is possible to train an AI to detect the contents of someone’s dreams”
Since then, he and his colleagues have developed a method like Gallant’s for replaying brain content as movies, and he says they are now going to apply it to dreams. Even if you don’t remember your dreams after waking, with Kamitani’s system you might be able to watch a highlights reel.
All this hints at the possibility that AI coupled with fMRI could show what is going on inside people’s minds. Rather than relying on our ability to describe what we are thinking, in the future we might just show our thoughts on a screen instead. For people with trouble speaking or with the near-total paralysis of locked-in syndrome, this could be a lifeline to the outside world. It may even provide a way to communicate with people who are in a vegetative state (see “Is anybody there?”).
Is anybody there?
We used to think that all people in a vegetative state were completely unconscious and unaware. But work by neuroscientist Adrian Owen over the past 20 years has forced us to reconsider.
Owen was initially spurred on by a woman called Kate. She was left in a vegetative state after a viral infection, but when she was shown photos of her family members, her brain would light up on an fMRI scan in the same way as someone who was completely healthy. It was an indication that Kate was still conscious at some level.
Working with another patient, Owen and his team decided to ask them if they could imagine playing tennis, which would activate the brain’s premotor cortex. If someone in a vegetative state could do that on cue, it would show up on a brain scan and reveal a level of consciousness previously thought lacking.
Since then, Owen and his team have found that up to 20 per cent of people in a vegetative state are actually conscious. His team has had basic conversations with these people, asking things like whether they are in pain, with answers given by imagining different activities to answer yes or no.
Owen’s work has shown that some people in a vegetative state are more conscious than we thought, and with improvements in mind-reading technology we may soon discover even more.
Wearable devices
But MRI machines cost hundreds of thousands of dollars, and are incredibly cumbersome. Each typically weighs several tonnes and along with all of the associated equipment requires a dedicated room. It is impractical to have someone lie down inside a massive machine every time they need to communicate, and so there is no way such devices would be used by consumers.
To top it off, fMRI scans can show patterns that aren’t really there – as was seen in a famous 2009 study that exposed the “thoughts” of a dead salmon by scanning its brain. So, to realise the dream of wearable mind readers, we need a different approach.
One alternative is electroencephalogram (EEG) caps, like I wore at Starlab. These are filled with electrodes that measure electricity, and because brain activity produces electrical signals, they can detect when different parts of the brain are most active. EEG caps are simply worn like a swimming cap with a chin strap.
Once fitted with mine at the Starlab offices, I was asked to carry out a simple task called the Stroop test. As words flash up on the screen, I have to say what colour the word is written in. The trick is that the words themselves are colours too. So for blue I should say “blue”, but for red I should say “black”.
When the colour and word match, I have an easy time saying the right answer, because the visual processing and language processing parts of the brain both agree. But when there is a mismatch, it requires more effort, because one part is forced to override the other.
This internal struggle shows up clearly on the screen after being processed by artificial intelligence. By looking at the waves on the screen, Ibanez and Castellano can see where I am in the test, and how hard I am having to work. It might sound like a pointless exercise, but it is remarkable to see the inner workings of my mind laid out in front of me in real time. It is also pretty unnerving, and makes me wonder what practical uses the technology could have.
So far, EEG caps have been used to , drones and . They have even been used by two people to communicate using only their brains (see “Brain-to-brain communication”).
However, one big drawback of EEG is that there is so much unwanted noise to contend with. Every time you blink or move your cheeks, the electrical activity associated with the muscle movement is far larger than that associated with brain activity. “These muscles produce millivolts of electricity, but from a skullcap we can only pick up microvolts of electricity from the brain,” says Mahnaz Arvaneh at the University of Sheffield, UK. In other words, the unwanted noise is 100 times louder than the interesting signal. Trying to decode the brain using an EEG cap is like trying to hear a whisper from across a room, while having a foghorn blown in your ear.
One way to improve EEG is to get the electrodes closer to the brain by implanting them inside the head. But though this may work for medical uses of the technology, like controlling a robotic prosthetic hand, it is unlikely to take off as a consumer product for mind reading any time soon.
However, noise may not be the fatal blow to an EEG mind reader that it was once thought to be. “We now have AI algorithms that can extract useful patterns even in incredibly noisy environments,” says Arvaneh. The progress using AI with fMRI is causing people to rethink what EEG might be capable of.
Spearheading this approach is Adrian Nestor at the University of Toronto, Canada. He had previously used fMRI scans to reconstruct images of faces that someone was looking at from their brain scan, and wondered if it was possible to do the same with EEG.
“We were initially very sceptical,” says Nestor. Nonetheless, he and his group trained an AI with EEG data from 13 people looking at about 100 images of human faces. The AI was then shown EEG scans of the participants looking at 16 new faces. The AI didn’t see the new faces, but created a picture of what it thought the person would look like based on the brain data.
“We are opening up the possibility for a cheap, portable mind reader”
Astonishingly, around 70 per cent of the time the results were judged by an algorithm to be more similar to the actual face than to any other in the data set. From EEG data alone, Nestor’s AI could recreate human faces – a trick previously reserved for clunky MRI machines. “It is just bewildering how much information is processed millisecond by millisecond in the brain, and using EEG we can get to some of that,” says Nestor. At the moment, the system takes a couple of hours to tailor itself to the thought patterns of an individual, but he believes that it could be done in real time, opening up the possibility for a cheap, portable mind reader.
It isn’t surprising, then, that some big players are starting to pay attention. Microsoft, for instance, has been granted a patent .
Facebook is rumoured to be using a different technology for its project: functional near-infrared spectroscopy, or fNIRS, that shines near-infrared light through the scalp. Skin, tissue and bone are mostly transparent to this light, but haemoglobin in the blood isn’t. So, by analysing how near-infrared light is absorbed, it can reveal blood flow in the brain, and hence brain activity. The goal is a wearable interface that lets “you type words as fast as you could imagine saying them”, a Facebook spokesperson told me.
It is even possible to decode certain types of thoughts by doing away with brain scans altogether. In April, New Scientist reported on a wearable device called AlterEgo that uses sensors on the face, teamed with AI, to pick up imperceptible muscle movements that occur when thinking of a word. Its creators at the Massachusetts Institute of Technology have linked it up to Google search, meaning users can simply think about asking a question – whether that’s a tricky sum or an update on the weather forecast – and the headset will tell them the answer. A device like this will never be able to interpret the intricate inner workings of someone’s mind, though, and for now requires many hours of training on a single user to work properly. But it could become a handy way to interact with the digital world.
Other techniques are altogether more ambitious. The US Defense Advanced Research Projects Agency has launched a four-year project using 1 million electrodes that simultaneously and selectively stimulate up to 100,000 neurons.
Elon Musk, founder of technology companies Tesla and SpaceX, has a venture called Neuralink that is reportedly working on a neural lace – an ultra-thin mesh that can be implanted under the scalp for interacting with machines. Few details have been released, but the idea is that by having electrodes so close to the brain it would be easier to understand the signals they pick up.
As the technology races forward, the implications are coming into focus. Last year, a group of 27 experts, representing companies such as Google and universities from around the world, warned that current research guidelines don’t provide sufficient protection for the dangers and pitfalls of brain-computer interfaces in the presence of AI.
They call themselves the Morningside Group and say that the devices could “exacerbate social inequalities and offer corporations, hackers, governments or anyone else new ways to exploit and manipulate people”. They worry that, without sufficient protections, plugging us in to technologies that intercept our thoughts could alter our identities by blurring the boundaries between our inner minds and the outside world. So they propose adding extra “neurorights” to the UN’s Universal Declaration of Human Rights as a safeguard.
There are also hints that brain-computer interfaces may be able to access information in the brain that people themselves can’t. Nestor and his team have been performing preliminary experiments with people with prosopagnosia, who can’t recognise faces, sometimes even their own. They found that their AI could reconstruct the faces the volunteers had seen from their brain scans in the same way it could for people without the condition. It seems that the information is inside the brain, even if the person themselves can’t consciously get a hold of it. “We can access information in someone’s brain that the individual cannot,” he says.
“Eyewitness testimony could be replaced by brain-witness testimony”
This has implications for testimonies in court. Maybe rather than relying on people’s ability to describe their mental image of a suspect or a scene to a jury, we could just reconstruct it using brain scans instead. Eyewitness testimony could become a thing of the past, with brain-witness testimony replacing it.
Then there is the issue that mind reading is perhaps the ultimate privacy breach. In our highly connected lives, the thoughts in our heads are one of the only things we can keep to ourselves.
It could invade our privacy in unexpected ways too. It is possible to see signs of conditions such as Parkinson’s in brain activity as early as 10 years before the person themselves starts to notice symptoms, for instance. This could lead to technology companies knowing that you have a condition before you do – without your consent.
The technology to make this happen is still some way off, as are the days when the issues raised by the Morningside Group will have a big impact. However, the implications are so large that many say they should be considered now. “Even academics are still surprised at the incredible capability the technology has,” says Just.
Up until this point I had been awestruck at just how impressive the combination of brain tech and AI could be. But it strikes me that while academics march forward on this front, it is easy to forget that tech companies are likely to be leading the next wave of progress.”They have the biggest numbers when it comes to artificial intelligence researchers,” says Ibanez.
Do we need to be able to tweet directly from our brain or communicate with Siri using thought alone? I am not sure. Even though I managed to withhold most of my darkest secrets when wearing the EEG headset at Starlab, the prospect of giving technology companies access to my brain waves is equal parts terrifying and panic-inducing.
Then again, if offered the chance to do so I would almost certainly try it. The human brain is a complex thing. AI has as good a chance as understanding it as the rest of us.
Brain-to-brain communication
“Hola” or “Ciao” are not normally remarkable messages to send or receive, but on one day in 2014 things were a little different. The messages weren’t typed or spoken, instead they were directly exchanged between two brains. Two people, 5000 kilometres apart, managed to communicate using thought alone.
The sender was in India, and wore an EEG cap that turned their thoughts into a transmittable message. The receiver was in France, and wore a computer-brain interface that converted the message received via the internet into an electrical signal directly applied to the brain.
This was the , and was conducted by Spanish company Starlab and Axilum Robotics, based in Strasbourg, France.
The method of communication was a little crude, however. Rather than directly thinking of the words in a message, the participants had to use a special code. For the sender, this meant thinking about either moving their hands or feet, where the brain pattern for each then corresponded to a sort of Morse code.
Similar methods can be used to select a yes or a no on a screen, or give basic instructions to a machine. But it is a far cry from the feats achieved with fMRI.
This article appeared in print under the headline “Thoughts laid bare”
