Visualizing Thought

The revolution behind the revolution

In 1984, as I began to research the implications of the emerging digital revolution, I visited Ron MacNeil at the Visible Language Workshop, part of the MIT Media Lab. There he said something to me which profoundly changed the way I viewed what was happening: If you can think it, he said, the computer can do it.

Previously, like many, I had been thinking of the computer as a tool which could help to accomplish certain tasks more efficiently, such as a word processor supports writing or a graphics software might assist with lay-outs. But I had not thought of the computer as a collaborator that could go so much farther, investigating what could be possible based upon an idea.

As the saying goes, to a hammer everything looks like a nail. But for a computer, this I would come to understand makes little sense.

For the last several years I have been following with considerable interest and alarm the research on how to visualize what people are seeing, or even just thinking, by monitoring their brain activity and then reconstructing the images as they perceived them with the help of artificial intelligence. So that now, using a variety of technologies, including non-invasive ones, it is already possible to record brain functions and output them with increasing accuracy so as to replicate the image that a person or animal had been looking at.

For example, in an experiment at Radboud University in the Netherlands participants were shown photorealistic synthetic faces and then, using an fMRI machine and artificial intelligence, what they had seen was nearly replicated directly from their brains (see below). AI researcher and cognitive neuroscientist Thirza Dado, who led the 2022 study, believes that “we can train the algorithm not only to picture accurately a face you’re looking at, but also any face you imagine vividly, such as your mother’s.” She also reports, according to the Daily Mail, that they are working on “developing brain-implant cameras that will stimulate people's brains so they can see again.”

Or the reconstructed images (see below) of what a monkey perceived are in the bottom row, presented under the original images that had been shown to the monkey.

Work being done to visualize what a person had only been thinking may make it possible for someone’s dreams to be recorded and viewed. This was the obsession of the character Claire Tourneur played by Solveig Dommartin (below), in Wim Wenders’s extraordinarily prescient 1991 film, Until the End of the World. In the film Tourneur becomes addicted to viewing her dreams and suffers a highly painful withdrawal when the batteries for her viewer no longer work.

Similar work is being done therapeutically to record and express thoughts that can be output as words so as to allow those who suffer from the effects of a stroke or other physical limitations to better communicate. Currently, Meta’s Brain2Qwerty, for example, functions by reading brain activity as the person thinks they are typing on a QWERTY keyboard and turning the output into coherent sentences.

---

So that one might then also imagine a new form of photography in which there is no camera, but a non-invasive recording of mental processes that are then output as image. This might involve aspects of the unconscious including dreams, daydreams repressed memories, etc., actualizing the subjectivity of perception. But this would profoundly differ from a 20th-century notion of photography as representing a reliable reference point and shared reality. And, as importantly, it could also threaten the privacy of an individual’s own thoughts, especially if used coercively.

This might also allow a direct avenue for various kinds of targeted input, such as for movies, or advertising, or artwork, or governmental propaganda. The possibilities for a better understanding of the human psyche are enormous, as are the terrifying possibilities of mind control. It is here that one is reminded of the writings of Orwell and Huxley, or of science fiction writers such as Philip K. Dick.

Numerous questions arise. For example, would the ability to read thoughts directly from the brain lead to more invasive forms of interrogation in a courtroom or in a classroom, or of a more generalized thought control? What if the outputs are mistaken? The many problems with the internet, in comparison, might start to seem comparatively benign.

As with previous revolutions of enormous portent, there is far too little discussion of the potential consequences for both individuals and societies. In a BBC article earlier this year, Yu Takagi, an associate professor at Nagoya Institute of Technology in Japan, is quoted as being “excited about some of the potential applications of these approaches. They could recreate the auditory and visual hallucinations of psychiatric patients such as schizophrenics to better understand their conditions, he says. The techniques could be used to recreate what animals experience as they process the world, or even to reconstruct dreams.”

Dreaming seems to be a focal point. “‘Many people are asking about that,’ says Takagi with a laugh. He says he would like to recreate dreams one day, but right now, it remains extremely complicated.”

The BBC writer, Laurie Clarke, continues: “Some research has even raised the prospect of direct brain-to-brain communication, including with multiple people at once, although the ethical implications and human rights issues related to devices that allow this have still to be fully unravelled.”

And we may not have to wait very long: “For those hoping it might also be possible to stimulate visual or auditory experiences in the brain in the name of entertainment, Takagi advises patience. While this is theoretically possible, he says technical limitations means it probably won’t happen for another 10 to 20 years.”

There is much to be considered in the meantime, much of it dependent on how the technology develops and on how both corporations and governments respond. How is an individual’s privacy to be protected? What laws need to be passed? What kinds of consent need to be considered? What will it mean to be human? The Battle for Your Brain: Defending the Right to Think Freely in the Age of Neurotechnology by Nita A. Farahany is a good introduction to the gravity of what we are facing.

As with other revolutions, some of these issues are not completely new. There is, for example, a rather odd experiment in the 1960s (see the image above) when a Spanish scientist, Dr. Jose Delgado, stopped an enraged bull as it was charging with a radio transmitter that was used to electrically stimulate the bull’s brain. And much work has been done to help people control the movement of their limbs when they are otherwise unable to do so.

But now these interventions are being applied, some in real time, to internal thought processes. And while so many of us, including myself, have focused on AI-generated photorealistic images that simulate photographs and videos, concerned about the undermining of photographic credibility, the current research on recording and outputting internal processes is at least as profound in its ramifications.

And, as with a magician performing a trick, we may well be looking only at the hand that we are supposed to be seeing, not the one that is operating behind the scenes. It certainly behooves us to be aware of both.