“Reality” is constructed by your brain. Here’s what that means, and why it matters. What the science of visual illusions can teach us about our polarized world.

By Brian Resnick

Fix your gaze on the black dot on the left side of this image. But wait! Finish reading this paragraph first. As you gaze at the left dot, try to answer this question: In what direction is the object on the right moving? Is it drifting diagonally, or is it moving up and down?

Remember, focus on the dot on the left.

It appears as though the object on the right is moving diagonally, up to the right and then back down to the left. Right? Right?! Actually, it’s not. It’s moving up and down in a straight, vertical line.

See for yourself. Trace it with your finger.

This is a visual illusion. That alternating black-white patch inside the object suggests diagonal motion and confuses our senses. Like all misperceptions, it teaches us that our experience of reality is not perfect. But this particular illusion has recently reinforced scientists’ understanding of deeper, almost philosophical truths about the nature of our consciousness.

“It’s really important to understand we’re not seeing reality,” says neuroscientist Patrick Cavanagh, a research professor at Dartmouth College and a senior fellow at Glendon College in Canada. “We’re seeing a story that’s being created for us.”

Most of the time, the story our brains generate matches the real, physical world — but not always. Our brains also unconsciously bend our perception of reality to meet our desires or expectations. And they fill in gaps using our past experiences.

All of this can bias us. Visual illusions present clear and interesting challenges for how we live: How do we know what’s real? And once we know the extent of our brain’s limits, how do we live with more humility — and think with greater care about our perceptions?

Rather than showing us how our brains are broken, illusions give us the chance to reveal how they work. And how do they work? Well, as the owner of a human brain, I have to say it’s making me a little uneasy.

Where the conflict between perception and reality lies in the brain

My colleague Sigal Samuel recently explored the neuroscience of meditation. During her reporting, she found good evidence that a regular meditation practice is associated with increased compassion. That evidence, she writes, “feel[s] like a challenge, even a dare. If it takes such a small amount of time and effort to get better at regulating my emotions … am I not morally obligated to do it?”

Perception science, for me, provokes a similar question. If the science tells us our brains are making up a “story” about reality, shouldn’t we be curious about, and even seek out the answers to, how that reality might be wrong?

It’s not about doubting everything that comes through our senses. It’s about looking for our blind spots, with the goal of becoming better thinkers. It can also help with empathy. When other people misperceive reality, we may not agree with their interpretation, but we can understand where it comes from.

RELATED

Intellectual humility: the importance of knowing you might be wrong

To approach this challenge, I think it helps to know that the brain is telling us stories about the smallest things we perceive, like the motion of objects. But it also tells us stories about some of the most complex things we think about, creating assumptions about people based on race, among other social prejudices.

Let’s start with the small.

In 2019, Cavanagh and his colleagues Sirui Liu, Qing Yu, and Peter Tse used the above “double drift” illusion of the two dots to probe how our brains generate the illusory diagonal motion.

To figure this out, Cavanagh and his colleagues ran a neuroimaging study that compared how a brain processes the illusory animation with how it processes a similar, non-illusory animation. In this second animation, the object on the right really is moving diagonally. Trace it with your finger again.

With fMRI neuroimaging, which allows researchers to map brain activity, Cavanagh and his team could ask the question: Ifwe perceive each animation similarly, what in our brains makes that happen? What’s the source of the illusion in the first animation? “We want to find where the conscious perception diverges from the physical sensation,” Cavanagh says.

One possibility is that the illusion is generated in the visual cortex. Located at the back of your head, this is the part of your brain that directly processes the information coming from your eyes. Maybe the visual system “sees” it wrong. The alternative is that the visual system “sees” it just fine, but some other part of the brain overrides it, creating a new reality.

The experiment included only nine participants but collected a lot of data on each of them. Each participant completed the experiment (and was run through the brain scan) 10 times.

Here’s what the analysis found. That visual system in the back of the brain? It doesn’t seem fooled by the illusion. Each animation produces a different pattern of activation in the visual cortex. In other words, “the visual system thinks they are different,” Cavanagh says.

Okay, the visual system correctly “sees” these two animations differently. Then why do we perceive them as being the same?

The patterns of activation in the frontal lobes of the participants’ brains — the higher-level thinking area dedicated to anticipation and decision-making — were similar. That is: The front of the brain thinks both animations are traveling in a diagonal direction.

“There’s a whole world of visual analysis and computation and prediction that is happening outside of the visual system, happening in the frontal lobes,” Cavanagh says. That’s where the “story” of reality is constructed — at least in this one example, as evidenced by this one small study. (To be sure: Vision is a vastly complex system involving around 30 areas of the brain. There are other illusions that do seem to “fool” the visual cortex, because no story about the brain can be simple.)

But you don’t need an fMRI to conclude that somepart of your brain is overriding the plain truth about the path of the object. You can see it for yourself. “The remarkable thing is that — even when you are told what is happening — you still see it in the illusory form,” Justin Gardner, a Stanford University neuroscientist who wasn’t involved in this study, said in an email. “You can’t seem to consciously override the ‘wrong’ interpretation.”

The lesson: The stories our brains tell us about reality are extremely compelling, even when they are wrong.

We’re not seeing reality. Our vision runs 100 milliseconds behind the real world.

Why are we seeing a story about the world — a story — and not the real deal? It’s not because evolution made our minds flawed. It’s actually an adaptation.

“We don’t have the necessary machinery, and we wouldn’t even want it, to process carefully all of the amount of information that we’re constantly bombarded with,” says Susana Martinez-Conde, a neuroscientist and illusion researcher at SUNY Downstate Medical Center.

Think about what it takes to perceive something move, like the objects in the above animations. Once light hits the retinas at the back of our eyeballs, it’s converted into an electrical signal that then has to travel to the visual processing system at the back of our brains. From there, the signal travels forward through our brains, constructing what we see and creating our perception of it. This process just takes time.

“The dirty little secret about sensory systems is that they’re slow, they’re lagged, they’re not about what’s happening right now but what’s happening 50 milliseconds ago, or, in the case for vision, hundreds of milliseconds ago,” says Adam Hantman, a neuroscientist at Howard Hughes Medical Institute’s Janelia Research Campus.

If we relied solely on this outdated information, though, we wouldn’t be able to hit baseballs with bats, or swat annoying flies away from our faces. We’d be less coordinated, and possibly get hurt more often.

So the brain predicts the path of motion before it happens. It tells us a story about where the object is heading, and this story becomes our reality. That’s what’s likely happening with Cavanagh’s illusion. It happens all the time.

Don’t believe it? See for yourself. Here’s a simple illusion that reveals our visual system is a bit lagged.

It’s called the flash-lag illusion. The red dot is moving across the screen, and the green dot flashes exactly when the red dot and green dot are in perfect vertical alignment. Yet it’s incredibly hard to see the red dot and the green dot as being vertically aligned. The red dot always seems a little bit farther ahead.

This is our brain predicting the path of its motion, telling us a story about where it ought to be and not where it is. “For moving things — we see them ahead on their path of motion,” Cavanagh explains, “by just enough.” The illusion, he says, “is actually functional. It helps us overcome these delays and see things … where they will be when we get there.”

In Hantman’s view, what we experience as consciousness is primarily the prediction, not the real-time feed. The actual sensory information, he explains, just serves as error correction. “If you were always using sensory information, errors would accumulate in ways that would lead to quite catastrophic effects on your motor control,” Hantman says. Our brains like to predict as much as possible, then use our senses to course-correct when the predictions go wrong.

This is true not only for our perception of motion but also for so much of our conscious experience.

The stories our brain tells are influenced by life experience

The brain tells us a story about the motion of objects. But that’s not the only story it tells. It also tells us stories about more complicated aspects of our visual world, like color.AD

For some meta-insight, look at the illusion below from Japanese psychologist and artist Akiyoshi Kitaoka. You can observe your own brain, in real time, change its guess about the color of the moving square. Keep in mind that the physical color of the square is not changing. You might look at this illusion and feel like your brain is broken (I did when I first saw it). It is not. It just reveals that our perception of color isn’t absolute.

Color is an inference we make, and it serves a purpose to make meaningful decisions about objects in the world. But if our eyes acted as scientific instruments describing precise wavelengths of light, they’d constantly be fooled. Red may not appear red when bathed in blue light.

Our brains try to account for this. “We’re not trying to measure wavelengths, we’re trying to tell something about the color,” Sam Schwarzkopf, a vision scientist at the University of Auckland, says. “And the color is an illusion created by our brain.”

When we think an object is being bathed in blue light, we can filter out that blue light intuitively. That’s how many of these color illusions work. We use surrounding color cues and assumptions about lighting to guess an object’s true color. Sometimes those guesses are wrong, and sometimes we make different assumptions from others. Neuroscientists have some intriguing new insights into why our perceptions can diverge from one another.

You remember The Dress, yes?

In 2015, a bad cellphone photo of a dress in a UK store divided people across the internet. Some see this dress as blue and black; others see it as white and gold. Pascal Wallisch, a neuroscientist at New York University, believes he’s figured out the difference between those two groups of people.

Wallisch’s idea — which draws from earlier work by Rosa Lafer-Sousa, Katherine Hermann, and Bevil Conway — is that people make different assumptions about the quality of light that’s being cast on the dress. Is it in bright daylight? Or under an indoor light bulb? By unconsciously filtering out the color of light we think is falling on an object, we come to a judgment about its color.

Wallisch believes people who see this image differently are using different filtering schemes. Most interestingly, he suggests that life experience leads you to see the dress one way or the other.AD

His study of 13,000 people in an online survey found a correlation that at first seems odd. The time you naturally like to go to sleep and wake up — called a chronotype — was correlated with dress perception. Night owls, or people who like to go to bed really late and wake up later in the morning, are more likely to see the dress as black and blue. Larks, a.k.a. early risers, are more likely to see it as white and gold. What’s going on?

Wallisch believes the correlation is rooted in the life experience of being either a lark or a night owl. Larks, he hypothesizes, spend more time in daylight than night owls. They’re more familiar with it. So when confronted with an ill–lit image like the dress, they are more likely to assume it is being bathed in bright sunlight, which has a lot of blue in it, Wallisch points out. As a result, their brains filter it out. “If you assume it’s daylight, you will see it as white and gold. Because if you subtract blue, yellow is left,” he says.

Night owls, he thinks, are more likely to assume the dress is under artificial lighting, and filtering that out makes the dress appear black and blue. (The chronotype measure, he admits, is a little crude: Ideally, he’d want to estimate a person’s lifetime exposure to daylight.)

Has Wallisch solved the mystery of The Dress?

“The owls versus lark data seems quite compelling for explaining a large part of the individual differences,” Schwarzkopf says. But not all of it. “There are still lots of other factors that must have a strong influence here. It could be prior experience with the subject matter, or related to other aspects of people’s personality,” he says. “Yes, the dress continues to mystify.”

The mystery isn’t totally solved, but the lesson remains: When confronted with ambiguity — like the odd lighting in the photo of The Dress — our brains fill in the ambiguity using whatever we’re most familiar with. “People assume what they see more of,” Wallisch says. If we’re more familiar with bright, sunny light, we assume that’s the default lighting.

But we have no way of knowing how our experiences guide our perception. “Your brain makes a lot of unconscious inferences, and it doesn’t tell you that it’s an inference,” he explains. “You see whatever you see. Your brain doesn’t tell you, ‘I took into account how much daylight I’ve seen in my life.’”AD

Wallisch says the disagreements around The Dress, as well as other viral illusions like Yanny and Laurel, arise because our brains are filling in the uncertainties of these stimuli with different prior experiences. We bring our life histories to these small perceptions.

It’s believed another textbook illusion, the Kanizsa triangle, works a bit like this, too. In this illusion, the Pac-Man-like shapes give the impression of a triangle in our minds. It seems like a triangle is there because we’re used to seeing triangles. We only need the suggestion of one — implied via the corners — to fill in the rest of the picture with our minds.

In 2003, the journal Nature Neuroscience published an article on the case of a man (called “Patient MM”) who lost his vision at age 3 and had it restored by surgical intervention in his 40s. In a study, he didn’t fall for an illusion like this one. He couldn’t see the illusory triangle (in the case of that experiment, it was a square). It may be that a lifetime of looking at triangles is what makes the rest of us see one so plainly in this image. Patient MM didn’t build up a lifetime’s worth of visual experiences to make predictions about what he saw. He had to build them from scratch.

More than two years after his operation, Patient MM told researchers, “The difference between today and over two years ago is that I can better guess at what I am seeing. What is the same is that I am still guessing.”

Illusions of consequence

Some of these examples may seem frivolous. Why does it matter that one person sees a dress as black and blue and another sees it as white and gold?

It matters because scientists believe the same basic processes underlie many of our more complicated perceptions and thoughts. Neuroscience, then, can help explain stubborn polarization in our culture and politics, and why we’re so prone to motivated reasoning.

Sometimes, especially when the information we’re receiving is unclear, we see what we want to see. In the past, researchers have found that even slight rewards can change the way people perceive objects. Take this classic image used in psychological studies. What do you see?

It’s either a horse or a seal, and in 2006, psychologists Emily Balcetis and David Dunning showed they could motivate study participants to see one or the other. In one experiment, the participants played a game wherein they had to keep track of animals they saw on screen. If they saw farm animals, they’d get points. If they saw sea creatures, they’d lose points. In the end, a high score meant getting a candy treat (desirable!), and a low score meant they’d eat canned beans (kind of weird).

The very last thing the participants saw was the above image. If seeing the horse meant they’d win and get the candy, they’d see the horse.

In a more complex example, Balcetis has found that when she tells study participants to pay attention to either an officer or a civilian in a video of a police altercation, it can change their perception of what happened (depending on their prior experience with law enforcement and the person in the video with whom they more closely identified). “That instruction changes what their eyes do,” Balcetis told me last summer. “And it leads them to a different understanding of the nature of the altercation.”

You can’t completely remove bias from the brain. “You can’t change the fact that we’ve all grown up in different worlds,” Balcetis said. But you can encourage people to listen to other perspectives and be curious about the veracity of their own.

The neuroscientists I spoke to said the big principles that underlie how our brains process what we see also underlie most of our thinking. Illusions are “the basis of superstition, the basis of magical thinking,” Martinez-Conde says. “It’s the basis for a lot of erroneous beliefs. We’re very uncomfortable with uncertainty. The ambiguity is going to be resolved one way or another, and sometimes in a way that does not match reality.”

Just as we can look at an image and see things that aren’t really there, we can look out into the world with skewed perceptions of reality. Political scientists and psychologists have long documented how political partisans perceive the facts of current events differently depending on their political beliefs. The illusions and political thinking don’t involve the same brain processes, but they follow the similar overarching way the brain works.

In a way, you can think of bias as a social illusion. Studies find that many people perceive black men to be bigger (and, therefore, potentially more threatening) than they actually are, or generally associate darker skin tones and certain facial features with criminality. Cops can confuse people removing wallets from their pockets with people reaching for guns, often with tragic consequences. This isn’t to say that all instances of prejudice are mindless — many are enacted with clear malignant intention, but they can also be built from years of experience in an unjust society or as the result of systemic racism.

Our brains work hard to bend reality to meet our prior experiences, our emotions, and our discomfort with uncertainty. This happens with vision. But it also happens with more complicated processes, like thinking about politics, the pandemic, or the reality of climate change.

Wallisch has come up with a name for phenomena like The Dress that generate divergent perceptions based on our personal characteristics. He calls it “SURFPAD.” Spelled out, it’s an absolute mouthful: Substantial Uncertainty combined with Ramified or Forked Priors and Assumptions yields Disagreement. (Let’s stick with SURFPAD.) Simply, SURFPAD is a consequence of bias, or motivated perception. When an image, event, or some other stimulus isn’t perfectly clear, we fill in the gaps with our priors, or presumptions. And because we have different priors, that leads to disagreement about the image or event in question. Wallisch sees it everywhere in society.

I recently tweeted some frustration over how mass protests against police brutality might be perceived if it seems as though they led to increased Covid-19 cases.

“If there is a spike, it will be hard to discern whether it was reopening or protests, so people will go with their prior,” Wallisch replied. “As the priors are different, there will be massive disagreement. … What’s truly terrifying is that given this framework, no matter what happens, [people] will feel vindicated, reinforcing the strength of the prior and increasing polarization.”

Later, I emailed him and asked whether his inclination to see SURFPAD in these current events was just an instance of his own priors (that SURFPAD is a real and influential phenomenon) coloring his perception.

“Of course,” he says. “It’s SURFPAD all the way down.”

Neuroscience is deeply humbling

I don’t want people to read this and think we can’t believe our eyes, or we can’t incorporate evidence into our thinking. We can seek out verified sources of information. We can turn to expertise and also earnestly question it. (Don’t let people gaslight you, either — another phenomenon that preys on the brain’s tendency to generate illusory thoughts.)

Instead, the illusions and the science behind them raise a question: How do we go about our lives knowing our experiences might be a bit wrong?

There’s no one answer. And it’s a problem we’re unlikely to solve individually. I’d suggest that it should nudge us to be more intellectually humble and to cultivate a habit of seeking out perspectives that are not our own. We should be curious about our imperfections, as that curiosity may lead us closer to the truth. We can build cultures and institutions that celebrate humility and reduce the social cost for saying, “I was wrong.”

This isn’t easy. Our psychology makes it hard. “We have this naive realism that the way we see the world is the way that it really is,” Balcetis told me last year. Naive realism is the feeling that our perception of the world reflects the truth.

But illusions remind us it does not. This is why illusions aren’t just science — they’re provocative art. They force us to reinterpret our senses, and our sense of being in the world. They tell us about the true nature of how our brains work: The same neurological machinery that leads us to discover the truth can lead us to perceive illusions, and our brains don’t always tell us the difference.

Navigating this is the challenge of being a living, thinking person. But simply acknowledging it and trying to put it into practice is a good place to start.

I know I will try to keep remembering that reality always seems real. Even when I mess it up.