THE AI CONUNDRUM

Book supplement · 2023-04-30

Adelson's Checkerboard

An archived post from the original book website, presented in its historical context.

Checkerboard with a cylinder: Which Tile is Darker, A or B?
Which Tile is Darker, A or B?

Visual Understanding

It is incredibly difficult for humans to comprehend the limitations of an AI since a great deal of the experience we have as humans is easily taken for granted. Our intellect is fundamentally tied to our experience in our universe in ways we often don’t realize. Our cognitive structures reflect those realities of our physical world. The optical illusion, Figure 1.1 in the book, is from MIT vision researcher Edward Adelson, shows how our minds interpret the world around us. Are Tile A and Tile B different shades?

Try this interactive version. 

They certainly appear to be different colors. But look again with a masking that eliminates our brain adding the context of the shadow from the cylinder:


How did your brain manage to make them look different? To start, your brain perceives the two-dimensional image as representing a scene in three dimensions. It uses the context that the image represents three-dimensional space with other context clues to infer the direction the light is coming from, and deduces that the cylinder is blocking some of that light. Your brain knows that, when some light is blocked, your eyes receive less light from that object, so it will appear darker. Thus, your brain cancels out the effect of this shadow to see the real, underlying color. This is why an object that looks white is still white even under a shadow, despite the fact your eyes receive less color. Hence, when you look at the image, Tile B appears to be a lighter shade than Tile A—your brain is making it brighter to counteract the effect of the light. That is, when you look at this illusion, your brain immediately applies context to interpret it. 


Imagine asking a computer “After we remove the cylinder, which tile would be brighter, A or B?” It would have a very difficult time answering, because, at a pixel level, A and B are exactly the same color. To a computer, A and B “look” the same. Computers don’t have the context we humans gain from lived experience. 

We interpret meaning in images based on our physical experiences with objects, light, and shadows. This is only a single example of a plethora of other effects that our human brains consider automatically when we see an object, a still image or video. Just inferring that something is a shadow relies on an enormous amount of context, such as the physical context of the way light creates shadows and illuminates an object, or how objects are distorted when translated from a three-dimensional to a two-dimensional image. None of this information is readily available to today’s artificial intelligence when it analyzes a still image. 

AI instead receives only a matrix of numbers with no context. It may not know the precise way in which light bounces, nor may it know that the image is a two-dimensional representation of something that the human brain understands as three-dimensional. This lack of context has profound implications.