This AI model can sense how the physical world works

The original version to This story Featured in Quanta Magazine.

Here’s a test for infants: Show them a cup of water on the desk. Hide it behind a wooden board. Now move the plate towards the glass. If the painting continued to go beyond the glass, as if it wasn’t there, would they be surprised? Many babies by 6 months, and by 1 year almost all babies have an intuitive idea of ​​the permanence of an object, learned through observation. Now some AI models do it too.

Researchers have developed an artificial intelligence system that learns about the world through videos and shows the idea of ​​“surprise” when presented with information that contradicts the collected knowledge.

The model, created by Meta and called Video Joint Embedding Predictive Architecture (V-JEPA), makes no assumptions about the physics of the world contained in the videos. However, it is possible to begin to understand how the world works.

“Their claims are very plausible, and the results are very interesting,” says Mika Heilbron, a cognitive scientist at the University of Amsterdam, who studies how artificial minds and systems understand the world.

Higher abstractions

As engineers who make self-driving cars know, it can be difficult to get an AI system to reliably understand what it sees. Most systems designed to “understand” video clips either to classify their content (“a person playing tennis,” for example) or to determine the perimeter of an object — say, a car in front of you — operate in what is called “pixel space.” The model basically treats every pixel in the video as being of equal importance.

But these pixel space models come with limitations. Imagine you are trying to understand a suburban street. If the scene contains cars, traffic lights, and trees, the model may focus too much on irrelevant details such as the movement of leaves. It may miss the color of a traffic light, or the locations of nearby cars. “When you go to photos or video, you don’t want to work in them [pixel] “The space is big because there are a lot of details that you don’t want to design,” said Randall Ballestrero, a computer scientist at Brown University.

Yann LeCun, a computer scientist at New York University and director of AI research at Meta, created JEPA, a predecessor to V-JEPA that runs on still images, in 2022.

Photo: Ecole Polytechnique of the University of Paris-Saclay