Hybrid AI-powered computer vision combines physics and big data

Researchers from UCLA and the United States Army Research Laboratory have devised a new approach to enhancing artificial intelligence-enabled computer vision technology by adding physics-based awareness to data-driven techniques.

Published in Natural Machine Intelligence, This study offers an overview of hybrid methodologies designed to improve how AI-based machines perceive, interact, and respond to their environment in real time — such as how autonomous vehicles move and maneuver, or how robots use enhanced technology to perform precise actions.

Computer vision allows AI to see and understand their environment by decoding data and inferring properties of the physical world from images. While such images are formed through the physics of light and mechanics, traditional computer vision techniques have mostly focused on data-driven machine learning to drive performance. Physics-based research, on separate lines, has developed to explore the various physics principles behind many computer vision challenges.

It is a challenge to inject understandings of physics — the laws that govern mass, motion and more — into the development of neural networks, in which AI mimics the human brain with billions of nodes to crunch huge sets of image data until they gain an understanding of what they are. Look”. But there are now several promising lines of research that seek to add an element of physics awareness to already robust data-driven networks.

The UCLA study aims to harness the power of deep knowledge of data and knowledge of real-world physics to create a hybrid AI with enhanced capabilities.

“Visual machines — cars, robots, or medical instruments that use images to make sense of the world — ultimately perform tasks in our physical world,” said the study’s corresponding author Achuta Kadambi, assistant professor of electrical and computer engineering at UCLA’s Samueli School of Engineering. “A physics-aware form of inference could allow cars to drive more safely or surgical robots more precisely.”

The research team outlines three ways in which physics and data are starting to combine into computer vision artificial intelligence:

• Incorporating physics into AI datasets
  Mark objects with additional information, such as how fast they can move or how much they weigh, similar to characters in video games
• Incorporating physics into the network architecture
  Run data through network filters which encode physical properties into what the camera captures
• Incorporating physics into the network loss function
  Leverage the knowledge built on physics to help the AI ​​interpret training data about what it observes

These three lines of investigation have yielded encouraging results in the improvement of computer vision. For example, the hybrid approach allows AI to track and predict object movements more precisely and can generate accurate high-resolution images of scenes obscured by inclement weather.

With continued advances in this dual modality approach, deep learning-based AI might even begin to learn the laws of physics on its own, according to researchers.

The other authors on the paper are Army Research Laboratory computer scientist Celso de Melo and UCLA faculty member Stefano Soatto, a professor of computer science; Cho-Jui Hsieh, a professor of computer science and Mani Srivastava, a professor of electrical and computer engineering and computer science.

This research was supported in part by a grant from the Army Research Laboratory. Kadambi is supported by grants from the National Science Foundation, the Army’s Young Investigators Program and the Defense Advanced Research Projects Agency. Co-founder of Vayu Robotics, Kadambi also received funding from Intrinsic, an Alphabet company. Hsieh, Srivastava and Soatto received support from Amazon.