Stanford Announces New Embodied AI Benchmarks

By Stephen M

Artificial intelligence (AI) has become the game-changer in robotics and embodied cognition, spearheading physical and simulation models innovations. Recently, Stanford University researchers, through a new study, have introduced BEHAVIOR as a benchmark for embodied AI. This benchmark encompasses 100 normal daily household activities, including cooking, cleaning, and maintenance, incorporated in virtual, interactive, and ecological environments.

Image courtesy of Antonello Marangi / Shutterstock

In a release, award-winning AI pioneer and Co-Director of Stanford’s Human-Centered AI Institute, Dr. Fei-Fei Li, and her co-authors noted that the “activities are designed to be realistic, diverse, and complex, aiming to reproduce the challenges that agents must face in the real world.”

What is an Embodied AI?

Embodied AI is a branch of artificial intelligence, which involves the development of intelligent machines like robots and virtual assistants with a physical or virtual embodiment. The concept allows the machines to interact, observe, think, and engage with the environment.

Procedures

The researchers introduced BEHAVIOR Domain Definition Language (BDDL) to map simulated states to semantic symbols. This, according to the team, allowed them to outline 100 activities as initial goal conditions for the “generation of potentially infinite initial states and solutions for achieving the goal states.”

Using the U.S. Bureau of Labor Statistics’ American Time Use Survey, the team identified some real house activities and introduced them to the simulation environment iGibson 2.0. iGibson 2.0 has 15 fully interactive 3D models of real-life homes with furniture layouts. In addition, about 500 human demonstrations in virtual reality are added as human ground truth.

Antonello Marangi / Shutterstock

BEHAVIOR can quantify the performance of embodied AI using a broad set of metrics. The primary metric becomes the success score, whereas the secondary metric would measure efficiency. Dr. Li and her team will make BEHAVIOR an open-source software for easy access.