HM3D-M³: A Benchmark for Multi-Agent, Multi-Modal, Multi-Object Goal Navigation

Rapid progress in AI agents and robotics makes it a matter of the near future that humans are served by a team of robots. Such a team will receive several goals at once, each specified in whatever form is convenient for the human, such as a category name, a photo, or a spoken description. Existing benchmarks study multi-agent, multi-modal, and multi-object goal navigation in isolation, and a benchmark that places all three in one episode is missing. To fill this gap, we introduce HM3D-M³.

A team of K agents starts at distributed locations in a photorealistic indoor scene and searches for an unordered set of M object goals, each given as an open-vocabulary category, an instance image, or an instance description. HM3D-M³ provides 1,000 validation episodes over 3,926 goals from 379 open-vocabulary categories, 100K observation–assignment training samples, and two team-level metrics, goal-level SR and makespan-normalized SPL. We further contribute two baselines. HiGA is a trained hierarchical goal assigner on top of a frozen open-vocabulary navigation policy, and LaViRA-M³ is a training-free team of VLM agents with shared mapping and progress analysis. Adapted multi-robot and zero-shot methods reach 16–27% SR, while our baselines reach 38–41% with complementary strengths and distinct failure modes. Furthermore, we deploy LaViRA-M³ on a heterogeneous three-robot team.

A team of K ∈ {2, 3, 4} agents shares a map and searches for M ∈ {3, 4, 5} unordered goals, each given as a category (ON), an instance image (IIN), or an instance description (TN); a goal is found once any agent declares it within 1 m of the instance.

Benchmark statistics: 1,000 validation episodes on 36 held-out scenes, with 3,926 goals from 379 categories (1,299 ON / 1,313 TN / 1,314 IIN), and 100K training samples on disjoint scenes.

Two baselines, two paradigms: HiGA (trained) assigns goals to agents with a transformer over agent, goal, and map tokens, on top of a frozen open-vocabulary policy; LaViRA-M³ (training-free) runs a team of language–vision–robot VLM agents that share a top-down map, a goal board, and progress analysis.

Adapted methods find 16–27% of goals; ours reach 38–41%. HiGA leads on category goals, LaViRA-M³ on instance goals and efficiency — neither paradigm dominates.

Failure attribution: LaViRA-M³ misses 59.0% of goals, mostly at claim time (41.8% pose miss); HiGA misses 62.1%, mostly in perception (25.7% failed approach, 16.4% undetected nearby). The two failure profiles are complementary.

We deploy LaViRA-M³ on three heterogeneous robots in an office — an Agilex Cobot Magic, a Unitree Go1, and a Unitree Go2. The robots exchange the shared map as voxel records over Wi-Fi, along with the goal board and progress analysis, through the same channels as in simulation.