Embodied Agent Arena

Are Frontier VLM Agents Ready to Be Robot Generalists?

An Empirical Study with the Embodied Agent Arena

Ten VLM Agents · Five Capabilities · 1,000 Cases

Haojian Huang1,3Pukun Zhao3Zexi Li2,3Yehang Zhang1,3Yangkai Wei3Wenqian Li3Han Yang3Kaiwen Zhou3Ying-Cong Chen1,3,†Yinchuan Li3,†
1 HKUST (GZ)2 The Chinese University of Hong Kong3 Knowin AI† Corresponding authors

One Task. Ten Models.

Watch the same task unfold across models. Select any clip to explore.

Explore All Cases ↗

Compare the Same Task

One Task per Column · Click a Clip for Details

Ten models evaluated on the same native task in each column. Results remain visible when footage is unavailable.
Model / Task
GPT-6 Astra
GPT-6 Sol
Fable 5.1
Gemini 3.8 Flash
Qwen Max
Qwen 397B
Qwen 27B
Opus 5.5
DeepSeek-1017
Gemini 4 Argon
Recording CoverageEach column uses one fixed task. Scene only shows the shared task context when a model has no execution footage. API-interrupted attempts are marked API error and counted as unsuccessful; runtime errors and budget stops retain their outcomes. Toolbox clips show updated 24 October recordings, with their own outcomes; original scored records remain available in the details. Demos and updated recordings do not change scores. Download error inventory ↓

Early Success, Unfinished Tasks.

Models can estimate, reason, and act. The harder test is satisfying every goal within finite time and interaction budgets.

Cross-Model Results

Ten Agents. Five Robotic Capabilities.

1,000 cases with source-provided observations and task-specific budgets. Closed-weight models lead all five domains; the open-weight VLMs show useful spatial reasoning but complete fewer Planning and Manipulation tasks.

Full Results · All Eight Metrics Means, SDs & Units
Performance across five capability domains
AgentGeometrySpatial ReasoningAffordancePlanningManipulation
Rotation ↓degreesTranslation ↓cmPass@1 ↑%AbsRel ↓relative errorBox–mask IoU ↑overlapPolygon IoU ↑overlapPass@1 ↑%Task success ↑%
GPT-6 Astra17.2 ± 6.7106.9 ± 36.069.4 ± 1.00.214 ± 0.0160.286 ± 0.0100.37980.341.8
GPT-6 Sol27.0 ± 1.5151.9 ± 6.260.7 ± 2.50.273 ± 0.0160.264 ± 0.0100.32544.619.8
Claude Fable 5.124.2 ± 8.3116.6 ± 42.662.0 ± 4.60.185 ± 0.0110.284 ± 0.0130.32863.724.7
Qwen 3.8 Max20.5 ± 3.5138.1 ± 10.268.9 ± 1.10.206 ± 0.0330.272 ± 0.0170.30736.99.3
Gemini 3.8 Flash23.7 ± 5.6138.0 ± 41.765.4 ± 1.30.161 ± 0.0280.251 ± 0.0810.29148.418.7
Qwen 3.5 397B-A17B38.4 ± 1.1249.8 ± 10.250.6 ± 1.80.251 ± 0.0270.116 ± 0.0140.09219.19.9
Qwen 3.5 27B38.4 ± 3.6213.7 ± 12.648.0 ± 0.90.288 ± 0.0230.013 ± 0.0060.05119.111.0
Claude Opus 5.523.4 ± 3.7135.8 ± 12.767.5 ± 5.10.184 ± 0.0400.311 ± 0.0200.39748.447.8
DeepSeek 4.1 Flash-101730.2 ± 3.9182.0 ± 29.051.6 ± 1.50.309 ± 0.0350.217 ± 0.0110.24931.220.9
Gemini 4 Argon18.5 ± 1.8105.6 ± 9.671.6 ± 0.70.169 ± 0.0230.356 ± 0.0090.44170.144.0

What Prevents Task Completion?

Compare perceptual errors, unmet subgoals, and recovery across all ten models.

Action Is Not Completion.

None of the ten models completes the sitting or household tasks in the evaluation. Inspect the behavior behind the result.

HumanCLAW

0 / 8completed per model

A Sit Command Is Not Enough.

Gemini 4 Argon · Observed frames

Gemini 4 issues sit(), then stop(). The evaluator still marks the task incomplete.

Astra · Eight sitting tasks

6Target found
5≤20 cm + sit
0Completed

Limits: 101 code rounds · 2M tokens · 180 min

BEHAVIOR-1K

0 / 5completed per model

Tokens Run Out Before Time.

Gemini 4 Argon · Updated head-camera recording

In the original evaluation, the token guard blocks the next round after 17 responses, before the 180 min ceiling. The video shows the updated 24 October recording.

Inspect the stopping point →

Original evaluation · Goal not completed

Elapsed time37.5 / 180 min
Tokens used603k / 640k

Limits: 24 model rounds · 640k tokens · 180 min

From Early Subgoals to the Full Task

46 Tasks · 460 Episodes · 10 Models

Compare two agents on the same task. Videos show the behavior; recorded decisions reveal progress and stopping limits.

Loading the execution explorer…

Where Do Embodied Agents Get Stuck?

Read the Full Analysis ↗

Ten models on the same 157 planning tasks. Compare outcomes first, then inspect the recorded decisions.

15 / 157tasks completed by none of the ten models
142 / 157tasks completed by at least one model
8 / 157tasks completed by every model

Different Strengths, Shared Obstacles.

Eight tasks are completed by every model. At least one model completes 142 tasks; the remaining 15 defeat all ten in the recorded evaluation.

Read the task-level comparison ↗

From Seeing a Scene to Completing a Task.

Task decisions stay with the model: interpret observations, choose actions, and recover through source-provided interfaces. The evaluator checks the final environment state independently of the agent’s declaration.

1,000evaluation cases
32 + 1sources + GeoProbe
5capability domains
10frontier VLM agents
  1. EstimateGeometry & Spatial Reasoning

    What is where, in which reference frame?

  2. GroundAffordance

    Which object and region support the action?

  3. CompletePlanning & Manipulation

    Do all required goal conditions hold at the end?

Separate Object Motion from Camera Motion.

168 new cases isolate camera motion, object motion, depth, and scale. Controlled changes reveal which geometric ambiguities affect an estimate.

Five capability domains, GeoProbe’s estimation targets, and the evaluation workflow. Click the figure to enlarge.

Perception, Action, and Diagnosis.

Paper · Table 1 ↗

Across visual probes, navigation, and manipulation, the Arena measures both local capability and task completion. GeoProbe adds camera calibration and relative-pose diagnostics.

Swipe to compare all capabilities →

Benchmark & scaleEvaluation coverageDiagnostic analysis
BenchmarkInstancesFixed
views
Region
grounding
NavigationArm /
gripper
Cross-session
memory
Calibration
/ pose
Task
breakdowns
OpenEQA1,636
VLABenchNot reported
BEAR4,469
EmbodiedBench1,128
EmbodiedEval328
Embodied ArenaNot reported
CaP-X700/tier
WorldLines331
Hard/Easy VisionNot reported
Embodied Agent Arena1,000

✓ Evaluated— Not evaluatedBEAR manipulation: separate BEAR-Agent online validation.

Instances count QA items or task episodes; “Not reported” means the paper gives no aggregate count. Scale definitions and sources ↗

Manipulation suites such as LIBERO, RoboTwin 2.0, and RoboDojo test transfer and robustness through robot interaction. The Arena also measures geometry, spatial inference, and visual grounding directly, linking these capabilities to planning and manipulation outcomes.

Research overview · arXiv v3

From Local Capabilities to Complete Tasks

Embodied Agent Arena evaluates ten frontier vision-language agents across Geometry, Spatial Reasoning, Affordance, Task Planning, and Manipulation. Its 1,000 cases combine 32 established sources with GeoProbe, our 168-query geometric-estimation benchmark. A shared minimal harness leaves task interpretation, reasoning, and action selection to the model through source-native interfaces.

Explore by research question. The embodied AI research guide connects this study to VLM spatial reasoning, long-horizon robot tasks, agent self-correction, inference cost and benchmark design.

Five capability domains

Compare local estimates and final task success with explicit metrics, model versions, and recorded execution budgets.

Evaluation scope and settings →

Paper: v3 · 23 October 2026. Research pages updated . Cite this study.

Explore Every Capability.

Browse Full Gallery ↗

Compare ten models on the same task. Play recordings or inspect execution logs.

Illustrative cases from ten models. Select a case for its task, settings, and recorded outcome.

Protocol, Metrics & Model Versions

Metrics & Scoring

Rotation and translation measure pose error; AbsRel measures numerical spatial error. Box–mask IoU and Polygon IoU score region overlap on the same 70 Affordance cases using separate box and polygon outputs. Planning covers 157 cases; Manipulation covers 182 binary assignments, plus one continuous-reward case in the inventory. Episodes terminated by API errors, execution errors, or unmet goals count as unsuccessful attempts.

Protocol & Budgets

Ten models share the assigned tasks, native interfaces and success criteria. Manipulation retains each batch’s recorded RGB schedule and execution limits. Compare the recorded settings ↗

Polygon grounding, tracing, Planning, and Manipulation use one run per assignment. Repeated probes use five runs, except Fable (three). Full repetition settings ↗

Names identify evaluated checkpoints: Claude Opus 5.5 and DeepSeek-V4.1-Flash-1017 (the deepseek-flash API updated on 17 October 2026). All Model IDs & Settings ↗

Huang et al. (2026), arXiv:2610.00854v3 · DOI. Paper revised 23 October 2026; website updates do not imply new experiments.

Paper citation & BibTeXRead plain textDownload BibTeXDownload CSL-JSONCite a specific finding

Citation

@article{huang2026frontier,
  title={{Are Frontier VLM Agents Ready to Be Robot Generalists? An Empirical Study with the Embodied Agent Arena}},
  author={Huang, Haojian and Zhao, Pukun and Li, Zexi and Zhang, Yehang and Wei, Yangkai and Li, Wenqian and Yang, Han and Zhou, Kaiwen and Chen, Ying-Cong and Li, Yinchuan},
  journal={arXiv preprint arXiv:2610.00854},
  year={2026},
  doi={10.48550/arXiv.2610.00854},
  url={https://arxiv.org/abs/2610.00854v3},
  eprint={2610.00854v3},
  archivePrefix={arXiv},
  primaryClass={cs.RO},
  note={Version 3, 23 October 2026}
}