HumanCLAW
0 / 8completed per model
A Sit Command Is Not Enough.
Gemini 4 issues sit(), then stop(). The evaluator still marks the task incomplete.
Astra · Eight sitting tasks
Limits: 101 code rounds · 2M tokens · 180 min

Embodied Agent Arena
An Empirical Study with the Embodied Agent Arena
Ten VLM Agents · Five Capabilities · 1,000 Cases
Watch the same task unfold across models. Select any clip to explore.
| Model / Task | |||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| GPT-6 Astra | |||||||||||
| GPT-6 Sol | |||||||||||
| Fable 5.1 | |||||||||||
| Gemini 3.8 Flash | |||||||||||
| Qwen Max | |||||||||||
| Qwen 397B | |||||||||||
| Qwen 27B | |||||||||||
| Opus 5.5 | |||||||||||
| DeepSeek-1017 | |||||||||||
| Gemini 4 Argon |
Models can estimate, reason, and act. The harder test is satisfying every goal within finite time and interaction budgets.
Gemini 4 Argon leads both measures. Yet more than half its paths violate at least one required condition.
Compare Geometry → Task PlanningCombined successes cover 16 more tasks than Astra alone. All eight sitting tasks remain unfinished.
Explore Shared Failures → Execution & ResourcesAstra’s 31 failed tasks use more tokens than its 126 successes combined. Further interaction must produce progress.
Inspect Time & Tokens →Cross-Model Results
1,000 cases with source-provided observations and task-specific budgets. Closed-weight models lead all five domains; the open-weight VLMs show useful spatial reasoning but complete fewer Planning and Manipulation tasks.
| Agent | Geometry | Spatial Reasoning | Affordance | Planning | Manipulation | |||
|---|---|---|---|---|---|---|---|---|
| Rotation ↓degrees | Translation ↓cm | Pass@1 ↑% | AbsRel ↓relative error | Box–mask IoU ↑overlap | Polygon IoU ↑overlap | Pass@1 ↑% | Task success ↑% | |
| GPT-6 Astra | 17.2 ± 6.7 | 106.9 ± 36.0 | 69.4 ± 1.0 | 0.214 ± 0.016 | 0.286 ± 0.010 | 0.379 | 80.3 | 41.8 |
| GPT-6 Sol | 27.0 ± 1.5 | 151.9 ± 6.2 | 60.7 ± 2.5 | 0.273 ± 0.016 | 0.264 ± 0.010 | 0.325 | 44.6 | 19.8 |
| Claude Fable 5.1 | 24.2 ± 8.3 | 116.6 ± 42.6 | 62.0 ± 4.6 | 0.185 ± 0.011 | 0.284 ± 0.013 | 0.328 | 63.7 | 24.7 |
| Qwen 3.8 Max | 20.5 ± 3.5 | 138.1 ± 10.2 | 68.9 ± 1.1 | 0.206 ± 0.033 | 0.272 ± 0.017 | 0.307 | 36.9 | 9.3 |
| Gemini 3.8 Flash | 23.7 ± 5.6 | 138.0 ± 41.7 | 65.4 ± 1.3 | 0.161 ± 0.028 | 0.251 ± 0.081 | 0.291 | 48.4 | 18.7 |
| Qwen 3.5 397B-A17B | 38.4 ± 1.1 | 249.8 ± 10.2 | 50.6 ± 1.8 | 0.251 ± 0.027 | 0.116 ± 0.014 | 0.092 | 19.1 | 9.9 |
| Qwen 3.5 27B | 38.4 ± 3.6 | 213.7 ± 12.6 | 48.0 ± 0.9 | 0.288 ± 0.023 | 0.013 ± 0.006 | 0.051 | 19.1 | 11.0 |
| Claude Opus 5.5 | 23.4 ± 3.7 | 135.8 ± 12.7 | 67.5 ± 5.1 | 0.184 ± 0.040 | 0.311 ± 0.020 | 0.397 | 48.4 | 47.8 |
| DeepSeek 4.1 Flash-1017 | 30.2 ± 3.9 | 182.0 ± 29.0 | 51.6 ± 1.5 | 0.309 ± 0.035 | 0.217 ± 0.011 | 0.249 | 31.2 | 20.9 |
| Gemini 4 Argon | 18.5 ± 1.8 | 105.6 ± 9.6 | 71.6 ± 0.7 | 0.169 ± 0.023 | 0.356 ± 0.009 | 0.441 | 70.1 | 44.0 |
Compare perceptual errors, unmet subgoals, and recovery across all ten models.
None of the ten models completes the sitting or household tasks in the evaluation. Inspect the behavior behind the result.
HumanCLAW
0 / 8completed per model
Gemini 4 issues sit(), then stop(). The evaluator still marks the task incomplete.
Astra · Eight sitting tasks
Limits: 101 code rounds · 2M tokens · 180 min
BEHAVIOR-1K
0 / 5completed per model
In the original evaluation, the token guard blocks the next round after 17 responses, before the 180 min ceiling. The video shows the updated 24 October recording.
Inspect the stopping point →Original evaluation · Goal not completed
Limits: 24 model rounds · 640k tokens · 180 min
46 Tasks · 460 Episodes · 10 Models
Compare two agents on the same task. Videos show the behavior; recorded decisions reveal progress and stopping limits.
Loading the execution explorer…
Ten models on the same 157 planning tasks. Compare outcomes first, then inspect the recorded decisions.
Eight tasks are completed by every model. At least one model completes 142 tasks; the remaining 15 defeat all ten in the recorded evaluation.
Task decisions stay with the model: interpret observations, choose actions, and recover through source-provided interfaces. The evaluator checks the final environment state independently of the agent’s declaration.
What is where, in which reference frame?
Which object and region support the action?
Do all required goal conditions hold at the end?
168 new cases isolate camera motion, object motion, depth, and scale. Controlled changes reveal which geometric ambiguities affect an estimate.
Across visual probes, navigation, and manipulation, the Arena measures both local capability and task completion. GeoProbe adds camera calibration and relative-pose diagnostics.
Swipe to compare all capabilities →
| Benchmark & scale | Evaluation coverage | Diagnostic analysis | ||||||
|---|---|---|---|---|---|---|---|---|
| Benchmark | Instances | Fixed views | Region grounding | Navigation | Arm / gripper | Cross-session memory | Calibration / pose | Task breakdowns |
| OpenEQA | 1,636 | |||||||
| VLABench | Not reported | |||||||
| BEAR | 4,469 | |||||||
| EmbodiedBench | 1,128 | |||||||
| EmbodiedEval | 328 | |||||||
| Embodied Arena | Not reported | |||||||
| CaP-X | 700/tier | |||||||
| WorldLines | 331 | |||||||
| Hard/Easy Vision | Not reported | |||||||
| Embodied Agent Arena | 1,000 | |||||||
✓ Evaluated— Not evaluatedBEAR manipulation: separate BEAR-Agent online validation.
Instances count QA items or task episodes; “Not reported” means the paper gives no aggregate count. Scale definitions and sources ↗
Manipulation suites such as LIBERO, RoboTwin 2.0, and RoboDojo test transfer and robustness through robot interaction. The Arena also measures geometry, spatial inference, and visual grounding directly, linking these capabilities to planning and manipulation outcomes.
Research overview · arXiv v3
Embodied Agent Arena evaluates ten frontier vision-language agents across Geometry, Spatial Reasoning, Affordance, Task Planning, and Manipulation. Its 1,000 cases combine 32 established sources with GeoProbe, our 168-query geometric-estimation benchmark. A shared minimal harness leaves task interpretation, reasoning, and action selection to the model through source-native interfaces.
Explore by research question. The embodied AI research guide connects this study to VLM spatial reasoning, long-horizon robot tasks, agent self-correction, inference cost and benchmark design.
Compare local estimates and final task success with explicit metrics, model versions, and recorded execution budgets.
Evaluation scope and settings →Ten protocols over 130 unique RGB images isolate camera and object motion, calibration, depth, and scale.
Tasks, metrics, data, and limitations →Inspect shared planning failures, recovery after execution errors, and the token cost of unsuccessful attempts.
Findings, evidence, and citation →Paper: v3 · 23 October 2026. Research pages updated . Cite this study.
Compare ten models on the same task. Play recordings or inspect execution logs.
Illustrative cases from ten models. Select a case for its task, settings, and recorded outcome.
Rotation and translation measure pose error; AbsRel measures numerical spatial error. Box–mask IoU and Polygon IoU score region overlap on the same 70 Affordance cases using separate box and polygon outputs. Planning covers 157 cases; Manipulation covers 182 binary assignments, plus one continuous-reward case in the inventory. Episodes terminated by API errors, execution errors, or unmet goals count as unsuccessful attempts.
Ten models share the assigned tasks, native interfaces and success criteria. Manipulation retains each batch’s recorded RGB schedule and execution limits. Compare the recorded settings ↗
Polygon grounding, tracing, Planning, and Manipulation use one run per assignment. Repeated probes use five runs, except Fable (three). Full repetition settings ↗
Names identify evaluated checkpoints: Claude Opus 5.5 and DeepSeek-V4.1-Flash-1017 (the deepseek-flash API updated on 17 October 2026). All Model IDs & Settings ↗
Huang et al. (2026), arXiv:2610.00854v3 · DOI. Paper revised 23 October 2026; website updates do not imply new experiments.
Paper citation & BibTeXRead plain textDownload BibTeXDownload CSL-JSONCite a specific finding
@article{huang2026frontier,
title={{Are Frontier VLM Agents Ready to Be Robot Generalists? An Empirical Study with the Embodied Agent Arena}},
author={Huang, Haojian and Zhao, Pukun and Li, Zexi and Zhang, Yehang and Wei, Yangkai and Li, Wenqian and Yang, Han and Zhou, Kaiwen and Chen, Ying-Cong and Li, Yinchuan},
journal={arXiv preprint arXiv:2610.00854},
year={2026},
doi={10.48550/arXiv.2610.00854},
url={https://arxiv.org/abs/2610.00854v3},
eprint={2610.00854v3},
archivePrefix={arXiv},
primaryClass={cs.RO},
note={Version 3, 23 October 2026}
}