Research Findings & Evidence
Robot Planning, Long-Horizon Tasks & Agent Reliability
When does a capable vision-language agent still fail to finish a robot task? Five findings from Embodied Agent Arena connect model diversity, execution recovery, token use, sequence depth and benchmark composition to measured outcomes.
Evaluation setting. The 13 Manipulation source groups reported here, including CALVIN, use source-native simulators, as documented in the released runtime specifications. These rates do not establish physical robot deployment performance. GeoProbe’s real photographs are separate fixed-image diagnostics. Link to evaluation setting.
Read each result with its own population. Planning coverage counts 157 distinct tasks across ten models; CALVIN recovery and sequence completion use the same 100 model–sequence episodes; Manipulation weighting uses 182 binary assignments per model. These populations are not interchangeable. This page documents the ten-model paper v3 evaluation, not an additional experiment.
Investigating a broader research question? The research topic guide connects these results to embodied AI evaluation, spatial reasoning, self-correction and test-time computation, and explains when each result is relevant to a literature review.
Two results to read first
Keep progress and completion separate.
The counts below describe two different retained populations. Each chart keeps its own denominator.
CALVIN simulation · Execution recovery
Running again is only an intermediate event.
100 retained simulator episodes = 10 sequence IDs × 10 models
65 / 100episodes encounter an execution error
Planning · Complementary coverage
Different models cover more of the same tasks.
157 task identities · One retained assignment per model and task
15 tasks remain unsuccessful for all ten.
01 / Planning · Same 157 Tasks
Model Diversity Expands Coverage, but Shared Failures Remain
How much can model complementarity improve embodied planning coverage?
Across the same 157 Planning tasks, at least one of ten evaluated models completes 142 tasks; 15 tasks are unsuccessful for all ten. GPT-6 Astra alone completes 126. The ten-model union therefore adds 16 completed task identities, not 16 percentage points. The Astra–Gemini 4 Argon pair covers 137 tasks: 11 of those 16 additional tasks. The remaining eight models add five more tasks to that pair’s coverage.
| Measure | Tasks | Counting rule |
|---|---|---|
| At least one model succeeds | 142 / 157 | Count each task once across the union of successes. |
| All ten models fail | 15 / 157 | No recorded successful assignment for that task. |
| All ten models succeed | 8 / 157 | Every model completes its assignment. |
| Astra alone succeeds | 126 / 157 | One retained assignment per task for Astra. |
| Astra or Gemini 4 Argon succeeds | 137 / 157 | Count each task once if either model succeeds. |
Population and protocol. The analysis includes 157 task identities × ten models = 1,570 retained model–task assignments, with one retained assignment per model and task, from ALFRED, ALFWorld, DiscoveryWorld, HumanCLAW, ScienceWorld, and VirtualHome. Tasks retain their source-native success criteria and budgets. Each union counts a successful task once, regardless of how many models succeed. Only eight tasks are completed by all ten models; 39 are completed by exactly one or two models. The 15 common failures comprise two ALFRED tasks, two ALFWorld tasks, nine HumanCLAW tasks, and two ScienceWorld tasks.
Scope. This is retrospective success coverage over recorded runs. It does not measure a deployed ensemble, a model-routing policy, or an equal-compute improvement. A shared failure in this sample is not evidence that the task is impossible for these models under other settings.
Suggested attribution. Huang et al. (2026), Embodied Agent Arena, Appendix J.1 and Figure A15, report complementary successes alongside a shared failure set on 157 Planning tasks.
02 / CALVIN · 100 Retained Episodes
Resuming Execution Does Not Establish Task Completion
Does repairing a robot agent’s execution error mean the task is solved?
Among 100 CALVIN episodes, 65 encounter an execution error. Of those 65 episodes, 62 later execute code without an error, but only three subsequently complete the full task. Thus 62 / 65 measures resumed executability, while 3 / 65 measures completion after an error. These are counts of episodes, not counts of error events or repaired tasks. Across all 100 episodes, nine complete all five consecutive subgoals.
| Measure | Episodes | Definition |
|---|---|---|
| Encounter an execution error | 65 / 100 | At least one failed code-execution round. |
| Resume error-free execution | 62 / 65 error episodes | A later round records execution_ok=true. |
| Complete after an error | 3 / 65 error episodes | A later round passes the native task-success verifier. |
| Complete all five subgoals | 9 / 100 | Full-sequence success in the complete retained population. |
Population and protocol. In the CALVIN PyBullet simulator, ten sequence IDs × ten models = 100 retained episodes, one per model–sequence assignment. The 62 episodes with resumed execution must not be described as 62 completed tasks. Execution errors exclude API transport failures, timeouts, and budget guards. Six episodes lack a verified completed-goal prefix and receive no goal credit; they remain in the 100-episode denominator. Recorded model-specific observation schedules and budgets are retained.
Main evaluation and supplementary examples. These counts use the adopted CALVIN simulator episodes and do not estimate physical robot success rates. The separate RGB-expanded sequence-0526 comparisons illustrate recorded behavior under their own settings and must not be substituted into these aggregates. This observational analysis does not establish that error repair causes failure, or that more computation could never help.
Suggested attribution. Huang et al. (2026), Embodied Agent Arena, Figure 8 and Appendix D.5.1, distinguish renewed executability from verified task completion in 100 CALVIN episodes.
03 / Resource Use · Astra Planning Episodes
A Small Failure Set Can Account for Most Token Use
How much token use can unsuccessful embodied-agent tasks account for?
In Astra’s 157 recorded Planning episodes, the 31 unsuccessful episodes account for 55.5% of recorded tokens. An unsuccessful episode uses approximately 5.1 times as many tokens on average as a successful episode: 591,162 versus 116,435 tokens.
| Outcome | Episodes | Total tokens | Mean tokens per episode |
|---|---|---|---|
| Successful | 126 | 14,670,756 | 116,435 |
| Unsuccessful | 31 | 18,326,033 | 591,162 |
| All assignments | 157 | 32,996,789 | 210,171 |
Population and accounting. Usage is recorded for all 157 Astra assignments. Tokens are summed within each episode before grouping by terminal outcome; means are rounded to the nearest token. The failed-token share is 18,326,033 / 32,996,789 = 55.5%; its denominator is tokens across all 157 episodes, not the number of episodes. The 5.1× ratio compares mean tokens per unsuccessful episode with mean tokens per successful episode, not the two token totals. Input tokens are counted when recorded in a call, so repeated context is not deduplicated into unique text.
Scope. Task composition and budgets matter. HumanCLAW contributes 81.7% of Astra’s failed-episode tokens, not 81.7% of all its tokens. Excluding all HumanCLAW episodes changes both numerator and denominator: 3,347,238 tokens in unsuccessful remaining episodes / 17,219,012 tokens in all remaining episodes = 19.4%. These observations do not demonstrate that using more tokens causes failure, and token counts are not a dollar-cost comparison across providers.
Relation to test-time computation. The paper’s separate fixed-input review study evaluates Astra and Qwen 3.5 27B on the same 46 cases: 19 Geometry, 18 Spatial Reasoning and nine Affordance cases. Its review budgets and outcomes are a different experiment from these 157 Astra Planning episodes and the 100 CALVIN episodes. The token association here does not estimate the causal benefit of additional review or inference. See Appendix E.2, Table A34.
Suggested attribution. Huang et al. (2026), Embodied Agent Arena, Figure A15 and Appendix J.6, report that unsuccessful episodes consume 55.5% of Astra’s recorded Planning tokens, with a substantial contribution from HumanCLAW.
04 / Long-Horizon Manipulation · 100 CALVIN Episodes
Early Subgoal Success Overstates Complete-Sequence Reliability
How should long-horizon robot success be measured?
In the same 100 retained CALVIN episodes, 60 reach the first verified subgoal, but only nine complete all five consecutive subgoals. The counts reaching at least one, two, three, four and five goals are 60, 40, 25, 18 and 9, respectively. This is a direct example of the gap between early progress and long-horizon task completion.
| At least this many consecutive goals | Episodes | Share of all episodes |
|---|---|---|
| 1 | 60 / 100 | 60% |
| 2 | 40 / 100 | 40% |
| 3 | 25 / 100 | 25% |
| 4 | 18 / 100 | 18% |
| 5 · Complete sequence | 9 / 100 | 9% |
Population and protocol. In the CALVIN PyBullet simulator, ten sequence IDs × ten models, one retained run per assignment. Six episodes have no verified goal prefix; they receive no goal credit and remain in the denominator. Model-specific observation schedules and budgets are retained, as in the protocol audit.
Scope. These are nested counts, not independent per-step success probabilities: an episode counted at five goals is also counted at one, two, three and four. The five rows must not be added to obtain an episode total, and 9% is complete-sequence success across all 100 episodes, not the probability of completing the fifth goal after reaching the fourth. They do not identify memory, planning or control as the unique cause of failure. Separate RGB-expanded demonstrations are not added to the evaluation population. These simulator completion rates do not establish performance on physical robot hardware.
Suggested attribution. Huang et al. (2026), Embodied Agent Arena, Section 4.3 and Appendix D.5.1, report that 60 of 100 retained CALVIN episodes reach the first subgoal, while nine finish the full five-goal sequence.
05 / Benchmark Design · 13 Manipulation Sources
Changing Source Weights Can Reverse the Manipulation Leader
Can benchmark weighting change which manipulation model ranks first?
With the same recorded outcomes, weighting each of 182 binary Manipulation assignments per model equally gives Opus 5.5 a 47.8% success rate and Argon 44.0%. Giving each of the 13 source benchmarks equal weight instead gives Opus 44.5% and Argon 47.4%. The leading model changes because the aggregation weights change.
| Model | Completed binary assignments | Assignment-weighted success | Equal-source mean success |
|---|---|---|---|
| Claude Opus 5.5 | 87 / 182 | 47.8% | 44.5% |
| Gemini 4 Argon | 80 / 182 | 44.0% | 47.4% |
Population and calculation. For each model, assignment weighting divides total successful binary assignments by 182. Equal-source weighting computes that model’s success rate within each source, then takes the unweighted mean of the 13 source rates; it does not divide the total successes by 13. The 183-case Manipulation inventory also contains one continuous-reward task, which is outside this binary-success comparison.
Scope. This sensitivity analysis holds outcomes and recorded protocols fixed across the 13 source simulators. The resulting rates compare simulated task completion, not physical robot deployment performance. It does not introduce a new evaluation, establish equal compute across models, or show that one weighting is correct for every deployment. A model-selection claim should name the task mixture and aggregation rule.
Suggested attribution. Huang et al. (2026), Embodied Agent Arena, Section 4.2, show that the Manipulation leader changes under equal-source weighting of the same adopted results.
Citation & Version
Cite the Paper and Preserve the Evidence Scope
Huang, Haojian; Zhao, Pukun; Li, Zexi; Zhang, Yehang; Wei, Yangkai; Li, Wenqian; Yang, Han; Zhou, Kaiwen; Chen, Ying-Cong; and Li, Yinchuan. Are Frontier VLM Agents Ready to Be Robot Generalists? An Empirical Study with the Embodied Agent Arena. 2026. doi:10.48550/arXiv.2610.00854.
These findings accompany arXiv version 3, released 23 October 2026. This page’s 14 November 2026 update clarifies simulation and physical robot applicability in existing evidence; it does not announce a new evaluation run. Use the paper citation for the research, and include the finding’s stable section link when pointing readers to its data and definitions.
The linked tables and records are available in the fixed source snapshot used for this page. The CALVIN provenance notes describe missing-prefix handling and how report hashes identify retained episodes.
Paper citation · BibTeX
@article{huang2026frontier,
title={{Are Frontier VLM Agents Ready to Be Robot Generalists? An Empirical Study with the Embodied Agent Arena}},
author={Huang, Haojian and Zhao, Pukun and Li, Zexi and Zhang, Yehang and Wei, Yangkai and Li, Wenqian and Yang, Han and Zhou, Kaiwen and Chen, Ying-Cong and Li, Yinchuan},
journal={arXiv preprint arXiv:2610.00854},
year={2026},
doi={10.48550/arXiv.2610.00854},
url={https://arxiv.org/abs/2610.00854v3},
eprint={2610.00854v3},
archivePrefix={arXiv},
primaryClass={cs.RO},
note={Version 3, 23 October 2026}
}