Citable findings and population definitions · BibTeX · Paper v3 · 23 October 2026
01 / Common Tasks
Model Diversity Does Not Remove the Shared Failure Set.
Astra completes 126 of the 157 Planning tasks; Gemini 4 Argon completes 110. Across all ten models, at least one succeeds on 142 tasks: 16 more than Astra alone. Fifteen tasks remain incomplete for every model, while eight are completed by all ten. Astra and Gemini 4 Argon alone cover 137 tasks, recovering 11 of the 16 extra successes available across the ten-model union. Complementarity is concentrated, while a shared failure set remains.
The next evaluation target is the shared failure set, alongside the tasks on which models complement one another. The 15 common failures comprise two ALFRED tasks, two ALFWorld tasks, nine HumanCLAW tasks and two ScienceWorld tasks. Combined coverage counts a task once when at least one model succeeds.
| Model | ALFRED | ALFWorld | DiscoveryWorld | HumanCLAW | ScienceWorld | VirtualHome | All |
|---|---|---|---|---|---|---|---|
| GPT-6 Astra | 15/20 | 14/20 | 10/10 | 3/13 | 64/73 | 20/21 | 126/157 |
| GPT-6 Sol | 3/20 | 4/20 | 4/10 | 2/13 | 38/73 | 19/21 | 70/157 |
| Claude Fable 5.1 | 7/20 | 6/20 | 7/10 | 1/13 | 59/73 | 20/21 | 100/157 |
| Gemini 3.8 Flash | 2/20 | 4/20 | 5/10 | 3/13 | 45/73 | 17/21 | 76/157 |
| Qwen 3.8 Max | 3/20 | 2/20 | 3/10 | 2/13 | 29/73 | 19/21 | 58/157 |
| Qwen 3.5 397B-A17B | 1/20 | 3/20 | 1/10 | 2/13 | 11/73 | 12/21 | 30/157 |
| Qwen 3.5 27B | 0/20 | 4/20 | 1/10 | 0/13 | 14/73 | 11/21 | 30/157 |
| Claude Opus 5.5 | 3/20 | 9/20 | 3/10 | 3/13 | 43/73 | 15/21 | 76/157 |
| DeepSeek 4.1 Flash-1017 | 1/20 | 4/20 | 2/10 | 2/13 | 28/73 | 12/21 | 49/157 |
| Gemini 4 Argon | 14/20 | 16/20 | 6/10 | 2/13 | 52/73 | 20/21 | 110/157 |
Explore individual tasks and model pairs →
Task composition changes the headline. Excluding ScienceWorld and VirtualHome leaves 63 Planning tasks: Astra completes 42, Gemini 4 Argon 38, and Fable 21. In Manipulation, equal weighting of the thirteen sources reverses the two leaders: Opus 5.5 changes from 47.8% to 44.5%, and Gemini 4 Argon from 44.0% to 47.4%. These calculations reweight the same recorded outcomes. Main Results ↗
02 / Resources
Unfinished Tasks Consume Most Tokens.
Median Planning episode duration ranges from 57.1 seconds for DeepSeek-1017 to 300.0 seconds for Qwen 3.8 Max. Completion rates differ at the same time, so a short run can reflect either efficient success or early failure. The source filter on the project page shows these quantities together.
Unsuccessful episodes consume 55.5–94.3% of each model’s recorded tokens, but two different patterns produce this cost. For the strongest planners, failed-token expenditure concentrates in prolonged physical interactions. HumanCLAW accounts for 13 of 157 tasks yet 81.7% of Astra’s failed-token expenditure. Removing it lowers the failed-token share from 55.5% to 19.4% for Astra and from 64.0% to 33.5% for Gemini 4 Argon. Its larger budgets permit repeated attempts at body control and sitting, even while the required state remains unmet.
For the other models, failures extend across ordinary procedures. Outside HumanCLAW, unsuccessful episodes still consume 55.3–67.8% of recorded tokens for the other closed-weight models and 73.6–85.8% for the open-weight VLMs. On the same ScienceWorld animal-transfer task, DeepSeek spends three of eight decision rounds inspecting state and stops with 89 environment steps unused. Opus identifies the blue-jay egg as the requested animal and completes the procedure. The difference concerns resolving the next task requirement, rather than the availability of more environment actions.
| Model | Completed | Median time (s) | Time IQR (s) | Tokens in failed episodes | Usage records | Recovered / error episodes |
|---|---|---|---|---|---|---|
| GPT-6 Astra | 126/157 | 231.7 | 153.2–361.8 | 55.5% | 157/157 | 1/1 |
| GPT-6 Sol | 70/157 | 280.7 | 199.9–427.1 | 76.0% | 157/157 | 1/6 |
| Claude Fable 5.1 | 100/157 | 109.5 | 62.0–201.4 | 64.9% | 157/157 | 6/25 |
| Gemini 3.8 Flash | 76/157 | 61.8 | 28.6–175.9 | 77.1% | 157/157 | 4/24 |
| Qwen 3.8 Max | 58/157 | 300.0 | 216.8–506.7 | 81.2% | 145/157 | 10/38 |
| Qwen 3.5 397B-A17B | 30/157 | 106.5 | 75.8–166.7 | 91.9% | 157/157 | 2/44 |
| Qwen 3.5 27B | 30/157 | 94.5 | 60.8–146.8 | 94.3% | 157/157 | 3/31 |
| Claude Opus 5.5 | 76/157 | 135.6 | 105.3–211.0 | 74.9% | 157/157 | 48/105 |
| DeepSeek 4.1 Flash-1017 | 49/157 | 57.1 | 36.1–78.5 | 83.8% | 157/157 | 9/50 |
| Gemini 4 Argon | 110/157 | 159.5 | 51.5–612.1 | 64.0% | 157/157 | 16/28 |
In the same HumanCLAW chair-navigation task, Gemini 4 Argon completes in 17 rounds and 437.3 seconds; Astra takes 18 rounds and 470.7 seconds; DeepSeek-1017 takes 65 rounds but 405.7 seconds. Model decisions, environment actions and wall time therefore need separate counters.
03 / Recovery
Recovery Must Change the Task State.
Gemini 4 Argon completes 16 of 28 Planning episodes containing an execution error; Opus 5.5 completes 48 of 105. These counts separate how often a model encounters errors from whether it can recover and finish. Errors include malformed code and rejected API use.
The same Archaeology Dating task makes the distinction visible. Gemini 4 Argon encounters an error in round two, resumes measurable progress in round three and completes in round eight. Opus 5.5’s first two rounds fail before an environment action. It then collects tools, excavates the sites, reads artifact ages and places the flag at the oldest site, reaching the native score of 1.0 in round seven. DeepSeek-1017 reaches 0.5 by round five, then its last three rounds end in code errors with no further environment steps.
Recovery is useful when it changes the task state. Successful correction restores the missing prerequisite and enables the next required action.
Compare recovery and returned feedback →
Early Progress Rarely Reaches the Goal.
Across ten models on ten CALVIN sequences each, 60 of 100 episodes reach the first goal, but only nine finish all five. Gemini 4 Argon completes four sequences, GPT-6 Astra three, and Claude Opus 5.5 and DeepSeek 4.1 Flash-1017 one each.
The recovery gap spans models. DeepSeek 4.1 Flash-1017 resumes code in all ten episodes with an execution error; Claude Opus 5.5 and Claude Fable 5.1 each resume in all nine. Only one, one, and zero of those episodes finish, respectively. Across all ten models, 62 of 65 error episodes resume execution and only three complete the task. Clearing an execution error leaves the remaining subgoals to be solved.
04 / Physical Progress
A Valid Action Is Not a Completed Subgoal.
In the supplementary RGB-expanded CALVIN 0526 comparison, Opus 5.5 completes five subgoals using 338 action submissions, 18 model calls and 229.4 seconds. Gemini 3.8 uses 501 action submissions and 464 state observations, yet finishes with zero of five subgoals after 36 calls and 796.5 seconds. Its final stop is the token budget. Gemini 4 Argon completes all five goals with 475 action submissions, 408 state observations, 18 model calls and 351.9 seconds. The current-subgoal queries show Opus 5.5 advancing through retrieval, storage, stacking and the light switch; Gemini 3.8’s retained queries remain on the first lifting goal.
ALFRED makes the final conditions explicit: Gemini 4 Argon satisfies all three conditions for cleaning a soap bar and placing it in the bathtub in six rounds; DeepSeek-1017 ends at one of three after eight rounds. On RoboCasa’s left-drawer task, Gemini 4 Argon completes in 13 rounds despite an error in round ten, while Sol reaches its 24-round ceiling without the required final state.
Count verified changes in the task state, then relate them to actions, observations and elapsed time. This separates purposeful progress from activity that leaves the goal unsatisfied.
CALVIN sequence → · ALFRED goal conditions → · RoboCasa drawer →
Which budget actually stopped the episode?
In the selected BEHAVIOR-1K toolbox task, none of the ten models completes the native goal. Gemini 4 Argon uses 603,393 recorded tokens before the next round is blocked by the 640,000-token limit, after 37.5 minutes of a 180-minute ceiling. On charger insertion, it also reaches a token guard: after 23 native steps the final goal distance is still 15.9 cm. More wall time alone would leave these stopping limits in place.
05 / Region Grounding
Precise Outlines Still Need the Right Target.
Astra leads rotation estimation and Planning; Gemini 3.8 Flash has the lowest numerical spatial error; Gemini 4 Argon leads Spatial Pass@1 and both region-overlap metrics; Opus 5.5 has the highest reported Manipulation success. These rank changes make a single proxy inadequate for selecting an embodied generalist.

| Model | Box–mask IoU | Polygon IoU | Difference | Planning (%) | Manipulation (%) |
|---|---|---|---|---|---|
| GPT-6 Astra | 0.286 | 0.379 | +0.093 | 80.3 | 41.8 |
| GPT-6 Sol | 0.264 | 0.325 | +0.061 | 44.6 | 19.8 |
| Claude Fable 5.1 | 0.284 | 0.328 | +0.044 | 63.7 | 24.7 |
| Qwen 3.8 Max | 0.272 | 0.307 | +0.035 | 36.9 | 9.3 |
| Gemini 3.8 Flash | 0.251 | 0.291 | +0.040 | 48.4 | 18.7 |
| Qwen 3.5 397B-A17B | 0.116 | 0.092 | -0.024 | 19.1 | 9.9 |
| Qwen 3.5 27B | 0.013 | 0.051 | +0.038 | 19.1 | 11.0 |
| Claude Opus 5.5 | 0.311 | 0.397 | +0.086 | 48.4 | 47.8 |
| DeepSeek 4.1 Flash-1017 | 0.217 | 0.249 | +0.032 | 31.2 | 20.9 |
| Gemini 4 Argon | 0.356 | 0.441 | +0.085 | 70.1 | 44.0 |
Direct polygons increase mean region overlap for nine of ten models on the same 70 Affordance cases. Qwen 397B is the exception (0.116 box–mask IoU versus 0.092 polygon IoU). Polygons improve boundary fit. Grasping and interaction additionally require choosing a usable action point and reaching the corresponding physical state.
An embodied generalist must use evidence to finish every required condition, before its decision and interaction budgets run out. The full benchmark measures that end-to-end outcome; the execution traces show where it is lost.
Evaluation Protocol
Tasks, Outcomes, and Resource Use.
Ten models are evaluated on the same 157 Planning tasks, with one evaluated episode per model and task. Completion requires the source evaluator’s final success condition. Episodes terminated by API or execution errors count as unsuccessful attempts; each record retains its stopping reason. The three Qwen models use the latest HumanCLAW attempts on all 13 tasks: Qwen Max and Qwen 397B each complete two navigation tasks, and none completes a sitting task. Compare all 39 HumanCLAW runs →
Elapsed time includes environment interaction and setup where recorded. For the Qwen HumanCLAW attempts it spans worker scheduling, including setup, teardown, and retry waits. IQR denotes the 25th–75th percentile. Token totals sum reported input and output usage; missing usage is excluded, with coverage shown for each model. Across 34 Qwen HumanCLAW records, 536 transport attempts have no reported usage. Failure-token share divides recorded usage in unsuccessful episodes by total recorded usage. Execution recovery denotes final task success after a code-execution error; API transport interruptions, budget guards, and timeouts are separate stopping causes.
The execution explorer compares 46 tasks across ten models, with 460 episodes and 6,856 recorded rounds. Each trace links decisions and feedback to its own task, model, settings, and outcome. Videos retain their playback clock; observation replays show the frames available to the model. Manipulation observation schedules and execution budgets are specified per model in the protocol.
Manuscript source ↓ · Editable figures ↓ · Planning episode data ↓ · Data provenance ↗ · All execution comparisons ↗