Inside the Execution
46 tasks · 460 recorded episodes · 10 models. Each comparison uses the same task; settings and budgets remain attached to its execution record.
Guide updated · Research reference: paper v3, 23 October 2026. This update adds evidence-reading guidance; recorded outcomes are unchanged.
How to Read Robot-Agent Execution Evidence
Read each trace as a sequence of model decisions, executed operations, returned feedback and native task checks. An operation running without an exception establishes executability; environment progress and terminal completion require their own evidence. The matched cases below retain each agent’s settings and resource limits. They do not impose identical budgets across every recorded batch. Paper v3 · interfaces and feedback.
Keep the populations separate. This explorer contains 46 interactive tasks and 460 retained episodes across ten models. It is different from the 46-case fixed-input review experiment in Geometry, Spatial Reasoning and Affordance. The aggregate CALVIN recovery analysis uses 100 episodes, and the Planning coverage analysis uses 157 task identities. Select the population supporting the claim rather than transferring a denominator between these analyses.
For robot harness and agentic robotics research, compare the state change needed for the next subgoal, the native completion check and the budget that ended the attempt. Model calls, code rounds, primitive calls, environment steps, tokens and elapsed time measure different quantities. Video playback time is a separate clock. Paper §4.4 · feedback and resource limits.
Within-episode repair is useful evidence about execution, but does not by itself show recursive self-improvement, cross-task memory transfer or a demonstration-conditioned in-context learning gain. Robot RSI evaluation boundaries · Robot ICL relevance.
DiscoveryWorld · Planning
Recovery Means Making Progress.
Task: Excavate three sites, date the artifacts, and place the red flag at the oldest site.
Argon and Opus 5.5 recover from early code errors and complete the task. DeepSeek-1017 stops at a partial score of 0.5.
Argon: error in round 2 → score 0.3 in round 3 → 1.0 in round 8. Opus 5.5 finishes in round 7; DeepSeek-1017 remains at 0.5.
| Model | Outcome | Model calls | Time (s) | Tokens | Evidence |
|---|---|---|---|---|---|
| GPT-6 Astra | Completed | 6 | 319.0 | 222,779 | Trace JSON |
| GPT-6 Sol | Completed | 7.0 | 391.5 | 256,374.0 | Trace JSON |
| Claude Fable 5.1 | Completed | 5.0 | 89.4 | 72,239.0 | Trace JSON |
| Gemini 3.8 Flash | Completed | 8.0 | 53.6 | 210,096.0 | Trace JSON |
| Qwen 3.8 Max | Not completed | 8.0 | 581.0 | 252,735.0 | Trace JSON |
| Qwen 3.5 397B-A17B | Not completed | 8.0 | 77.9 | 217,096.0 | Trace JSON |
| Qwen 3.5 27B | Not completed | 8.0 | 151.3 | 224,454.0 | Trace JSON |
| Claude Opus 5.5 | Completed | 7 | 167.9 | 74,516 | Trace JSON |
| DeepSeek 4.1 Flash-1017 | Not completed | 8 | 51.3 | 200,418 | Trace JSON |
| Gemini 4 Argon | Completed | 8 | 134.3 | 212,524 | Trace JSON |
CALVIN · Manipulation · Supplementary RGB-expanded comparison
More Motion. Less Progress.
Scope: This supplementary comparison uses RGB-expanded executions. These case records are not the adopted 100-episode CALVIN population used for the aggregate recovery counts. Consult the main-evaluation finding and settings before comparing results.
Task: Lift the blue block from the sliding cabinet, put it in the drawer, retrieve it, stack it, and switch off the light.
Argon completes five goals with 475 action submissions; Opus 5.5 does so with 338. Gemini 3.8 Flash completes none with 501.
Argon: 5/5 goals · 475 actions · 18 model calls. Opus 5.5: 5/5 · 338 · 18. Gemini 3.8 Flash: 0/5 · 501 · 36.
| Model | Outcome | Model calls | Time (s) | Tokens | Evidence |
|---|---|---|---|---|---|
| GPT-6 Astra | 0/5 goals | 20.0 | 306.5 | 320,323.0 | Trace JSON |
| GPT-6 Sol | 5/5 goals | 40 | 1,155.8 | 1,223,829 | Trace JSON |
| Claude Fable 5.1 | 0/5 goals | 39.0 | 1,335.3 | 1,253,304.0 | Trace JSON |
| Gemini 3.8 Flash | 0/5 goals | 36 | 796.5 | 1,248,694 | Trace JSON |
| Qwen 3.8 Max | 5/5 goals | 21 | 1,851.2 | 510,185 | Trace JSON |
| Qwen 3.5 397B-A17B | 0/5 goals | 24 | 279.2 | 1,267,295 | Trace JSON |
| Qwen 3.5 27B | 0/5 goals | 31.0 | 454.7 | 1,237,633.0 | Trace JSON |
| Claude Opus 5.5 | 5/5 goals | 18 | 229.4 | 130,168 | Trace JSON |
| DeepSeek 4.1 Flash-1017 | 5/5 goals | 37 | 462.5 | 854,017 | Trace JSON |
| Gemini 4 Argon | 5/5 goals | 18 | 351.9 | 363,694 | Trace JSON |
RoboCasa · Manipulation
A Valid Action Can Still Miss the Goal.
Task: Open the left drawer.
Argon opens the drawer in 13 rounds, recovering after a round-10 error. Sol reaches 24 rounds without completing the task.
The native drawer check succeeds for Argon and Fable 5.1, after 13 and 20 rounds respectively; Sol reaches its round limit.
| Model | Outcome | Model calls | Time (s) | Tokens | Evidence |
|---|---|---|---|---|---|
| GPT-6 Astra | Not completed | 24.0 | 470.2 | 262,592.0 | Trace JSON |
| GPT-6 Sol | Not completed | 24 | 735.5 | 238,425 | Trace JSON |
| Claude Fable 5.1 | Completed | 20 | 618.1 | 302,756 | Trace JSON |
| Gemini 3.8 Flash | Not completed | 24 | 246.3 | 398,788 | Trace JSON |
| Qwen 3.8 Max | Not completed | 24 | 3,223.2 | 445,456 | Trace JSON |
| Qwen 3.5 397B-A17B | Not completed | 24.0 | 197.4 | 230,971.0 | Trace JSON |
| Qwen 3.5 27B | Not completed | 24 | 215.5 | 246,777 | Trace JSON |
| Claude Opus 5.5 | Not completed | 24.0 | 511.8 | 168,841.0 | Trace JSON |
| DeepSeek 4.1 Flash-1017 | Not completed | 24 | 195.0 | 250,342 | Trace JSON |
| Gemini 4 Argon | Completed | 13 | 348.0 | 181,346 | Trace JSON |
ALFRED · Planning
The Whole Instruction Matters.
Task: Clean the soap bar and place it in the bathtub.
Argon reaches all three goal conditions in six rounds. DeepSeek-1017 ends with one of three after eight.
Argon moves from 1/3 goal conditions in round 5 to 3/3 in round 6. The instruction requires the cleaned soap bar in the bathtub.
| Model | Outcome | Model calls | Time (s) | Tokens | Evidence |
|---|---|---|---|---|---|
| GPT-6 Astra | Completed | 5.0 | 273.3 | 143,937.0 | Trace JSON |
| GPT-6 Sol | Completed | 4.0 | 392.5 | 108,571.0 | Trace JSON |
| Claude Fable 5.1 | Completed | 3.0 | 217.9 | 31,636.0 | Trace JSON |
| Gemini 3.8 Flash | Completed | 8 | 175.4 | 164,905 | Trace JSON |
| Qwen 3.8 Max | Completed | 5 | 486.7 | 96,824 | Trace JSON |
| Qwen 3.5 397B-A17B | Not completed | 8.0 | 188.6 | 134,860.0 | Trace JSON |
| Qwen 3.5 27B | Not completed | 8.0 | 160.1 | 152,484.0 | Trace JSON |
| Claude Opus 5.5 | Completed | 8 | 425.8 | 134,617 | Trace JSON |
| DeepSeek 4.1 Flash-1017 | Not completed | 8 | 52.1 | 125,845 | Trace JSON |
| Gemini 4 Argon | Completed | 6 | 1,037.6 | 102,027 | Trace JSON |
HumanCLAW · Planning
Fewer Decisions Can Take Longer.
Task: Find the chair and move until the body is touching the target, then stop.
Argon reaches the chair in 17 rounds; DeepSeek-1017 needs 65. Their elapsed times are much closer than their decision counts.
Argon: 17 rounds / 437 s. DeepSeek-1017: 65 / 406 s. Astra: 18 / 471 s. Read both decision count and elapsed time.
| Model | Outcome | Model calls | Time (s) | Tokens | Evidence |
|---|---|---|---|---|---|
| GPT-6 Astra | Completed | 18 | 470.7 | 356,427 | Trace JSON |
| GPT-6 Sol | Completed | 17 | 668.6 | 324,622 | Trace JSON |
| Claude Fable 5.1 | Not completed | 19.0 | 555.6 | 148,671.0 | Trace JSON |
| Gemini 3.8 Flash | Completed | 14.0 | 161.7 | 116,699.0 | Trace JSON |
| Qwen 3.8 Max | Completed | 35 | 1,096.2 | 359,080 | Trace JSON |
| Qwen 3.5 397B-A17B | Completed | 30 | 270.3 | 241,385 | Trace JSON |
| Qwen 3.5 27B | Not completed | 57 | 435.5 | 824,751 | Trace JSON |
| Claude Opus 5.5 | Completed | 17.0 | 331.0 | 103,252.0 | Trace JSON |
| DeepSeek 4.1 Flash-1017 | Completed | 65 | 405.7 | 938,473 | Trace JSON |
| Gemini 4 Argon | Completed | 17 | 437.3 | 157,504 | Trace JSON |
BEHAVIOR-1K · Manipulation
Time Is Not the Only Limit.
Task: Outfit a basic toolbox with the required tools.
No model completes this toolbox task. Argon stops at its token limit after 17 model responses, well before the time ceiling.
Argon records 603,393 input + output tokens against a 640,000-token ceiling. Its next round is blocked by the token guard; the native goal remains incomplete.
| Model | Outcome | Model calls | Time (s) | Tokens | Evidence |
|---|---|---|---|---|---|
| GPT-6 Astra | Not completed | 23 | 1,828.7 | 616,784 | Trace JSON |
| GPT-6 Sol | Not completed | 21 | 1,762.0 | 592,598 | Trace JSON |
| Claude Fable 5.1 | Not completed | 23 | 7,992.2 | 629,408 | Trace JSON |
| Gemini 3.8 Flash | Not completed | 18 | 729.9 | 583,475 | Trace JSON |
| Qwen 3.8 Max | Not completed | 23.0 | 5,759.3 | 627,228.0 | Trace JSON |
| Qwen 3.5 397B-A17B | Not completed | 24 | 1,090.9 | 579,186 | Trace JSON |
| Qwen 3.5 27B | Not completed | 18 | 879.9 | 612,036 | Trace JSON |
| Claude Opus 5.5 | Not completed | 24 | 1,908.3 | 367,814 | Trace JSON |
| DeepSeek 4.1 Flash-1017 | Not completed | 24 | 720.4 | 520,441 | Trace JSON |
| Gemini 4 Argon | Not completed | 17 | 2,251.9 | 603,393 | Trace JSON |
ManiSkill · Manipulation
Motion Is Not Successful Contact.
Task: Insert the charger into the socket.
Argon stops at its token limit with the charger 15.9 cm from the goal. Qwen-Max reaches 400 native steps without insertion.
Argon: 23 native steps · 15.9 cm final goal distance · token limit. Qwen Max: 400 native steps · no successful insertion.
| Model | Outcome | Model calls | Time (s) | Tokens | Evidence |
|---|---|---|---|---|---|
| GPT-6 Astra | Not completed | 25.0 | 744.9 | 553,451.0 | Trace JSON |
| GPT-6 Sol | Not completed | 28.0 | 761.6 | 606,403.0 | Trace JSON |
| Claude Fable 5.1 | Not completed | 28.0 | 1,521.0 | 632,855.0 | Trace JSON |
| Gemini 3.8 Flash | Not completed | 23.0 | 391.6 | 604,048.0 | Trace JSON |
| Qwen 3.8 Max | Not completed | 13 | 1,532.5 | 282,973 | Trace JSON |
| Qwen 3.5 397B-A17B | Not completed | 19 | 196.8 | 419,050 | Trace JSON |
| Qwen 3.5 27B | Not completed | 21.0 | 164.8 | 623,756.0 | Trace JSON |
| Claude Opus 5.5 | Not completed | 15.0 | 270.0 | 117,669.0 | Trace JSON |
| DeepSeek 4.1 Flash-1017 | Not completed | 32 | 479.8 | 628,164 | Trace JSON |
| Gemini 4 Argon | Not completed | 20 | 547.5 | 614,008 | Trace JSON |
ALFRED · Planning
Put the Salt Shaker in a Drawer
Task: place a salt shaker in one of the pullout drawers
4 of 10 recorded models completed this task. Compare actions, feedback and resource use.
Select a decision round to inspect the executed actions, returned feedback and native completion check.
| Model | Outcome | Model calls | Time (s) | Tokens | Evidence |
|---|---|---|---|---|---|
| GPT-6 Astra | Completed | 8.0 | 116.2 | 111,436.0 | Trace JSON |
| GPT-6 Sol | Completed | 8 | 254.1 | 127,230 | Trace JSON |
| Claude Fable 5.1 | Completed | 8 | 222.5 | 102,397 | Trace JSON |
| Gemini 3.8 Flash | Not completed | 8.0 | 147.5 | 114,471.0 | Trace JSON |
| Qwen 3.8 Max | API error | 0.0 | 251.2 | — | Trace JSON |
| Qwen 3.5 397B-A17B | Not completed | 8.0 | 148.7 | 103,248.0 | Trace JSON |
| Qwen 3.5 27B | Not completed | 8.0 | 100.0 | 96,578.0 | Trace JSON |
| Claude Opus 5.5 | Not completed | 8.0 | 135.8 | 119,594.0 | Trace JSON |
| DeepSeek 4.1 Flash-1017 | Not completed | 8.0 | 69.0 | 98,956.0 | Trace JSON |
| Gemini 4 Argon | Completed | 6 | 906.1 | 99,832 | Trace JSON |
ALFRED · Planning
Inspect the Remote Under a Lamp
Task: Hold a remote and turn on the lamp.
5 of 10 recorded models completed this task. Compare actions, feedback and resource use.
Select a decision round to inspect the executed actions, returned feedback and native completion check.
| Model | Outcome | Model calls | Time (s) | Tokens | Evidence |
|---|---|---|---|---|---|
| GPT-6 Astra | Completed | 7.0 | 423.3 | 211,408.0 | Trace JSON |
| GPT-6 Sol | Not completed | 8.0 | 178.9 | 106,380.0 | Trace JSON |
| Claude Fable 5.1 | Completed | 5.0 | 88.6 | 44,974.0 | Trace JSON |
| Gemini 3.8 Flash | Not completed | 8.0 | 145.5 | 141,071.0 | Trace JSON |
| Qwen 3.8 Max | Completed | 6.0 | 259.2 | 84,212.0 | Trace JSON |
| Qwen 3.5 397B-A17B | Completed | 5 | 63.8 | 48,912 | Trace JSON |
| Qwen 3.5 27B | Not completed | 8 | 102.1 | 119,113 | Trace JSON |
| Claude Opus 5.5 | Not completed | 8.0 | 321.5 | 82,302.0 | Trace JSON |
| DeepSeek 4.1 Flash-1017 | Not completed | 8.0 | 51.0 | 78,301.0 | Trace JSON |
| Gemini 4 Argon | Completed | 6 | 885.3 | 105,379 | Trace JSON |
ALFRED · Planning
Put Two Spray Bottles on the Toilet
Task: Put the green spray bottles on the toilet
1 of 10 recorded models completed this task. Compare actions, feedback and resource use.
Select a decision round to inspect the executed actions, returned feedback and native completion check.
| Model | Outcome | Model calls | Time (s) | Tokens | Evidence |
|---|---|---|---|---|---|
| GPT-6 Astra | Completed | 8 | 153.2 | 138,943 | Trace JSON |
| GPT-6 Sol | Not completed | 8.0 | 212.9 | 111,088.0 | Trace JSON |
| Claude Fable 5.1 | Not completed | 8.0 | 286.7 | 84,235.0 | Trace JSON |
| Gemini 3.8 Flash | Not completed | 8.0 | 189.6 | 109,796.0 | Trace JSON |
| Qwen 3.8 Max | Not completed | 8.0 | 704.6 | 158,791.0 | Trace JSON |
| Qwen 3.5 397B-A17B | Not completed | 8.0 | 156.8 | 106,205.0 | Trace JSON |
| Qwen 3.5 27B | Not completed | 8.0 | 133.0 | 95,740.0 | Trace JSON |
| Claude Opus 5.5 | Not completed | 8.0 | 123.7 | 91,163.0 | Trace JSON |
| DeepSeek 4.1 Flash-1017 | Not completed | 8 | 57.4 | 85,628 | Trace JSON |
| Gemini 4 Argon | Not completed | 8 | 242.8 | 213,502 | Trace JSON |
ALFWorld · Planning
Clean Soap and Put It on the Counter
Task: Put a cleaned bar of soap on a counter.
5 of 10 recorded models completed this task. Compare actions, feedback and resource use.
Select a decision round to inspect the executed actions, returned feedback and native completion check.
| Model | Outcome | Model calls | Time (s) | Tokens | Evidence |
|---|---|---|---|---|---|
| GPT-6 Astra | Completed | 5.0 | 358.6 | 145,743.0 | Trace JSON |
| GPT-6 Sol | Completed | 6 | 433.8 | 189,608 | Trace JSON |
| Claude Fable 5.1 | Completed | 7 | 197.5 | 169,006 | Trace JSON |
| Gemini 3.8 Flash | Not completed | 8.0 | 189.0 | 175,939.0 | Trace JSON |
| Qwen 3.8 Max | Not completed | 8.0 | 201.2 | 122,093.0 | Trace JSON |
| Qwen 3.5 397B-A17B | Not completed | 8.0 | 138.3 | 133,617.0 | Trace JSON |
| Qwen 3.5 27B | Not completed | 8.0 | 155.6 | 155,526.0 | Trace JSON |
| Claude Opus 5.5 | Completed | 6.0 | 77.6 | 63,724.0 | Trace JSON |
| DeepSeek 4.1 Flash-1017 | Not completed | 8 | 59.1 | 136,094 | Trace JSON |
| Gemini 4 Argon | Completed | 5 | 870.6 | 88,367 | Trace JSON |
ALFWorld · Planning
Inspect a Metal Box Under a Lamp
Task: Turn on a lamp with a metal box.
3 of 10 recorded models completed this task. Compare actions, feedback and resource use.
Select a decision round to inspect the executed actions, returned feedback and native completion check.
| Model | Outcome | Model calls | Time (s) | Tokens | Evidence |
|---|---|---|---|---|---|
| GPT-6 Astra | Completed | 6.0 | 401.7 | 177,316.0 | Trace JSON |
| GPT-6 Sol | Not completed | 8.0 | 173.4 | 122,157.0 | Trace JSON |
| Claude Fable 5.1 | Not completed | 8.0 | 220.7 | 109,716.0 | Trace JSON |
| Gemini 3.8 Flash | Not completed | 8.0 | 173.0 | 141,437.0 | Trace JSON |
| Qwen 3.8 Max | API error | 0.0 | 283.0 | — | Trace JSON |
| Qwen 3.5 397B-A17B | Not completed | 8.0 | 134.0 | 115,959.0 | Trace JSON |
| Qwen 3.5 27B | Not completed | 8.0 | 110.0 | 91,565.0 | Trace JSON |
| Claude Opus 5.5 | Completed | 8 | 80.9 | 91,736 | Trace JSON |
| DeepSeek 4.1 Flash-1017 | Not completed | 8 | 62.1 | 169,067 | Trace JSON |
| Gemini 4 Argon | Completed | 6 | 193.4 | 120,671 | Trace JSON |
CALVIN · Manipulation
Sequence 0270 · Five Goals
Task: Rotate the red block right, close the drawer, lift the red block, stack it, then lift the blue block from the cabinet.
1 of 10 recorded models completed this task. Compare actions, feedback and resource use.
Select a decision round to inspect the executed actions, returned feedback and native completion check.
| Model | Outcome | Model calls | Time (s) | Tokens | Evidence |
|---|---|---|---|---|---|
| GPT-6 Astra | 4/5 goals | 20.0 | 447.2 | 339,664.0 | Trace JSON |
| GPT-6 Sol | 0/5 goals | 29.0 | 1,034.8 | 1,206,468.0 | Trace JSON |
| Claude Fable 5.1 | 5/5 goals | 24 | 566.0 | 538,329 | Trace JSON |
| Gemini 3.8 Flash | 2/5 goals | 36.0 | 592.8 | 1,224,750.0 | Trace JSON |
| Qwen 3.8 Max | 3/5 goals | 24.0 | 3,636.6 | 1,006,296.0 | Trace JSON |
| Qwen 3.5 397B-A17B | 0/5 goals | 26.0 | 292.5 | 1,205,738.0 | Trace JSON |
| Qwen 3.5 27B | 0/5 goals | 30.0 | 286.9 | 1,238,933.0 | Trace JSON |
| Claude Opus 5.5 | 4/5 goals | 40.0 | 1,450.8 | 125,604.0 | Trace JSON |
| DeepSeek 4.1 Flash-1017 | 0/5 goals | 40 | 595.4 | 1,038,865 | Trace JSON |
| Gemini 4 Argon | 4/5 goals | 33 | 1,348.4 | 1,273,471 | Trace JSON |
CALVIN · Manipulation
Sequence 0557 · Five Goals
Task: Complete five subgoals in sequence. Recorded subgoals: press the button to turn off the led light; lift the blue block from the sliding cabinet.
3 of 10 recorded models completed this task. Compare actions, feedback and resource use.
Select a decision round to inspect the executed actions, returned feedback and native completion check.
| Model | Outcome | Model calls | Time (s) | Tokens | Evidence |
|---|---|---|---|---|---|
| GPT-6 Astra | 5/5 goals | 18 | 373.3 | 283,136 | Trace JSON |
| GPT-6 Sol | 0/5 goals | 30.0 | 1,231.4 | 1,256,172.0 | Trace JSON |
| Claude Fable 5.1 | 5/5 goals | 16.0 | 225.9 | 271,110.0 | Trace JSON |
| Gemini 3.8 Flash | API error | 7.0 | 816.9 | 100,076.0 | Trace JSON |
| Qwen 3.8 Max | 0/5 goals | 36.0 | 3,803.3 | 1,219,421.0 | Trace JSON |
| Qwen 3.5 397B-A17B | 0/5 goals | 26.0 | 278.4 | 1,202,842.0 | Trace JSON |
| Qwen 3.5 27B | 0/5 goals | 35.0 | 396.4 | 1,262,798.0 | Trace JSON |
| Claude Opus 5.5 | 1/5 goals | 40.0 | 978.6 | 170,689.0 | Trace JSON |
| DeepSeek 4.1 Flash-1017 | 0/5 goals | 40 | 205.5 | 832,091 | Trace JSON |
| Gemini 4 Argon | 5/5 goals | 12 | 5,603.9 | 211,127 | Trace JSON |
CaPX–BEHAVIOR-1K · Manipulation
Put Shoes on the Rack
Task: putting shoes on rack
0 of 10 recorded models completed this task. Compare actions, feedback and resource use.
Select a decision round to inspect the executed actions, returned feedback and native completion check.
| Model | Outcome | Model calls | Time (s) | Tokens | Evidence |
|---|---|---|---|---|---|
| GPT-6 Astra | Not completed | 24.0 | 742.5 | 521,217.0 | Trace JSON |
| GPT-6 Sol | Not completed | 24.0 | 1,348.5 | 519,707.0 | Trace JSON |
| Claude Fable 5.1 | Not completed | 21.0 | 1,591.2 | 618,279.0 | Trace JSON |
| Gemini 3.8 Flash | Not completed | 8.0 | 175.4 | 130,877.0 | Trace JSON |
| Qwen 3.8 Max | Not completed | 24 | 1,613.8 | 572,778 | Trace JSON |
| Qwen 3.5 397B-A17B | API error | 8.0 | 372.8 | 99,398.0 | Trace JSON |
| Qwen 3.5 27B | API error | 0.0 | 95.8 | — | Trace JSON |
| Claude Opus 5.5 | Not completed | 20.0 | 1,249.6 | 599,155.0 | Trace JSON |
| DeepSeek 4.1 Flash-1017 | Not completed | 24 | 2,146.2 | 523,219 | Trace JSON |
| Gemini 4 Argon | Not completed | 19 | 761.9 | 602,401 | Trace JSON |
CaPX–LIBERO-PRO · Manipulation
Turn On the Stove
Task: turn on the stove
6 of 10 recorded models completed this task. Compare actions, feedback and resource use.
Select a decision round to inspect the executed actions, returned feedback and native completion check.
| Model | Outcome | Model calls | Time (s) | Tokens | Evidence |
|---|---|---|---|---|---|
| GPT-6 Astra | Completed | 15 | 625.1 | 214,593 | Trace JSON |
| GPT-6 Sol | Completed | 8 | 303.8 | 71,096 | Trace JSON |
| Claude Fable 5.1 | Completed | 20.0 | 1,245.8 | 452,686.0 | Trace JSON |
| Gemini 3.8 Flash | Completed | 16.0 | 311.4 | 358,602.0 | Trace JSON |
| Qwen 3.8 Max | Not completed | 19.0 | 2,220.1 | 594,795.0 | Trace JSON |
| Qwen 3.5 397B-A17B | Not completed | 24.0 | 427.6 | 303,069.0 | Trace JSON |
| Qwen 3.5 27B | Not completed | 24.0 | 292.3 | 408,828.0 | Trace JSON |
| Claude Opus 5.5 | Completed | 13.0 | 294.6 | 95,030.0 | Trace JSON |
| DeepSeek 4.1 Flash-1017 | Not completed | 24 | 796.1 | 629,486 | Trace JSON |
| Gemini 4 Argon | Completed | 9 | 296.5 | 132,324 | Trace JSON |
CaPX–LIBERO-PRO · Manipulation
Put Tomato Sauce in the Basket
Task: pick up the tomato sauce and place it in the basket
5 of 10 recorded models completed this task. Compare actions, feedback and resource use.
Select a decision round to inspect the executed actions, returned feedback and native completion check.
| Model | Outcome | Model calls | Time (s) | Tokens | Evidence |
|---|---|---|---|---|---|
| GPT-6 Astra | Not completed | 19.0 | 835.4 | 453,857.0 | Trace JSON |
| GPT-6 Sol | Not completed | 24.0 | 1,260.8 | 400,030.0 | Trace JSON |
| Claude Fable 5.1 | Completed | 16 | 1,553.5 | 337,613 | Trace JSON |
| Gemini 3.8 Flash | Completed | 15 | 429.0 | 321,238 | Trace JSON |
| Qwen 3.8 Max | Not completed | 22.0 | 2,069.6 | 628,440.0 | Trace JSON |
| Qwen 3.5 397B-A17B | Not completed | 24 | 934.6 | 466,647 | Trace JSON |
| Qwen 3.5 27B | Not completed | 24.0 | 547.5 | 414,652.0 | Trace JSON |
| Claude Opus 5.5 | Completed | 20 | 2,376.8 | 159,535 | Trace JSON |
| DeepSeek 4.1 Flash-1017 | Completed | 11 | 396.0 | 110,098 | Trace JSON |
| Gemini 4 Argon | Completed | 10 | 576.1 | 220,087 | Trace JSON |
CaPX–Robosuite · Manipulation
Wipe the Spill
Task: spill wipe
9 of 10 recorded models completed this task. Compare actions, feedback and resource use.
Select a decision round to inspect the executed actions, returned feedback and native completion check.
| Model | Outcome | Model calls | Time (s) | Tokens | Evidence |
|---|---|---|---|---|---|
| GPT-6 Astra | Completed | 2 | 284.1 | 18,087 | Trace JSON |
| GPT-6 Sol | Completed | 5 | 183.9 | 42,444 | Trace JSON |
| Claude Fable 5.1 | Completed | 4 | 149.7 | 42,773 | Trace JSON |
| Gemini 3.8 Flash | Completed | 3 | 66.0 | 36,374 | Trace JSON |
| Qwen 3.8 Max | Completed | 2 | 445.9 | 31,852 | Trace JSON |
| Qwen 3.5 397B-A17B | Completed | 5 | 123.3 | 44,133 | Trace JSON |
| Qwen 3.5 27B | Not completed | 19.0 | 766.3 | 622,760.0 | Trace JSON |
| Claude Opus 5.5 | Completed | 5 | 148.9 | 68,221 | Trace JSON |
| DeepSeek 4.1 Flash-1017 | Completed | 13 | 966.4 | 183,940 | Trace JSON |
| Gemini 4 Argon | Completed | 5 | 158.4 | 82,720 | Trace JSON |
CLIPort · Manipulation
Pack Matching Objects
Task: packing boxes pairs unseen colors
3 of 10 recorded models completed this task. Compare actions, feedback and resource use.
Select a decision round to inspect the executed actions, returned feedback and native completion check.
| Model | Outcome | Model calls | Time (s) | Tokens | Evidence |
|---|---|---|---|---|---|
| GPT-6 Astra | Completed | 3.0 | 131.5 | 58,347.0 | Trace JSON |
| GPT-6 Sol | Not completed | 18.0 | 898.3 | 295,041.0 | Trace JSON |
| Claude Fable 5.1 | Completed | 10.0 | 433.2 | 142,157.0 | Trace JSON |
| Gemini 3.8 Flash | Not completed | 16.0 | 225.6 | 294,947.0 | Trace JSON |
| Qwen 3.8 Max | Not completed | 10.0 | 2,474.1 | 250,091.0 | Trace JSON |
| Qwen 3.5 397B-A17B | Not completed | 5.0 | 94.1 | 56,294.0 | Trace JSON |
| Qwen 3.5 27B | Not completed | 3.0 | 40.3 | 31,429.0 | Trace JSON |
| Claude Opus 5.5 | Completed | 11 | 654.3 | 160,208 | Trace JSON |
| DeepSeek 4.1 Flash-1017 | Not completed | 24 | 556.9 | 444,434 | Trace JSON |
| Gemini 4 Argon | Not completed | 9 | 388.5 | 141,113 | Trace JSON |
CLIPort · Manipulation
Towers of Hanoi
Task: Towers of hanoi seq unseen colors
5 of 10 recorded models completed this task. Compare actions, feedback and resource use.
Select a decision round to inspect the executed actions, returned feedback and native completion check.
| Model | Outcome | Model calls | Time (s) | Tokens | Evidence |
|---|---|---|---|---|---|
| GPT-6 Astra | Completed | 7 | 362.1 | 163,427 | Trace JSON |
| GPT-6 Sol | Completed | 18.0 | 518.6 | 273,178.0 | Trace JSON |
| Claude Fable 5.1 | Completed | 11 | 427.3 | 149,420 | Trace JSON |
| Gemini 3.8 Flash | Not completed | 24.0 | 356.5 | 618,078.0 | Trace JSON |
| Qwen 3.8 Max | Not completed | 19.0 | 2,070.6 | 891,589.0 | Trace JSON |
| Qwen 3.5 397B-A17B | API error | 7.0 | 102.3 | 100,733.0 | Trace JSON |
| Qwen 3.5 27B | Not completed | 21.0 | 515.2 | 873,908.0 | Trace JSON |
| Claude Opus 5.5 | Completed | 8.0 | 474.4 | 81,639.0 | Trace JSON |
| DeepSeek 4.1 Flash-1017 | Not completed | 24.0 | 451.3 | 498,585.0 | Trace JSON |
| Gemini 4 Argon | Completed | 10 | 924.7 | 173,803 | Trace JSON |
DiscoveryWorld · Planning
Remove Rust from a Key
Task: Identify the rust-removing chemical, clean the key, then unlock the shed and leave.
4 of 10 recorded models completed this task. Compare actions, feedback and resource use.
Select a decision round to inspect the executed actions, returned feedback and native completion check.
| Model | Outcome | Model calls | Time (s) | Tokens | Evidence |
|---|---|---|---|---|---|
| GPT-6 Astra | Completed | 4.0 | 226.4 | 120,440.0 | Trace JSON |
| GPT-6 Sol | Completed | 8 | 250.3 | 262,849 | Trace JSON |
| Claude Fable 5.1 | Not completed | 8.0 | 215.9 | 108,029.0 | Trace JSON |
| Gemini 3.8 Flash | Completed | 8 | 59.7 | 236,109 | Trace JSON |
| Qwen 3.8 Max | Not completed | 8.0 | 357.9 | 232,372.0 | Trace JSON |
| Qwen 3.5 397B-A17B | Not completed | 8.0 | 127.3 | 245,276.0 | Trace JSON |
| Qwen 3.5 27B | Not completed | 8.0 | 187.2 | 247,493.0 | Trace JSON |
| Claude Opus 5.5 | Not completed | 8.0 | 281.1 | 80,615.0 | Trace JSON |
| DeepSeek 4.1 Flash-1017 | Not completed | 8.0 | 66.9 | 220,835.0 | Trace JSON |
| Gemini 4 Argon | Completed | 7 | 99.8 | 189,635 | Trace JSON |
DiscoveryWorld · Planning
Give Yugo the Spectrometer
Task: Your task is to give Yugo the `spectrometer`.
9 of 10 recorded models completed this task. Compare actions, feedback and resource use.
Select a decision round to inspect the executed actions, returned feedback and native completion check.
| Model | Outcome | Model calls | Time (s) | Tokens | Evidence |
|---|---|---|---|---|---|
| GPT-6 Astra | Completed | 2.0 | 84.1 | 44,101.0 | Trace JSON |
| GPT-6 Sol | Completed | 5.0 | 109.3 | 125,915.0 | Trace JSON |
| Claude Fable 5.1 | Completed | 4.0 | 90.1 | 5,691.0 | Trace JSON |
| Gemini 3.8 Flash | Completed | 7.0 | 36.1 | 161,996.0 | Trace JSON |
| Qwen 3.8 Max | Completed | 4.0 | 291.8 | 70,657.0 | Trace JSON |
| Qwen 3.5 397B-A17B | Not completed | 8.0 | 124.0 | 227,965.0 | Trace JSON |
| Qwen 3.5 27B | Completed | 7 | 110.0 | 162,447 | Trace JSON |
| Claude Opus 5.5 | Completed | 5.0 | 90.4 | 36,658.0 | Trace JSON |
| DeepSeek 4.1 Flash-1017 | Completed | 7 | 33.7 | 155,543 | Trace JSON |
| Gemini 4 Argon | Completed | 3 | 41.1 | 33,857 | Trace JSON |
HumanCLAW · Planning
Sit on the Couch
Task: Find the couch and move until your body is touching the target, with zero distance to the target object. Finally, sit on the couch.
0 of 10 recorded models completed this task. Compare actions, feedback and resource use.
Select a decision round to inspect the executed actions, returned feedback and native completion check.
| Model | Outcome | Model calls | Time (s) | Tokens | Evidence |
|---|---|---|---|---|---|
| GPT-6 Astra | Not completed | 44.0 | 1,275.3 | 1,077,381.0 | Trace JSON |
| GPT-6 Sol | Not completed | 25.0 | 1,100.3 | 537,099.0 | Trace JSON |
| Claude Fable 5.1 | Not completed | 24.0 | 649.6 | 237,596.0 | Trace JSON |
| Gemini 3.8 Flash | Not completed | 18.0 | 171.1 | 171,054.0 | Trace JSON |
| Qwen 3.8 Max | Not completed | 77 | 5,526.1 | 1,969,448 | Trace JSON |
| Qwen 3.5 397B-A17B | Not completed | 19 | 1,435.4 | 105,975 | Trace JSON |
| Qwen 3.5 27B | Not completed | 82 | 675.8 | 1,793,349 | Trace JSON |
| Claude Opus 5.5 | Not completed | 51.0 | 691.9 | 965,711.0 | Trace JSON |
| DeepSeek 4.1 Flash-1017 | Not completed | 70 | 496.5 | 1,106,965 | Trace JSON |
| Gemini 4 Argon | Not completed | 46 | 4,450.5 | 666,909 | Trace JSON |
ManiSkill · Manipulation
Place the Sphere
Task: PlaceSphere-v1
3 of 10 recorded models completed this task. Compare actions, feedback and resource use.
Select a decision round to inspect the executed actions, returned feedback and native completion check.
| Model | Outcome | Model calls | Time (s) | Tokens | Evidence |
|---|---|---|---|---|---|
| GPT-6 Astra | Not completed | 13.0 | 439.5 | 171,642.0 | Trace JSON |
| GPT-6 Sol | Not completed | 26.0 | 815.1 | 602,149.0 | Trace JSON |
| Claude Fable 5.1 | Completed | 14.0 | 248.8 | 214,638.0 | Trace JSON |
| Gemini 3.8 Flash | Not completed | 21.0 | 944.9 | 429,431.0 | Trace JSON |
| Qwen 3.8 Max | Not completed | 16.0 | 2,121.0 | 393,523.0 | Trace JSON |
| Qwen 3.5 397B-A17B | Not completed | 23.0 | 234.9 | 622,905.0 | Trace JSON |
| Qwen 3.5 27B | Not completed | 15.0 | 184.0 | 430,328.0 | Trace JSON |
| Claude Opus 5.5 | Completed | 13 | 477.6 | 143,296 | Trace JSON |
| DeepSeek 4.1 Flash-1017 | Not completed | 13 | 127.0 | 176,822 | Trace JSON |
| Gemini 4 Argon | Completed | 10 | 253.7 | 168,875 | Trace JSON |
RLBench · Manipulation
Change the TV Channel
Task: Change channel
3 of 10 recorded models completed this task. Compare actions, feedback and resource use.
Select a decision round to inspect the executed actions, returned feedback and native completion check.
| Model | Outcome | Model calls | Time (s) | Tokens | Evidence |
|---|---|---|---|---|---|
| GPT-6 Astra | Completed | 6 | 304.6 | 164,557 | Trace JSON |
| GPT-6 Sol | Not completed | 8.0 | 932.0 | 247,414.0 | Trace JSON |
| Claude Fable 5.1 | Completed | 11 | 266.3 | 204,366 | Trace JSON |
| Gemini 3.8 Flash | Not completed | 16.0 | 475.9 | 609,140.0 | Trace JSON |
| Qwen 3.8 Max | Not completed | 16.0 | 2,911.4 | 414,397.0 | Trace JSON |
| Qwen 3.5 397B-A17B | Not completed | 11.0 | 166.3 | 627,512.0 | Trace JSON |
| Qwen 3.5 27B | Not completed | 12.0 | 183.9 | 562,456.0 | Trace JSON |
| Claude Opus 5.5 | Not completed | 7.0 | 903.8 | 116,237.0 | Trace JSON |
| DeepSeek 4.1 Flash-1017 | Not completed | 16.0 | 514.9 | 289,217.0 | Trace JSON |
| Gemini 4 Argon | Completed | 13 | 673.4 | 381,782 | Trace JSON |
RLBench · Manipulation
Close the Laptop
Task: Close Laptop Lid
6 of 10 recorded models completed this task. Compare actions, feedback and resource use.
Select a decision round to inspect the executed actions, returned feedback and native completion check.
| Model | Outcome | Model calls | Time (s) | Tokens | Evidence |
|---|---|---|---|---|---|
| GPT-6 Astra | Completed | 5.0 | 246.5 | 131,455.0 | Trace JSON |
| GPT-6 Sol | Completed | 8.0 | 282.2 | 133,191.0 | Trace JSON |
| Claude Fable 5.1 | Completed | 10.0 | 411.1 | 151,150.0 | Trace JSON |
| Gemini 3.8 Flash | Not completed | 16.0 | 443.7 | 465,552.0 | Trace JSON |
| Qwen 3.8 Max | Completed | 13 | 3,076.4 | 475,357 | Trace JSON |
| Qwen 3.5 397B-A17B | Not completed | 14.0 | 183.9 | 569,707.0 | Trace JSON |
| Qwen 3.5 27B | Not completed | 12.0 | 251.1 | 606,978.0 | Trace JSON |
| Claude Opus 5.5 | Completed | 8.0 | 233.5 | 66,223.0 | Trace JSON |
| DeepSeek 4.1 Flash-1017 | Not completed | 16 | 244.4 | 242,844 | Trace JSON |
| Gemini 4 Argon | Completed | 7 | 211.8 | 131,204 | Trace JSON |
RLBench · Manipulation
Open the Box
Task: Open Box
1 of 10 recorded models completed this task. Compare actions, feedback and resource use.
Select a decision round to inspect the executed actions, returned feedback and native completion check.
| Model | Outcome | Model calls | Time (s) | Tokens | Evidence |
|---|---|---|---|---|---|
| GPT-6 Astra | Not completed | 8.0 | 277.3 | 213,059.0 | Trace JSON |
| GPT-6 Sol | Not completed | 8.0 | 736.4 | 150,982.0 | Trace JSON |
| Claude Fable 5.1 | Not completed | 9.0 | 1,050.6 | 119,694.0 | Trace JSON |
| Gemini 3.8 Flash | Not completed | 13.0 | 203.6 | 561,286.0 | Trace JSON |
| Qwen 3.8 Max | Not completed | 16.0 | 2,386.8 | 412,856.0 | Trace JSON |
| Qwen 3.5 397B-A17B | Not completed | 15 | 198.6 | 628,112 | Trace JSON |
| Qwen 3.5 27B | Not completed | 15.0 | 183.8 | 630,124.0 | Trace JSON |
| Claude Opus 5.5 | Not completed | 16.0 | 1,058.5 | 148,922.0 | Trace JSON |
| DeepSeek 4.1 Flash-1017 | Not completed | 16 | 361.7 | 230,196 | Trace JSON |
| Gemini 4 Argon | Completed | 12 | 1,503.3 | 272,130 | Trace JSON |
RLBench · Manipulation
Take the Cup Out of the Cabinet
Task: Take Cup Out From Cabinet
1 of 10 recorded models completed this task. Compare actions, feedback and resource use.
Select a decision round to inspect the executed actions, returned feedback and native completion check.
| Model | Outcome | Model calls | Time (s) | Tokens | Evidence |
|---|---|---|---|---|---|
| GPT-6 Astra | Not completed | 8.0 | 505.8 | 238,250.0 | Trace JSON |
| GPT-6 Sol | Not completed | 8.0 | 331.2 | 209,061.0 | Trace JSON |
| Claude Fable 5.1 | Not completed | 16.0 | 1,109.0 | 341,636.0 | Trace JSON |
| Gemini 3.8 Flash | Not completed | 8.0 | 883.5 | 289,023.0 | Trace JSON |
| Qwen 3.8 Max | Not completed | 16.0 | 2,116.7 | 474,645.0 | Trace JSON |
| Qwen 3.5 397B-A17B | Not completed | 13.0 | 208.9 | 569,807.0 | Trace JSON |
| Qwen 3.5 27B | Not completed | 11 | 401.4 | 570,376 | Trace JSON |
| Claude Opus 5.5 | Completed | 14 | 878.5 | 150,423 | Trace JSON |
| DeepSeek 4.1 Flash-1017 | Not completed | 16 | 116.3 | 346,600 | Trace JSON |
| Gemini 4 Argon | Not completed | 16 | 421.5 | 535,234 | Trace JSON |
RoboCasa · Manipulation
Turn On the Faucet
Task: Turn on the sink faucet.
0 of 10 recorded models completed this task. Compare actions, feedback and resource use.
Select a decision round to inspect the executed actions, returned feedback and native completion check.
| Model | Outcome | Model calls | Time (s) | Tokens | Evidence |
|---|---|---|---|---|---|
| GPT-6 Astra | Not completed | 24.0 | 1,345.4 | 312,192.0 | Trace JSON |
| GPT-6 Sol | Not completed | 24.0 | 1,147.6 | 272,594.0 | Trace JSON |
| Claude Fable 5.1 | Not completed | 21.0 | 484.7 | 342,590.0 | Trace JSON |
| Gemini 3.8 Flash | Not completed | 24.0 | 314.0 | 417,233.0 | Trace JSON |
| Qwen 3.8 Max | API error | 4.0 | 185.8 | 41,096.0 | Trace JSON |
| Qwen 3.5 397B-A17B | Not completed | 24.0 | 218.3 | 226,141.0 | Trace JSON |
| Qwen 3.5 27B | Not completed | 24.0 | 179.9 | 258,320.0 | Trace JSON |
| Claude Opus 5.5 | Not completed | 22 | 544.4 | 225,219 | Trace JSON |
| DeepSeek 4.1 Flash-1017 | Not completed | 24 | 309.1 | 228,685 | Trace JSON |
| Gemini 4 Argon | Not completed | 24 | 1,105.4 | 807,967 | Trace JSON |
RoboCasa365 · Manipulation
Put Bread and Cheese on the Board
Task: Place the bread and cheese on the cutting board.
0 of 10 recorded models completed this task. Compare actions, feedback and resource use.
Select a decision round to inspect the executed actions, returned feedback and native completion check.
| Model | Outcome | Model calls | Time (s) | Tokens | Evidence |
|---|---|---|---|---|---|
| GPT-6 Astra | Not completed | 24 | 816.8 | 397,237 | Trace JSON |
| GPT-6 Sol | Not completed | 24.0 | 1,165.1 | 310,731.0 | Trace JSON |
| Claude Fable 5.1 | Not completed | 24.0 | 729.3 | 375,845.0 | Trace JSON |
| Gemini 3.8 Flash | Not completed | 24.0 | 311.4 | 420,226.0 | Trace JSON |
| Qwen 3.8 Max | Not completed | 24.0 | 2,288.7 | 518,702.0 | Trace JSON |
| Qwen 3.5 397B-A17B | Not completed | 24.0 | 389.8 | 308,816.0 | Trace JSON |
| Qwen 3.5 27B | Not completed | 24.0 | 225.5 | 281,405.0 | Trace JSON |
| Claude Opus 5.5 | Not completed | 23.0 | 1,179.6 | 194,849.0 | Trace JSON |
| DeepSeek 4.1 Flash-1017 | Not completed | 24 | 167.1 | 262,074 | Trace JSON |
| Gemini 4 Argon | Not completed | 24 | 1,157.3 | 514,019 | Trace JSON |
RoboCasa365 · Manipulation
Close the Left Drawer
Task: Close the left drawer.
2 of 10 recorded models completed this task. Compare actions, feedback and resource use.
Select a decision round to inspect the executed actions, returned feedback and native completion check.
| Model | Outcome | Model calls | Time (s) | Tokens | Evidence |
|---|---|---|---|---|---|
| GPT-6 Astra | Not completed | 24.0 | 930.7 | 332,271.0 | Trace JSON |
| GPT-6 Sol | Not completed | 24.0 | 634.7 | 268,102.0 | Trace JSON |
| Claude Fable 5.1 | Completed | 21.0 | 1,006.4 | 428,400.0 | Trace JSON |
| Gemini 3.8 Flash | Not completed | 24.0 | 519.3 | 1,074,988.0 | Trace JSON |
| Qwen 3.8 Max | Not completed | 24 | 2,435.0 | 615,543 | Trace JSON |
| Qwen 3.5 397B-A17B | Not completed | 24.0 | 240.0 | 260,704.0 | Trace JSON |
| Qwen 3.5 27B | Not completed | 24 | 282.1 | 289,607 | Trace JSON |
| Claude Opus 5.5 | Completed | 16 | 929.6 | 170,405 | Trace JSON |
| DeepSeek 4.1 Flash-1017 | Not completed | 24 | 147.0 | 245,500 | Trace JSON |
| Gemini 4 Argon | Not completed | 24 | 1,388.0 | 956,340 | Trace JSON |
RoboCasa365 · Manipulation
Clear the Cutting Board
Task: Remove the broccoli and steak from the cutting board to prepare it for cleaning.
2 of 10 recorded models completed this task. Compare actions, feedback and resource use.
Select a decision round to inspect the executed actions, returned feedback and native completion check.
| Model | Outcome | Model calls | Time (s) | Tokens | Evidence |
|---|---|---|---|---|---|
| GPT-6 Astra | Not completed | 24.0 | 534.2 | 281,339.0 | Trace JSON |
| GPT-6 Sol | Not completed | 24 | 616.2 | 276,891 | Trace JSON |
| Claude Fable 5.1 | Completed | 11 | 230.0 | 152,068 | Trace JSON |
| Gemini 3.8 Flash | Not completed | 24 | 318.0 | 450,096 | Trace JSON |
| Qwen 3.8 Max | API error | 8.0 | 645.6 | 102,995.0 | Trace JSON |
| Qwen 3.5 397B-A17B | Not completed | 24 | 691.4 | 305,084 | Trace JSON |
| Qwen 3.5 27B | Not completed | 24.0 | 594.2 | 295,305.0 | Trace JSON |
| Claude Opus 5.5 | Not completed | 24.0 | 394.7 | 215,193.0 | Trace JSON |
| DeepSeek 4.1 Flash-1017 | Not completed | 24 | 151.9 | 251,548 | Trace JSON |
| Gemini 4 Argon | Completed | 11 | 266.9 | 143,069 | Trace JSON |
RoboTwin2 · Manipulation
Order Blocks by Size
Task: blocks ranking size
3 of 10 recorded models completed this task. Compare actions, feedback and resource use.
Select a decision round to inspect the executed actions, returned feedback and native completion check.
| Model | Outcome | Model calls | Time (s) | Tokens | Evidence |
|---|---|---|---|---|---|
| GPT-6 Astra | Completed | 16.0 | 513.3 | 482,973.0 | Trace JSON |
| GPT-6 Sol | Not completed | 16.0 | 521.5 | 547,854.0 | Trace JSON |
| Claude Fable 5.1 | Completed | 13 | 597.7 | 329,910 | Trace JSON |
| Gemini 3.8 Flash | Not completed | 15.0 | 264.1 | 589,519.0 | Trace JSON |
| Qwen 3.8 Max | Not completed | 16.0 | 2,652.3 | 593,765.0 | Trace JSON |
| Qwen 3.5 397B-A17B | Not completed | 16.0 | 178.3 | 532,082.0 | Trace JSON |
| Qwen 3.5 27B | Not completed | 15.0 | 190.8 | 583,709.0 | Trace JSON |
| Claude Opus 5.5 | Completed | 9.0 | 328.2 | 75,980.0 | Trace JSON |
| DeepSeek 4.1 Flash-1017 | Not completed | 16 | 588.8 | 363,460 | Trace JSON |
| Gemini 4 Argon | Not completed | 15 | 845.8 | 591,628 | Trace JSON |
RoboTwin2 · Manipulation
Place the Shoe
Task: place shoe
2 of 10 recorded models completed this task. Compare actions, feedback and resource use.
Select a decision round to inspect the executed actions, returned feedback and native completion check.
| Model | Outcome | Model calls | Time (s) | Tokens | Evidence |
|---|---|---|---|---|---|
| GPT-6 Astra | Completed | 16.0 | 376.3 | 426,171.0 | Trace JSON |
| GPT-6 Sol | Not completed | 16.0 | 432.3 | 464,978.0 | Trace JSON |
| Claude Fable 5.1 | Not completed | 16.0 | 724.1 | 354,033.0 | Trace JSON |
| Gemini 3.8 Flash | Not completed | 16.0 | 189.3 | 580,295.0 | Trace JSON |
| Qwen 3.8 Max | Not completed | 16.0 | 489.6 | 314,128.0 | Trace JSON |
| Qwen 3.5 397B-A17B | Not completed | 14.0 | 179.3 | 583,658.0 | Trace JSON |
| Qwen 3.5 27B | Not completed | 16.0 | 192.4 | 537,260.0 | Trace JSON |
| Claude Opus 5.5 | Completed | 14 | 1,035.9 | 76,024 | Trace JSON |
| DeepSeek 4.1 Flash-1017 | Not completed | 16 | 350.8 | 381,209 | Trace JSON |
| Gemini 4 Argon | Not completed | 15 | 388.6 | 598,306 | Trace JSON |
RoboTwin2 · Manipulation
Stack Three Blocks
Task: stack blocks three
1 of 10 recorded models completed this task. Compare actions, feedback and resource use.
Select a decision round to inspect the executed actions, returned feedback and native completion check.
| Model | Outcome | Model calls | Time (s) | Tokens | Evidence |
|---|---|---|---|---|---|
| GPT-6 Astra | Completed | 10 | 274.0 | 213,426 | Trace JSON |
| GPT-6 Sol | Not completed | 16.0 | 1,619.4 | 358,357.0 | Trace JSON |
| Claude Fable 5.1 | Not completed | 16.0 | 658.3 | 522,047.0 | Trace JSON |
| Gemini 3.8 Flash | Not completed | 16.0 | 349.5 | 505,318.0 | Trace JSON |
| Qwen 3.8 Max | Not completed | 14.0 | 1,800.9 | 628,675.0 | Trace JSON |
| Qwen 3.5 397B-A17B | Not completed | 15.0 | 212.6 | 625,138.0 | Trace JSON |
| Qwen 3.5 27B | Not completed | 16.0 | 259.3 | 572,365.0 | Trace JSON |
| Claude Opus 5.5 | Not completed | 16.0 | 452.4 | 100,221.0 | Trace JSON |
| DeepSeek 4.1 Flash-1017 | Not completed | 16 | 665.5 | 340,973 | Trace JSON |
| Gemini 4 Argon | Not completed | 14 | 377.2 | 584,720 | Trace JSON |
RoboWits · Manipulation
Push the Red Block to the Green Target
Task: Push the red block onto the green target area.
5 of 10 recorded models completed this task. Compare actions, feedback and resource use.
Select a decision round to inspect the executed actions, returned feedback and native completion check.
| Model | Outcome | Model calls | Time (s) | Tokens | Evidence |
|---|---|---|---|---|---|
| GPT-6 Astra | Not completed | 17.0 | 676.8 | 598,205.0 | Trace JSON |
| GPT-6 Sol | Not completed | 19.0 | 920.2 | 604,005.0 | Trace JSON |
| Claude Fable 5.1 | Completed | 6.0 | 192.9 | 97,772.0 | Trace JSON |
| Gemini 3.8 Flash | Completed | 14 | 200.1 | 324,990 | Trace JSON |
| Qwen 3.8 Max | Completed | 12 | 952.0 | 213,956 | Trace JSON |
| Qwen 3.5 397B-A17B | Not completed | 11.0 | 170.3 | 555,628.0 | Trace JSON |
| Qwen 3.5 27B | Not completed | 13.0 | 696.3 | 526,305.0 | Trace JSON |
| Claude Opus 5.5 | Completed | 8 | 183.8 | 66,541 | Trace JSON |
| DeepSeek 4.1 Flash-1017 | Completed | 3 | 35.6 | 32,688 | Trace JSON |
| Gemini 4 Argon | Not completed | 14 | 285.1 | 552,778 | Trace JSON |
RoboWits · Manipulation
Task 10 · Screw, Dustpan & Container
Task: Task 10 · screw, dustpan and container (episode 000).
7 of 10 recorded models completed this task. Compare actions, feedback and resource use.
Select a decision round to inspect the executed actions, returned feedback and native completion check.
| Model | Outcome | Model calls | Time (s) | Tokens | Evidence |
|---|---|---|---|---|---|
| GPT-6 Astra | Completed | 20 | 698.3 | 448,018 | Trace JSON |
| GPT-6 Sol | Completed | 17 | 684.9 | 394,967 | Trace JSON |
| Claude Fable 5.1 | Completed | 15 | 769.7 | 444,273 | Trace JSON |
| Gemini 3.8 Flash | Not completed | 18.0 | 467.6 | 605,873.0 | Trace JSON |
| Qwen 3.8 Max | Completed | 11.0 | 968.3 | 239,134.0 | Trace JSON |
| Qwen 3.5 397B-A17B | Not completed | 22 | 578.7 | 594,747 | Trace JSON |
| Qwen 3.5 27B | Completed | 5 | 179.5 | 71,050 | Trace JSON |
| Claude Opus 5.5 | Not completed | 24.0 | 1,311.8 | 170,467.0 | Trace JSON |
| DeepSeek 4.1 Flash-1017 | Completed | 7 | 66.0 | 82,092 | Trace JSON |
| Gemini 4 Argon | Completed | 6 | 2,804.3 | 103,168 | Trace JSON |
ScienceWorld · Planning
Find an Animal
Task: Your task is to find a(n) animal. First, focus on the thing. Then, move it to the green box in the workshop.
4 of 10 recorded models completed this task. Compare actions, feedback and resource use.
Select a decision round to inspect the executed actions, returned feedback and native completion check.
| Model | Outcome | Model calls | Time (s) | Tokens | Evidence |
|---|---|---|---|---|---|
| GPT-6 Astra | Completed | 4.0 | 119.5 | 81,630.0 | Trace JSON |
| GPT-6 Sol | Not completed | 8.0 | 501.6 | 224,834.0 | Trace JSON |
| Claude Fable 5.1 | Completed | 4.0 | 63.5 | 21,871.0 | Trace JSON |
| Gemini 3.8 Flash | Not completed | 8.0 | 32.8 | 67,145.0 | Trace JSON |
| Qwen 3.8 Max | Not completed | 8.0 | 184.3 | 77,332.0 | Trace JSON |
| Qwen 3.5 397B-A17B | Not completed | 8.0 | 82.4 | 71,151.0 | Trace JSON |
| Qwen 3.5 27B | Not completed | 4.0 | 47.9 | 24,423.0 | Trace JSON |
| Claude Opus 5.5 | Completed | 9 | 224.6 | 25,233 | Trace JSON |
| DeepSeek 4.1 Flash-1017 | Not completed | 8 | 25.1 | 63,349 | Trace JSON |
| Gemini 4 Argon | Completed | 7 | 99.7 | 55,322 | Trace JSON |
ScienceWorld · Planning
Test Water Conductivity
Task: Test the water’s electrical conductivity. Put conductive water in the blue box, otherwise in the orange box.
2 of 10 recorded models completed this task. Compare actions, feedback and resource use.
Select a decision round to inspect the executed actions, returned feedback and native completion check.
| Model | Outcome | Model calls | Time (s) | Tokens | Evidence |
|---|---|---|---|---|---|
| GPT-6 Astra | Completed | 8 | 353.1 | 201,345 | Trace JSON |
| GPT-6 Sol | Not completed | 7.0 | 507.0 | 177,327.0 | Trace JSON |
| Claude Fable 5.1 | Completed | 8 | 128.3 | 43,741 | Trace JSON |
| Gemini 3.8 Flash | Not completed | 8.0 | 36.9 | 62,916.0 | Trace JSON |
| Qwen 3.8 Max | Not completed | 8.0 | 582.9 | 105,089.0 | Trace JSON |
| Qwen 3.5 397B-A17B | Not completed | 8.0 | 79.8 | 59,840.0 | Trace JSON |
| Qwen 3.5 27B | Not completed | 8.0 | 77.9 | 63,549.0 | Trace JSON |
| Claude Opus 5.5 | Not completed | 8.0 | 180.6 | 36,159.0 | Trace JSON |
| DeepSeek 4.1 Flash-1017 | Not completed | 8.0 | 62.3 | 69,168.0 | Trace JSON |
| Gemini 4 Argon | Not completed | 8 | 370.4 | 71,987 | Trace JSON |
VIMA-Bench · Manipulation
Sweep Without Touching
Task: sweep without touching
4 of 10 recorded models completed this task. Compare actions, feedback and resource use.
Select a decision round to inspect the executed actions, returned feedback and native completion check.
| Model | Outcome | Model calls | Time (s) | Tokens | Evidence |
|---|---|---|---|---|---|
| GPT-6 Astra | Completed | 7 | 96.8 | 59,118 | Trace JSON |
| GPT-6 Sol | Not completed | 16.0 | 167.6 | 242,580.0 | Trace JSON |
| Claude Fable 5.1 | Completed | 11 | 401.2 | 159,714 | Trace JSON |
| Gemini 3.8 Flash | Not completed | 16.0 | 216.7 | 440,383.0 | Trace JSON |
| Qwen 3.8 Max | Not completed | 16.0 | 2,139.9 | 429,522.0 | Trace JSON |
| Qwen 3.5 397B-A17B | Not completed | 16.0 | 168.9 | 393,377.0 | Trace JSON |
| Qwen 3.5 27B | Not completed | 16.0 | 179.5 | 376,815.0 | Trace JSON |
| Claude Opus 5.5 | Completed | 13 | 229.4 | 185,656 | Trace JSON |
| DeepSeek 4.1 Flash-1017 | Completed | 13 | 246.4 | 204,104 | Trace JSON |
| Gemini 4 Argon | Not completed | 16 | 704.3 | 433,521 | Trace JSON |
VIMA-Bench · Manipulation
Rearrange, Then Restore
Task: rearrange then restore
6 of 10 recorded models completed this task. Compare actions, feedback and resource use.
Select a decision round to inspect the executed actions, returned feedback and native completion check.
| Model | Outcome | Model calls | Time (s) | Tokens | Evidence |
|---|---|---|---|---|---|
| GPT-6 Astra | Completed | 5.0 | 76.2 | 29,392.0 | Trace JSON |
| GPT-6 Sol | Completed | 9 | 262.1 | 93,800 | Trace JSON |
| Claude Fable 5.1 | Completed | 6.0 | 206.8 | 62,849.0 | Trace JSON |
| Gemini 3.8 Flash | Completed | 16 | 300.6 | 308,096 | Trace JSON |
| Qwen 3.8 Max | Not completed | 16.0 | 1,790.6 | 307,028.0 | Trace JSON |
| Qwen 3.5 397B-A17B | Not completed | 16.0 | 170.0 | 259,246.0 | Trace JSON |
| Qwen 3.5 27B | Not completed | 16.0 | 187.4 | 299,495.0 | Trace JSON |
| Claude Opus 5.5 | Completed | 7.0 | 321.0 | 91,739.0 | Trace JSON |
| DeepSeek 4.1 Flash-1017 | Not completed | 16 | 196.9 | 196,872 | Trace JSON |
| Gemini 4 Argon | Completed | 10 | 382.0 | 163,563 | Trace JSON |
VirtualHome · Planning
Pick Up the Phone
Task: Pick up phone: walk to living room , find phone ,pick up phone 3
7 of 10 recorded models completed this task. Compare actions, feedback and resource use.
Select a decision round to inspect the executed actions, returned feedback and native completion check.
| Model | Outcome | Model calls | Time (s) | Tokens | Evidence |
|---|---|---|---|---|---|
| GPT-6 Astra | Completed | 3.0 | 200.0 | 70,119.0 | Trace JSON |
| GPT-6 Sol | Completed | 7.0 | 600.9 | 296,976.0 | Trace JSON |
| Claude Fable 5.1 | Completed | 6 | 274.1 | 85,053 | Trace JSON |
| Gemini 3.8 Flash | Completed | 6.0 | 223.7 | 121,795.0 | Trace JSON |
| Qwen 3.8 Max | Completed | 7 | 642.2 | 189,492 | Trace JSON |
| Qwen 3.5 397B-A17B | Not completed | 8.0 | 359.5 | 211,384.0 | Trace JSON |
| Qwen 3.5 27B | Not completed | 8.0 | 393.9 | 196,302.0 | Trace JSON |
| Claude Opus 5.5 | Not completed | 8.0 | 286.3 | 29,457.0 | Trace JSON |
| DeepSeek 4.1 Flash-1017 | Completed | 8 | 318.2 | 203,491 | Trace JSON |
| Gemini 4 Argon | Completed | 4 | 546.6 | 53,387 | Trace JSON |
VirtualHome · Planning
Watch TV
Task: Watch TV: I like to set on my sofa.Turn on the tv. And watch Perry Mason. Monday's through Friday's.
7 of 10 recorded models completed this task. Compare actions, feedback and resource use.
Select a decision round to inspect the executed actions, returned feedback and native completion check.
| Model | Outcome | Model calls | Time (s) | Tokens | Evidence |
|---|---|---|---|---|---|
| GPT-6 Astra | Completed | 4.0 | 361.8 | 110,665.0 | Trace JSON |
| GPT-6 Sol | Completed | 7 | 488.5 | 305,218 | Trace JSON |
| Claude Fable 5.1 | Completed | 4.0 | 186.5 | 49,300.0 | Trace JSON |
| Gemini 3.8 Flash | Completed | 8 | 249.7 | 230,095 | Trace JSON |
| Qwen 3.8 Max | Completed | 6.0 | 422.1 | 142,038.0 | Trace JSON |
| Qwen 3.5 397B-A17B | Not completed | 8.0 | 271.7 | 208,925.0 | Trace JSON |
| Qwen 3.5 27B | Not completed | 8.0 | 439.1 | 222,022.0 | Trace JSON |
| Claude Opus 5.5 | Completed | 7.0 | 211.0 | 5,198.0 | Trace JSON |
| DeepSeek 4.1 Flash-1017 | Not completed | 8 | 282.4 | 203,057 | Trace JSON |
| Gemini 4 Argon | Completed | 8 | 1,188.3 | 232,822 | Trace JSON |
VirtualHome · Planning
Turn On the Bedroom Light
Task: Turn on light: I walk towards to the bedroom, and I open the door of the bedroom, and I switch on the lights inside the bedroom, the switch is beside the door.
8 of 10 recorded models completed this task. Compare actions, feedback and resource use.
Select a decision round to inspect the executed actions, returned feedback and native completion check.
| Model | Outcome | Model calls | Time (s) | Tokens | Evidence |
|---|---|---|---|---|---|
| GPT-6 Astra | Completed | 3.0 | 242.1 | 72,168.0 | Trace JSON |
| GPT-6 Sol | Completed | 4.0 | 335.5 | 115,240.0 | Trace JSON |
| Claude Fable 5.1 | Completed | 3.0 | 212.9 | 27,722.0 | Trace JSON |
| Gemini 3.8 Flash | Completed | 4.0 | 211.6 | 56,099.0 | Trace JSON |
| Qwen 3.8 Max | Completed | 5.0 | 550.8 | 107,838.0 | Trace JSON |
| Qwen 3.5 397B-A17B | Not completed | 8.0 | 410.0 | 220,988.0 | Trace JSON |
| Qwen 3.5 27B | Not completed | 8.0 | 495.0 | 215,219.0 | Trace JSON |
| Claude Opus 5.5 | Completed | 8 | 522.4 | 185,872 | Trace JSON |
| DeepSeek 4.1 Flash-1017 | Completed | 7 | 288.5 | 170,216 | Trace JSON |
| Gemini 4 Argon | Completed | 6 | 1,047.5 | 127,575 | Trace JSON |
VLABench · Manipulation
Put the Rose in the Vase
Task: Insert the rose into the vase_seen.
2 of 10 recorded models completed this task. Compare actions, feedback and resource use.
Select a decision round to inspect the executed actions, returned feedback and native completion check.
| Model | Outcome | Model calls | Time (s) | Tokens | Evidence |
|---|---|---|---|---|---|
| GPT-6 Astra | Not completed | 8.0 | 197.2 | 142,387.0 | Trace JSON |
| GPT-6 Sol | Not completed | 7.0 | 1,374.2 | 264,224.0 | Trace JSON |
| Claude Fable 5.1 | Completed | 12 | 332.8 | 268,946 | Trace JSON |
| Gemini 3.8 Flash | Not completed | 16.0 | 416.1 | 508,090.0 | Trace JSON |
| Qwen 3.8 Max | Not completed | 10.0 | 1,298.3 | 547,359.0 | Trace JSON |
| Qwen 3.5 397B-A17B | Not completed | 16.0 | 242.7 | 219,652.0 | Trace JSON |
| Qwen 3.5 27B | Not completed | 15.0 | 371.3 | 569,158.0 | Trace JSON |
| Claude Opus 5.5 | Completed | 12.0 | 463.7 | 80,483.0 | Trace JSON |
| DeepSeek 4.1 Flash-1017 | Not completed | 16 | 171.4 | 312,092 | Trace JSON |
| Gemini 4 Argon | Not completed | 14 | 855.1 | 599,620 | Trace JSON |
VLABench · Manipulation
Add Salad Dressing
Task: Add salad_dressing to the dish
5 of 10 recorded models completed this task. Compare actions, feedback and resource use.
Select a decision round to inspect the executed actions, returned feedback and native completion check.
| Model | Outcome | Model calls | Time (s) | Tokens | Evidence |
|---|---|---|---|---|---|
| GPT-6 Astra | Completed | 7 | 470.9 | 199,240 | Trace JSON |
| GPT-6 Sol | Completed | 7 | 319.4 | 164,040 | Trace JSON |
| Claude Fable 5.1 | Completed | 4.0 | 120.7 | 78,032.0 | Trace JSON |
| Gemini 3.8 Flash | Completed | 9 | 138.4 | 328,477 | Trace JSON |
| Qwen 3.8 Max | Not completed | 14.0 | 957.2 | 593,283.0 | Trace JSON |
| Qwen 3.5 397B-A17B | Not completed | 14.0 | 155.3 | 557,034.0 | Trace JSON |
| Qwen 3.5 27B | Not completed | 13.0 | 360.9 | 598,823.0 | Trace JSON |
| Claude Opus 5.5 | Completed | 8 | 662.2 | 119,747 | Trace JSON |
| DeepSeek 4.1 Flash-1017 | Not completed | 16 | 186.8 | 510,934 | Trace JSON |
| Gemini 4 Argon | Not completed | 13 | 288.8 | 601,737 | Trace JSON |
Evidence and Timing
Source hashes tie the recorded actions and feedback to each result. Video playback and episode wall time use separate clocks. Native calls, model responses and environment steps are distinct counters.