← Interactive Execution Explorer

Inside the Execution

46 tasks · 460 recorded episodes · 10 models. Each comparison uses the same task; settings and budgets remain attached to its execution record.

Guide updated · Research reference: paper v3, 23 October 2026. This update adds evidence-reading guidance; recorded outcomes are unchanged.

How to Read Robot-Agent Execution Evidence

Read each trace as a sequence of model decisions, executed operations, returned feedback and native task checks. An operation running without an exception establishes executability; environment progress and terminal completion require their own evidence. The matched cases below retain each agent’s settings and resource limits. They do not impose identical budgets across every recorded batch. Paper v3 · interfaces and feedback.

Keep the populations separate. This explorer contains 46 interactive tasks and 460 retained episodes across ten models. It is different from the 46-case fixed-input review experiment in Geometry, Spatial Reasoning and Affordance. The aggregate CALVIN recovery analysis uses 100 episodes, and the Planning coverage analysis uses 157 task identities. Select the population supporting the claim rather than transferring a denominator between these analyses.

For robot harness and agentic robotics research, compare the state change needed for the next subgoal, the native completion check and the budget that ended the attempt. Model calls, code rounds, primitive calls, environment steps, tokens and elapsed time measure different quantities. Video playback time is a separate clock. Paper §4.4 · feedback and resource limits.

Within-episode repair is useful evidence about execution, but does not by itself show recursive self-improvement, cross-task memory transfer or a demonstration-conditioned in-context learning gain. Robot RSI evaluation boundaries · Robot ICL relevance.

DiscoveryWorld · Planning

Recovery Means Making Progress.

Task: Excavate three sites, date the artifacts, and place the red flag at the oldest site.

Argon and Opus 5.5 recover from early code errors and complete the task. DeepSeek-1017 stops at a partial score of 0.5.

Argon: error in round 2 → score 0.3 in round 3 → 1.0 in round 8. Opus 5.5 finishes in round 7; DeepSeek-1017 remains at 0.5.

ModelOutcomeModel callsTime (s)TokensEvidence
GPT-6 AstraCompleted6319.0222,779Trace JSON
GPT-6 SolCompleted7.0391.5256,374.0Trace JSON
Claude Fable 5.1Completed5.089.472,239.0Trace JSON
Gemini 3.8 FlashCompleted8.053.6210,096.0Trace JSON
Qwen 3.8 MaxNot completed8.0581.0252,735.0Trace JSON
Qwen 3.5 397B-A17BNot completed8.077.9217,096.0Trace JSON
Qwen 3.5 27BNot completed8.0151.3224,454.0Trace JSON
Claude Opus 5.5Completed7167.974,516Trace JSON
DeepSeek 4.1 Flash-1017Not completed851.3200,418Trace JSON
Gemini 4 ArgonCompleted8134.3212,524Trace JSON

Watch and inspect this comparison →

CALVIN · Manipulation · Supplementary RGB-expanded comparison

More Motion. Less Progress.

Scope: This supplementary comparison uses RGB-expanded executions. These case records are not the adopted 100-episode CALVIN population used for the aggregate recovery counts. Consult the main-evaluation finding and settings before comparing results.

Task: Lift the blue block from the sliding cabinet, put it in the drawer, retrieve it, stack it, and switch off the light.

Argon completes five goals with 475 action submissions; Opus 5.5 does so with 338. Gemini 3.8 Flash completes none with 501.

Argon: 5/5 goals · 475 actions · 18 model calls. Opus 5.5: 5/5 · 338 · 18. Gemini 3.8 Flash: 0/5 · 501 · 36.

ModelOutcomeModel callsTime (s)TokensEvidence
GPT-6 Astra0/5 goals20.0306.5320,323.0Trace JSON
GPT-6 Sol5/5 goals401,155.81,223,829Trace JSON
Claude Fable 5.10/5 goals39.01,335.31,253,304.0Trace JSON
Gemini 3.8 Flash0/5 goals36796.51,248,694Trace JSON
Qwen 3.8 Max5/5 goals211,851.2510,185Trace JSON
Qwen 3.5 397B-A17B0/5 goals24279.21,267,295Trace JSON
Qwen 3.5 27B0/5 goals31.0454.71,237,633.0Trace JSON
Claude Opus 5.55/5 goals18229.4130,168Trace JSON
DeepSeek 4.1 Flash-10175/5 goals37462.5854,017Trace JSON
Gemini 4 Argon5/5 goals18351.9363,694Trace JSON

Watch and inspect this comparison →

RoboCasa · Manipulation

A Valid Action Can Still Miss the Goal.

Task: Open the left drawer.

Argon opens the drawer in 13 rounds, recovering after a round-10 error. Sol reaches 24 rounds without completing the task.

The native drawer check succeeds for Argon and Fable 5.1, after 13 and 20 rounds respectively; Sol reaches its round limit.

ModelOutcomeModel callsTime (s)TokensEvidence
GPT-6 AstraNot completed24.0470.2262,592.0Trace JSON
GPT-6 SolNot completed24735.5238,425Trace JSON
Claude Fable 5.1Completed20618.1302,756Trace JSON
Gemini 3.8 FlashNot completed24246.3398,788Trace JSON
Qwen 3.8 MaxNot completed243,223.2445,456Trace JSON
Qwen 3.5 397B-A17BNot completed24.0197.4230,971.0Trace JSON
Qwen 3.5 27BNot completed24215.5246,777Trace JSON
Claude Opus 5.5Not completed24.0511.8168,841.0Trace JSON
DeepSeek 4.1 Flash-1017Not completed24195.0250,342Trace JSON
Gemini 4 ArgonCompleted13348.0181,346Trace JSON

Watch and inspect this comparison →

ALFRED · Planning

The Whole Instruction Matters.

Task: Clean the soap bar and place it in the bathtub.

Argon reaches all three goal conditions in six rounds. DeepSeek-1017 ends with one of three after eight.

Argon moves from 1/3 goal conditions in round 5 to 3/3 in round 6. The instruction requires the cleaned soap bar in the bathtub.

ModelOutcomeModel callsTime (s)TokensEvidence
GPT-6 AstraCompleted5.0273.3143,937.0Trace JSON
GPT-6 SolCompleted4.0392.5108,571.0Trace JSON
Claude Fable 5.1Completed3.0217.931,636.0Trace JSON
Gemini 3.8 FlashCompleted8175.4164,905Trace JSON
Qwen 3.8 MaxCompleted5486.796,824Trace JSON
Qwen 3.5 397B-A17BNot completed8.0188.6134,860.0Trace JSON
Qwen 3.5 27BNot completed8.0160.1152,484.0Trace JSON
Claude Opus 5.5Completed8425.8134,617Trace JSON
DeepSeek 4.1 Flash-1017Not completed852.1125,845Trace JSON
Gemini 4 ArgonCompleted61,037.6102,027Trace JSON

Watch and inspect this comparison →

HumanCLAW · Planning

Fewer Decisions Can Take Longer.

Task: Find the chair and move until the body is touching the target, then stop.

Argon reaches the chair in 17 rounds; DeepSeek-1017 needs 65. Their elapsed times are much closer than their decision counts.

Argon: 17 rounds / 437 s. DeepSeek-1017: 65 / 406 s. Astra: 18 / 471 s. Read both decision count and elapsed time.

ModelOutcomeModel callsTime (s)TokensEvidence
GPT-6 AstraCompleted18470.7356,427Trace JSON
GPT-6 SolCompleted17668.6324,622Trace JSON
Claude Fable 5.1Not completed19.0555.6148,671.0Trace JSON
Gemini 3.8 FlashCompleted14.0161.7116,699.0Trace JSON
Qwen 3.8 MaxCompleted351,096.2359,080Trace JSON
Qwen 3.5 397B-A17BCompleted30270.3241,385Trace JSON
Qwen 3.5 27BNot completed57435.5824,751Trace JSON
Claude Opus 5.5Completed17.0331.0103,252.0Trace JSON
DeepSeek 4.1 Flash-1017Completed65405.7938,473Trace JSON
Gemini 4 ArgonCompleted17437.3157,504Trace JSON

Watch and inspect this comparison →

BEHAVIOR-1K · Manipulation

Time Is Not the Only Limit.

Task: Outfit a basic toolbox with the required tools.

No model completes this toolbox task. Argon stops at its token limit after 17 model responses, well before the time ceiling.

Argon records 603,393 input + output tokens against a 640,000-token ceiling. Its next round is blocked by the token guard; the native goal remains incomplete.

ModelOutcomeModel callsTime (s)TokensEvidence
GPT-6 AstraNot completed231,828.7616,784Trace JSON
GPT-6 SolNot completed211,762.0592,598Trace JSON
Claude Fable 5.1Not completed237,992.2629,408Trace JSON
Gemini 3.8 FlashNot completed18729.9583,475Trace JSON
Qwen 3.8 MaxNot completed23.05,759.3627,228.0Trace JSON
Qwen 3.5 397B-A17BNot completed241,090.9579,186Trace JSON
Qwen 3.5 27BNot completed18879.9612,036Trace JSON
Claude Opus 5.5Not completed241,908.3367,814Trace JSON
DeepSeek 4.1 Flash-1017Not completed24720.4520,441Trace JSON
Gemini 4 ArgonNot completed172,251.9603,393Trace JSON

Watch and inspect this comparison →

ManiSkill · Manipulation

Motion Is Not Successful Contact.

Task: Insert the charger into the socket.

Argon stops at its token limit with the charger 15.9 cm from the goal. Qwen-Max reaches 400 native steps without insertion.

Argon: 23 native steps · 15.9 cm final goal distance · token limit. Qwen Max: 400 native steps · no successful insertion.

ModelOutcomeModel callsTime (s)TokensEvidence
GPT-6 AstraNot completed25.0744.9553,451.0Trace JSON
GPT-6 SolNot completed28.0761.6606,403.0Trace JSON
Claude Fable 5.1Not completed28.01,521.0632,855.0Trace JSON
Gemini 3.8 FlashNot completed23.0391.6604,048.0Trace JSON
Qwen 3.8 MaxNot completed131,532.5282,973Trace JSON
Qwen 3.5 397B-A17BNot completed19196.8419,050Trace JSON
Qwen 3.5 27BNot completed21.0164.8623,756.0Trace JSON
Claude Opus 5.5Not completed15.0270.0117,669.0Trace JSON
DeepSeek 4.1 Flash-1017Not completed32479.8628,164Trace JSON
Gemini 4 ArgonNot completed20547.5614,008Trace JSON

Watch and inspect this comparison →

ALFRED · Planning

Put the Salt Shaker in a Drawer

Task: place a salt shaker in one of the pullout drawers

4 of 10 recorded models completed this task. Compare actions, feedback and resource use.

Select a decision round to inspect the executed actions, returned feedback and native completion check.

ModelOutcomeModel callsTime (s)TokensEvidence
GPT-6 AstraCompleted8.0116.2111,436.0Trace JSON
GPT-6 SolCompleted8254.1127,230Trace JSON
Claude Fable 5.1Completed8222.5102,397Trace JSON
Gemini 3.8 FlashNot completed8.0147.5114,471.0Trace JSON
Qwen 3.8 MaxAPI error0.0251.2—Trace JSON
Qwen 3.5 397B-A17BNot completed8.0148.7103,248.0Trace JSON
Qwen 3.5 27BNot completed8.0100.096,578.0Trace JSON
Claude Opus 5.5Not completed8.0135.8119,594.0Trace JSON
DeepSeek 4.1 Flash-1017Not completed8.069.098,956.0Trace JSON
Gemini 4 ArgonCompleted6906.199,832Trace JSON

Watch and inspect this comparison →

ALFRED · Planning

Inspect the Remote Under a Lamp

Task: Hold a remote and turn on the lamp.

5 of 10 recorded models completed this task. Compare actions, feedback and resource use.

Select a decision round to inspect the executed actions, returned feedback and native completion check.

ModelOutcomeModel callsTime (s)TokensEvidence
GPT-6 AstraCompleted7.0423.3211,408.0Trace JSON
GPT-6 SolNot completed8.0178.9106,380.0Trace JSON
Claude Fable 5.1Completed5.088.644,974.0Trace JSON
Gemini 3.8 FlashNot completed8.0145.5141,071.0Trace JSON
Qwen 3.8 MaxCompleted6.0259.284,212.0Trace JSON
Qwen 3.5 397B-A17BCompleted563.848,912Trace JSON
Qwen 3.5 27BNot completed8102.1119,113Trace JSON
Claude Opus 5.5Not completed8.0321.582,302.0Trace JSON
DeepSeek 4.1 Flash-1017Not completed8.051.078,301.0Trace JSON
Gemini 4 ArgonCompleted6885.3105,379Trace JSON

Watch and inspect this comparison →

ALFRED · Planning

Put Two Spray Bottles on the Toilet

Task: Put the green spray bottles on the toilet

1 of 10 recorded models completed this task. Compare actions, feedback and resource use.

Select a decision round to inspect the executed actions, returned feedback and native completion check.

ModelOutcomeModel callsTime (s)TokensEvidence
GPT-6 AstraCompleted8153.2138,943Trace JSON
GPT-6 SolNot completed8.0212.9111,088.0Trace JSON
Claude Fable 5.1Not completed8.0286.784,235.0Trace JSON
Gemini 3.8 FlashNot completed8.0189.6109,796.0Trace JSON
Qwen 3.8 MaxNot completed8.0704.6158,791.0Trace JSON
Qwen 3.5 397B-A17BNot completed8.0156.8106,205.0Trace JSON
Qwen 3.5 27BNot completed8.0133.095,740.0Trace JSON
Claude Opus 5.5Not completed8.0123.791,163.0Trace JSON
DeepSeek 4.1 Flash-1017Not completed857.485,628Trace JSON
Gemini 4 ArgonNot completed8242.8213,502Trace JSON

Watch and inspect this comparison →

ALFWorld · Planning

Clean Soap and Put It on the Counter

Task: Put a cleaned bar of soap on a counter.

5 of 10 recorded models completed this task. Compare actions, feedback and resource use.

Select a decision round to inspect the executed actions, returned feedback and native completion check.

ModelOutcomeModel callsTime (s)TokensEvidence
GPT-6 AstraCompleted5.0358.6145,743.0Trace JSON
GPT-6 SolCompleted6433.8189,608Trace JSON
Claude Fable 5.1Completed7197.5169,006Trace JSON
Gemini 3.8 FlashNot completed8.0189.0175,939.0Trace JSON
Qwen 3.8 MaxNot completed8.0201.2122,093.0Trace JSON
Qwen 3.5 397B-A17BNot completed8.0138.3133,617.0Trace JSON
Qwen 3.5 27BNot completed8.0155.6155,526.0Trace JSON
Claude Opus 5.5Completed6.077.663,724.0Trace JSON
DeepSeek 4.1 Flash-1017Not completed859.1136,094Trace JSON
Gemini 4 ArgonCompleted5870.688,367Trace JSON

Watch and inspect this comparison →

ALFWorld · Planning

Inspect a Metal Box Under a Lamp

Task: Turn on a lamp with a metal box.

3 of 10 recorded models completed this task. Compare actions, feedback and resource use.

Select a decision round to inspect the executed actions, returned feedback and native completion check.

ModelOutcomeModel callsTime (s)TokensEvidence
GPT-6 AstraCompleted6.0401.7177,316.0Trace JSON
GPT-6 SolNot completed8.0173.4122,157.0Trace JSON
Claude Fable 5.1Not completed8.0220.7109,716.0Trace JSON
Gemini 3.8 FlashNot completed8.0173.0141,437.0Trace JSON
Qwen 3.8 MaxAPI error0.0283.0—Trace JSON
Qwen 3.5 397B-A17BNot completed8.0134.0115,959.0Trace JSON
Qwen 3.5 27BNot completed8.0110.091,565.0Trace JSON
Claude Opus 5.5Completed880.991,736Trace JSON
DeepSeek 4.1 Flash-1017Not completed862.1169,067Trace JSON
Gemini 4 ArgonCompleted6193.4120,671Trace JSON

Watch and inspect this comparison →

CALVIN · Manipulation

Sequence 0270 · Five Goals

Task: Rotate the red block right, close the drawer, lift the red block, stack it, then lift the blue block from the cabinet.

1 of 10 recorded models completed this task. Compare actions, feedback and resource use.

Select a decision round to inspect the executed actions, returned feedback and native completion check.

ModelOutcomeModel callsTime (s)TokensEvidence
GPT-6 Astra4/5 goals20.0447.2339,664.0Trace JSON
GPT-6 Sol0/5 goals29.01,034.81,206,468.0Trace JSON
Claude Fable 5.15/5 goals24566.0538,329Trace JSON
Gemini 3.8 Flash2/5 goals36.0592.81,224,750.0Trace JSON
Qwen 3.8 Max3/5 goals24.03,636.61,006,296.0Trace JSON
Qwen 3.5 397B-A17B0/5 goals26.0292.51,205,738.0Trace JSON
Qwen 3.5 27B0/5 goals30.0286.91,238,933.0Trace JSON
Claude Opus 5.54/5 goals40.01,450.8125,604.0Trace JSON
DeepSeek 4.1 Flash-10170/5 goals40595.41,038,865Trace JSON
Gemini 4 Argon4/5 goals331,348.41,273,471Trace JSON

Watch and inspect this comparison →

CALVIN · Manipulation

Sequence 0557 · Five Goals

Task: Complete five subgoals in sequence. Recorded subgoals: press the button to turn off the led light; lift the blue block from the sliding cabinet.

3 of 10 recorded models completed this task. Compare actions, feedback and resource use.

Select a decision round to inspect the executed actions, returned feedback and native completion check.

ModelOutcomeModel callsTime (s)TokensEvidence
GPT-6 Astra5/5 goals18373.3283,136Trace JSON
GPT-6 Sol0/5 goals30.01,231.41,256,172.0Trace JSON
Claude Fable 5.15/5 goals16.0225.9271,110.0Trace JSON
Gemini 3.8 FlashAPI error7.0816.9100,076.0Trace JSON
Qwen 3.8 Max0/5 goals36.03,803.31,219,421.0Trace JSON
Qwen 3.5 397B-A17B0/5 goals26.0278.41,202,842.0Trace JSON
Qwen 3.5 27B0/5 goals35.0396.41,262,798.0Trace JSON
Claude Opus 5.51/5 goals40.0978.6170,689.0Trace JSON
DeepSeek 4.1 Flash-10170/5 goals40205.5832,091Trace JSON
Gemini 4 Argon5/5 goals125,603.9211,127Trace JSON

Watch and inspect this comparison →

CaPX–BEHAVIOR-1K · Manipulation

Put Shoes on the Rack

Task: putting shoes on rack

0 of 10 recorded models completed this task. Compare actions, feedback and resource use.

Select a decision round to inspect the executed actions, returned feedback and native completion check.

ModelOutcomeModel callsTime (s)TokensEvidence
GPT-6 AstraNot completed24.0742.5521,217.0Trace JSON
GPT-6 SolNot completed24.01,348.5519,707.0Trace JSON
Claude Fable 5.1Not completed21.01,591.2618,279.0Trace JSON
Gemini 3.8 FlashNot completed8.0175.4130,877.0Trace JSON
Qwen 3.8 MaxNot completed241,613.8572,778Trace JSON
Qwen 3.5 397B-A17BAPI error8.0372.899,398.0Trace JSON
Qwen 3.5 27BAPI error0.095.8—Trace JSON
Claude Opus 5.5Not completed20.01,249.6599,155.0Trace JSON
DeepSeek 4.1 Flash-1017Not completed242,146.2523,219Trace JSON
Gemini 4 ArgonNot completed19761.9602,401Trace JSON

Watch and inspect this comparison →

CaPX–LIBERO-PRO · Manipulation

Turn On the Stove

Task: turn on the stove

6 of 10 recorded models completed this task. Compare actions, feedback and resource use.

Select a decision round to inspect the executed actions, returned feedback and native completion check.

ModelOutcomeModel callsTime (s)TokensEvidence
GPT-6 AstraCompleted15625.1214,593Trace JSON
GPT-6 SolCompleted8303.871,096Trace JSON
Claude Fable 5.1Completed20.01,245.8452,686.0Trace JSON
Gemini 3.8 FlashCompleted16.0311.4358,602.0Trace JSON
Qwen 3.8 MaxNot completed19.02,220.1594,795.0Trace JSON
Qwen 3.5 397B-A17BNot completed24.0427.6303,069.0Trace JSON
Qwen 3.5 27BNot completed24.0292.3408,828.0Trace JSON
Claude Opus 5.5Completed13.0294.695,030.0Trace JSON
DeepSeek 4.1 Flash-1017Not completed24796.1629,486Trace JSON
Gemini 4 ArgonCompleted9296.5132,324Trace JSON

Watch and inspect this comparison →

CaPX–LIBERO-PRO · Manipulation

Put Tomato Sauce in the Basket

Task: pick up the tomato sauce and place it in the basket

5 of 10 recorded models completed this task. Compare actions, feedback and resource use.

Select a decision round to inspect the executed actions, returned feedback and native completion check.

ModelOutcomeModel callsTime (s)TokensEvidence
GPT-6 AstraNot completed19.0835.4453,857.0Trace JSON
GPT-6 SolNot completed24.01,260.8400,030.0Trace JSON
Claude Fable 5.1Completed161,553.5337,613Trace JSON
Gemini 3.8 FlashCompleted15429.0321,238Trace JSON
Qwen 3.8 MaxNot completed22.02,069.6628,440.0Trace JSON
Qwen 3.5 397B-A17BNot completed24934.6466,647Trace JSON
Qwen 3.5 27BNot completed24.0547.5414,652.0Trace JSON
Claude Opus 5.5Completed202,376.8159,535Trace JSON
DeepSeek 4.1 Flash-1017Completed11396.0110,098Trace JSON
Gemini 4 ArgonCompleted10576.1220,087Trace JSON

Watch and inspect this comparison →

CaPX–Robosuite · Manipulation

Wipe the Spill

Task: spill wipe

9 of 10 recorded models completed this task. Compare actions, feedback and resource use.

Select a decision round to inspect the executed actions, returned feedback and native completion check.

ModelOutcomeModel callsTime (s)TokensEvidence
GPT-6 AstraCompleted2284.118,087Trace JSON
GPT-6 SolCompleted5183.942,444Trace JSON
Claude Fable 5.1Completed4149.742,773Trace JSON
Gemini 3.8 FlashCompleted366.036,374Trace JSON
Qwen 3.8 MaxCompleted2445.931,852Trace JSON
Qwen 3.5 397B-A17BCompleted5123.344,133Trace JSON
Qwen 3.5 27BNot completed19.0766.3622,760.0Trace JSON
Claude Opus 5.5Completed5148.968,221Trace JSON
DeepSeek 4.1 Flash-1017Completed13966.4183,940Trace JSON
Gemini 4 ArgonCompleted5158.482,720Trace JSON

Watch and inspect this comparison →

CLIPort · Manipulation

Pack Matching Objects

Task: packing boxes pairs unseen colors

3 of 10 recorded models completed this task. Compare actions, feedback and resource use.

Select a decision round to inspect the executed actions, returned feedback and native completion check.

ModelOutcomeModel callsTime (s)TokensEvidence
GPT-6 AstraCompleted3.0131.558,347.0Trace JSON
GPT-6 SolNot completed18.0898.3295,041.0Trace JSON
Claude Fable 5.1Completed10.0433.2142,157.0Trace JSON
Gemini 3.8 FlashNot completed16.0225.6294,947.0Trace JSON
Qwen 3.8 MaxNot completed10.02,474.1250,091.0Trace JSON
Qwen 3.5 397B-A17BNot completed5.094.156,294.0Trace JSON
Qwen 3.5 27BNot completed3.040.331,429.0Trace JSON
Claude Opus 5.5Completed11654.3160,208Trace JSON
DeepSeek 4.1 Flash-1017Not completed24556.9444,434Trace JSON
Gemini 4 ArgonNot completed9388.5141,113Trace JSON

Watch and inspect this comparison →

CLIPort · Manipulation

Towers of Hanoi

Task: Towers of hanoi seq unseen colors

5 of 10 recorded models completed this task. Compare actions, feedback and resource use.

Select a decision round to inspect the executed actions, returned feedback and native completion check.

ModelOutcomeModel callsTime (s)TokensEvidence
GPT-6 AstraCompleted7362.1163,427Trace JSON
GPT-6 SolCompleted18.0518.6273,178.0Trace JSON
Claude Fable 5.1Completed11427.3149,420Trace JSON
Gemini 3.8 FlashNot completed24.0356.5618,078.0Trace JSON
Qwen 3.8 MaxNot completed19.02,070.6891,589.0Trace JSON
Qwen 3.5 397B-A17BAPI error7.0102.3100,733.0Trace JSON
Qwen 3.5 27BNot completed21.0515.2873,908.0Trace JSON
Claude Opus 5.5Completed8.0474.481,639.0Trace JSON
DeepSeek 4.1 Flash-1017Not completed24.0451.3498,585.0Trace JSON
Gemini 4 ArgonCompleted10924.7173,803Trace JSON

Watch and inspect this comparison →

DiscoveryWorld · Planning

Remove Rust from a Key

Task: Identify the rust-removing chemical, clean the key, then unlock the shed and leave.

4 of 10 recorded models completed this task. Compare actions, feedback and resource use.

Select a decision round to inspect the executed actions, returned feedback and native completion check.

ModelOutcomeModel callsTime (s)TokensEvidence
GPT-6 AstraCompleted4.0226.4120,440.0Trace JSON
GPT-6 SolCompleted8250.3262,849Trace JSON
Claude Fable 5.1Not completed8.0215.9108,029.0Trace JSON
Gemini 3.8 FlashCompleted859.7236,109Trace JSON
Qwen 3.8 MaxNot completed8.0357.9232,372.0Trace JSON
Qwen 3.5 397B-A17BNot completed8.0127.3245,276.0Trace JSON
Qwen 3.5 27BNot completed8.0187.2247,493.0Trace JSON
Claude Opus 5.5Not completed8.0281.180,615.0Trace JSON
DeepSeek 4.1 Flash-1017Not completed8.066.9220,835.0Trace JSON
Gemini 4 ArgonCompleted799.8189,635Trace JSON

Watch and inspect this comparison →

DiscoveryWorld · Planning

Give Yugo the Spectrometer

Task: Your task is to give Yugo the `spectrometer`.

9 of 10 recorded models completed this task. Compare actions, feedback and resource use.

Select a decision round to inspect the executed actions, returned feedback and native completion check.

ModelOutcomeModel callsTime (s)TokensEvidence
GPT-6 AstraCompleted2.084.144,101.0Trace JSON
GPT-6 SolCompleted5.0109.3125,915.0Trace JSON
Claude Fable 5.1Completed4.090.15,691.0Trace JSON
Gemini 3.8 FlashCompleted7.036.1161,996.0Trace JSON
Qwen 3.8 MaxCompleted4.0291.870,657.0Trace JSON
Qwen 3.5 397B-A17BNot completed8.0124.0227,965.0Trace JSON
Qwen 3.5 27BCompleted7110.0162,447Trace JSON
Claude Opus 5.5Completed5.090.436,658.0Trace JSON
DeepSeek 4.1 Flash-1017Completed733.7155,543Trace JSON
Gemini 4 ArgonCompleted341.133,857Trace JSON

Watch and inspect this comparison →

HumanCLAW · Planning

Sit on the Couch

Task: Find the couch and move until your body is touching the target, with zero distance to the target object. Finally, sit on the couch.

0 of 10 recorded models completed this task. Compare actions, feedback and resource use.

Select a decision round to inspect the executed actions, returned feedback and native completion check.

ModelOutcomeModel callsTime (s)TokensEvidence
GPT-6 AstraNot completed44.01,275.31,077,381.0Trace JSON
GPT-6 SolNot completed25.01,100.3537,099.0Trace JSON
Claude Fable 5.1Not completed24.0649.6237,596.0Trace JSON
Gemini 3.8 FlashNot completed18.0171.1171,054.0Trace JSON
Qwen 3.8 MaxNot completed775,526.11,969,448Trace JSON
Qwen 3.5 397B-A17BNot completed191,435.4105,975Trace JSON
Qwen 3.5 27BNot completed82675.81,793,349Trace JSON
Claude Opus 5.5Not completed51.0691.9965,711.0Trace JSON
DeepSeek 4.1 Flash-1017Not completed70496.51,106,965Trace JSON
Gemini 4 ArgonNot completed464,450.5666,909Trace JSON

Watch and inspect this comparison →

ManiSkill · Manipulation

Place the Sphere

Task: PlaceSphere-v1

3 of 10 recorded models completed this task. Compare actions, feedback and resource use.

Select a decision round to inspect the executed actions, returned feedback and native completion check.

ModelOutcomeModel callsTime (s)TokensEvidence
GPT-6 AstraNot completed13.0439.5171,642.0Trace JSON
GPT-6 SolNot completed26.0815.1602,149.0Trace JSON
Claude Fable 5.1Completed14.0248.8214,638.0Trace JSON
Gemini 3.8 FlashNot completed21.0944.9429,431.0Trace JSON
Qwen 3.8 MaxNot completed16.02,121.0393,523.0Trace JSON
Qwen 3.5 397B-A17BNot completed23.0234.9622,905.0Trace JSON
Qwen 3.5 27BNot completed15.0184.0430,328.0Trace JSON
Claude Opus 5.5Completed13477.6143,296Trace JSON
DeepSeek 4.1 Flash-1017Not completed13127.0176,822Trace JSON
Gemini 4 ArgonCompleted10253.7168,875Trace JSON

Watch and inspect this comparison →

RLBench · Manipulation

Change the TV Channel

Task: Change channel

3 of 10 recorded models completed this task. Compare actions, feedback and resource use.

Select a decision round to inspect the executed actions, returned feedback and native completion check.

ModelOutcomeModel callsTime (s)TokensEvidence
GPT-6 AstraCompleted6304.6164,557Trace JSON
GPT-6 SolNot completed8.0932.0247,414.0Trace JSON
Claude Fable 5.1Completed11266.3204,366Trace JSON
Gemini 3.8 FlashNot completed16.0475.9609,140.0Trace JSON
Qwen 3.8 MaxNot completed16.02,911.4414,397.0Trace JSON
Qwen 3.5 397B-A17BNot completed11.0166.3627,512.0Trace JSON
Qwen 3.5 27BNot completed12.0183.9562,456.0Trace JSON
Claude Opus 5.5Not completed7.0903.8116,237.0Trace JSON
DeepSeek 4.1 Flash-1017Not completed16.0514.9289,217.0Trace JSON
Gemini 4 ArgonCompleted13673.4381,782Trace JSON

Watch and inspect this comparison →

RLBench · Manipulation

Close the Laptop

Task: Close Laptop Lid

6 of 10 recorded models completed this task. Compare actions, feedback and resource use.

Select a decision round to inspect the executed actions, returned feedback and native completion check.

ModelOutcomeModel callsTime (s)TokensEvidence
GPT-6 AstraCompleted5.0246.5131,455.0Trace JSON
GPT-6 SolCompleted8.0282.2133,191.0Trace JSON
Claude Fable 5.1Completed10.0411.1151,150.0Trace JSON
Gemini 3.8 FlashNot completed16.0443.7465,552.0Trace JSON
Qwen 3.8 MaxCompleted133,076.4475,357Trace JSON
Qwen 3.5 397B-A17BNot completed14.0183.9569,707.0Trace JSON
Qwen 3.5 27BNot completed12.0251.1606,978.0Trace JSON
Claude Opus 5.5Completed8.0233.566,223.0Trace JSON
DeepSeek 4.1 Flash-1017Not completed16244.4242,844Trace JSON
Gemini 4 ArgonCompleted7211.8131,204Trace JSON

Watch and inspect this comparison →

RLBench · Manipulation

Open the Box

Task: Open Box

1 of 10 recorded models completed this task. Compare actions, feedback and resource use.

Select a decision round to inspect the executed actions, returned feedback and native completion check.

ModelOutcomeModel callsTime (s)TokensEvidence
GPT-6 AstraNot completed8.0277.3213,059.0Trace JSON
GPT-6 SolNot completed8.0736.4150,982.0Trace JSON
Claude Fable 5.1Not completed9.01,050.6119,694.0Trace JSON
Gemini 3.8 FlashNot completed13.0203.6561,286.0Trace JSON
Qwen 3.8 MaxNot completed16.02,386.8412,856.0Trace JSON
Qwen 3.5 397B-A17BNot completed15198.6628,112Trace JSON
Qwen 3.5 27BNot completed15.0183.8630,124.0Trace JSON
Claude Opus 5.5Not completed16.01,058.5148,922.0Trace JSON
DeepSeek 4.1 Flash-1017Not completed16361.7230,196Trace JSON
Gemini 4 ArgonCompleted121,503.3272,130Trace JSON

Watch and inspect this comparison →

RLBench · Manipulation

Take the Cup Out of the Cabinet

Task: Take Cup Out From Cabinet

1 of 10 recorded models completed this task. Compare actions, feedback and resource use.

Select a decision round to inspect the executed actions, returned feedback and native completion check.

ModelOutcomeModel callsTime (s)TokensEvidence
GPT-6 AstraNot completed8.0505.8238,250.0Trace JSON
GPT-6 SolNot completed8.0331.2209,061.0Trace JSON
Claude Fable 5.1Not completed16.01,109.0341,636.0Trace JSON
Gemini 3.8 FlashNot completed8.0883.5289,023.0Trace JSON
Qwen 3.8 MaxNot completed16.02,116.7474,645.0Trace JSON
Qwen 3.5 397B-A17BNot completed13.0208.9569,807.0Trace JSON
Qwen 3.5 27BNot completed11401.4570,376Trace JSON
Claude Opus 5.5Completed14878.5150,423Trace JSON
DeepSeek 4.1 Flash-1017Not completed16116.3346,600Trace JSON
Gemini 4 ArgonNot completed16421.5535,234Trace JSON

Watch and inspect this comparison →

RoboCasa · Manipulation

Turn On the Faucet

Task: Turn on the sink faucet.

0 of 10 recorded models completed this task. Compare actions, feedback and resource use.

Select a decision round to inspect the executed actions, returned feedback and native completion check.

ModelOutcomeModel callsTime (s)TokensEvidence
GPT-6 AstraNot completed24.01,345.4312,192.0Trace JSON
GPT-6 SolNot completed24.01,147.6272,594.0Trace JSON
Claude Fable 5.1Not completed21.0484.7342,590.0Trace JSON
Gemini 3.8 FlashNot completed24.0314.0417,233.0Trace JSON
Qwen 3.8 MaxAPI error4.0185.841,096.0Trace JSON
Qwen 3.5 397B-A17BNot completed24.0218.3226,141.0Trace JSON
Qwen 3.5 27BNot completed24.0179.9258,320.0Trace JSON
Claude Opus 5.5Not completed22544.4225,219Trace JSON
DeepSeek 4.1 Flash-1017Not completed24309.1228,685Trace JSON
Gemini 4 ArgonNot completed241,105.4807,967Trace JSON

Watch and inspect this comparison →

RoboCasa365 · Manipulation

Put Bread and Cheese on the Board

Task: Place the bread and cheese on the cutting board.

0 of 10 recorded models completed this task. Compare actions, feedback and resource use.

Select a decision round to inspect the executed actions, returned feedback and native completion check.

ModelOutcomeModel callsTime (s)TokensEvidence
GPT-6 AstraNot completed24816.8397,237Trace JSON
GPT-6 SolNot completed24.01,165.1310,731.0Trace JSON
Claude Fable 5.1Not completed24.0729.3375,845.0Trace JSON
Gemini 3.8 FlashNot completed24.0311.4420,226.0Trace JSON
Qwen 3.8 MaxNot completed24.02,288.7518,702.0Trace JSON
Qwen 3.5 397B-A17BNot completed24.0389.8308,816.0Trace JSON
Qwen 3.5 27BNot completed24.0225.5281,405.0Trace JSON
Claude Opus 5.5Not completed23.01,179.6194,849.0Trace JSON
DeepSeek 4.1 Flash-1017Not completed24167.1262,074Trace JSON
Gemini 4 ArgonNot completed241,157.3514,019Trace JSON

Watch and inspect this comparison →

RoboCasa365 · Manipulation

Close the Left Drawer

Task: Close the left drawer.

2 of 10 recorded models completed this task. Compare actions, feedback and resource use.

Select a decision round to inspect the executed actions, returned feedback and native completion check.

ModelOutcomeModel callsTime (s)TokensEvidence
GPT-6 AstraNot completed24.0930.7332,271.0Trace JSON
GPT-6 SolNot completed24.0634.7268,102.0Trace JSON
Claude Fable 5.1Completed21.01,006.4428,400.0Trace JSON
Gemini 3.8 FlashNot completed24.0519.31,074,988.0Trace JSON
Qwen 3.8 MaxNot completed242,435.0615,543Trace JSON
Qwen 3.5 397B-A17BNot completed24.0240.0260,704.0Trace JSON
Qwen 3.5 27BNot completed24282.1289,607Trace JSON
Claude Opus 5.5Completed16929.6170,405Trace JSON
DeepSeek 4.1 Flash-1017Not completed24147.0245,500Trace JSON
Gemini 4 ArgonNot completed241,388.0956,340Trace JSON

Watch and inspect this comparison →

RoboCasa365 · Manipulation

Clear the Cutting Board

Task: Remove the broccoli and steak from the cutting board to prepare it for cleaning.

2 of 10 recorded models completed this task. Compare actions, feedback and resource use.

Select a decision round to inspect the executed actions, returned feedback and native completion check.

ModelOutcomeModel callsTime (s)TokensEvidence
GPT-6 AstraNot completed24.0534.2281,339.0Trace JSON
GPT-6 SolNot completed24616.2276,891Trace JSON
Claude Fable 5.1Completed11230.0152,068Trace JSON
Gemini 3.8 FlashNot completed24318.0450,096Trace JSON
Qwen 3.8 MaxAPI error8.0645.6102,995.0Trace JSON
Qwen 3.5 397B-A17BNot completed24691.4305,084Trace JSON
Qwen 3.5 27BNot completed24.0594.2295,305.0Trace JSON
Claude Opus 5.5Not completed24.0394.7215,193.0Trace JSON
DeepSeek 4.1 Flash-1017Not completed24151.9251,548Trace JSON
Gemini 4 ArgonCompleted11266.9143,069Trace JSON

Watch and inspect this comparison →

RoboTwin2 · Manipulation

Order Blocks by Size

Task: blocks ranking size

3 of 10 recorded models completed this task. Compare actions, feedback and resource use.

Select a decision round to inspect the executed actions, returned feedback and native completion check.

ModelOutcomeModel callsTime (s)TokensEvidence
GPT-6 AstraCompleted16.0513.3482,973.0Trace JSON
GPT-6 SolNot completed16.0521.5547,854.0Trace JSON
Claude Fable 5.1Completed13597.7329,910Trace JSON
Gemini 3.8 FlashNot completed15.0264.1589,519.0Trace JSON
Qwen 3.8 MaxNot completed16.02,652.3593,765.0Trace JSON
Qwen 3.5 397B-A17BNot completed16.0178.3532,082.0Trace JSON
Qwen 3.5 27BNot completed15.0190.8583,709.0Trace JSON
Claude Opus 5.5Completed9.0328.275,980.0Trace JSON
DeepSeek 4.1 Flash-1017Not completed16588.8363,460Trace JSON
Gemini 4 ArgonNot completed15845.8591,628Trace JSON

Watch and inspect this comparison →

RoboTwin2 · Manipulation

Place the Shoe

Task: place shoe

2 of 10 recorded models completed this task. Compare actions, feedback and resource use.

Select a decision round to inspect the executed actions, returned feedback and native completion check.

ModelOutcomeModel callsTime (s)TokensEvidence
GPT-6 AstraCompleted16.0376.3426,171.0Trace JSON
GPT-6 SolNot completed16.0432.3464,978.0Trace JSON
Claude Fable 5.1Not completed16.0724.1354,033.0Trace JSON
Gemini 3.8 FlashNot completed16.0189.3580,295.0Trace JSON
Qwen 3.8 MaxNot completed16.0489.6314,128.0Trace JSON
Qwen 3.5 397B-A17BNot completed14.0179.3583,658.0Trace JSON
Qwen 3.5 27BNot completed16.0192.4537,260.0Trace JSON
Claude Opus 5.5Completed141,035.976,024Trace JSON
DeepSeek 4.1 Flash-1017Not completed16350.8381,209Trace JSON
Gemini 4 ArgonNot completed15388.6598,306Trace JSON

Watch and inspect this comparison →

RoboTwin2 · Manipulation

Stack Three Blocks

Task: stack blocks three

1 of 10 recorded models completed this task. Compare actions, feedback and resource use.

Select a decision round to inspect the executed actions, returned feedback and native completion check.

ModelOutcomeModel callsTime (s)TokensEvidence
GPT-6 AstraCompleted10274.0213,426Trace JSON
GPT-6 SolNot completed16.01,619.4358,357.0Trace JSON
Claude Fable 5.1Not completed16.0658.3522,047.0Trace JSON
Gemini 3.8 FlashNot completed16.0349.5505,318.0Trace JSON
Qwen 3.8 MaxNot completed14.01,800.9628,675.0Trace JSON
Qwen 3.5 397B-A17BNot completed15.0212.6625,138.0Trace JSON
Qwen 3.5 27BNot completed16.0259.3572,365.0Trace JSON
Claude Opus 5.5Not completed16.0452.4100,221.0Trace JSON
DeepSeek 4.1 Flash-1017Not completed16665.5340,973Trace JSON
Gemini 4 ArgonNot completed14377.2584,720Trace JSON

Watch and inspect this comparison →

RoboWits · Manipulation

Push the Red Block to the Green Target

Task: Push the red block onto the green target area.

5 of 10 recorded models completed this task. Compare actions, feedback and resource use.

Select a decision round to inspect the executed actions, returned feedback and native completion check.

ModelOutcomeModel callsTime (s)TokensEvidence
GPT-6 AstraNot completed17.0676.8598,205.0Trace JSON
GPT-6 SolNot completed19.0920.2604,005.0Trace JSON
Claude Fable 5.1Completed6.0192.997,772.0Trace JSON
Gemini 3.8 FlashCompleted14200.1324,990Trace JSON
Qwen 3.8 MaxCompleted12952.0213,956Trace JSON
Qwen 3.5 397B-A17BNot completed11.0170.3555,628.0Trace JSON
Qwen 3.5 27BNot completed13.0696.3526,305.0Trace JSON
Claude Opus 5.5Completed8183.866,541Trace JSON
DeepSeek 4.1 Flash-1017Completed335.632,688Trace JSON
Gemini 4 ArgonNot completed14285.1552,778Trace JSON

Watch and inspect this comparison →

RoboWits · Manipulation

Task 10 · Screw, Dustpan & Container

Task: Task 10 · screw, dustpan and container (episode 000).

7 of 10 recorded models completed this task. Compare actions, feedback and resource use.

Select a decision round to inspect the executed actions, returned feedback and native completion check.

ModelOutcomeModel callsTime (s)TokensEvidence
GPT-6 AstraCompleted20698.3448,018Trace JSON
GPT-6 SolCompleted17684.9394,967Trace JSON
Claude Fable 5.1Completed15769.7444,273Trace JSON
Gemini 3.8 FlashNot completed18.0467.6605,873.0Trace JSON
Qwen 3.8 MaxCompleted11.0968.3239,134.0Trace JSON
Qwen 3.5 397B-A17BNot completed22578.7594,747Trace JSON
Qwen 3.5 27BCompleted5179.571,050Trace JSON
Claude Opus 5.5Not completed24.01,311.8170,467.0Trace JSON
DeepSeek 4.1 Flash-1017Completed766.082,092Trace JSON
Gemini 4 ArgonCompleted62,804.3103,168Trace JSON

Watch and inspect this comparison →

ScienceWorld · Planning

Find an Animal

Task: Your task is to find a(n) animal. First, focus on the thing. Then, move it to the green box in the workshop.

4 of 10 recorded models completed this task. Compare actions, feedback and resource use.

Select a decision round to inspect the executed actions, returned feedback and native completion check.

ModelOutcomeModel callsTime (s)TokensEvidence
GPT-6 AstraCompleted4.0119.581,630.0Trace JSON
GPT-6 SolNot completed8.0501.6224,834.0Trace JSON
Claude Fable 5.1Completed4.063.521,871.0Trace JSON
Gemini 3.8 FlashNot completed8.032.867,145.0Trace JSON
Qwen 3.8 MaxNot completed8.0184.377,332.0Trace JSON
Qwen 3.5 397B-A17BNot completed8.082.471,151.0Trace JSON
Qwen 3.5 27BNot completed4.047.924,423.0Trace JSON
Claude Opus 5.5Completed9224.625,233Trace JSON
DeepSeek 4.1 Flash-1017Not completed825.163,349Trace JSON
Gemini 4 ArgonCompleted799.755,322Trace JSON

Watch and inspect this comparison →

ScienceWorld · Planning

Test Water Conductivity

Task: Test the water’s electrical conductivity. Put conductive water in the blue box, otherwise in the orange box.

2 of 10 recorded models completed this task. Compare actions, feedback and resource use.

Select a decision round to inspect the executed actions, returned feedback and native completion check.

ModelOutcomeModel callsTime (s)TokensEvidence
GPT-6 AstraCompleted8353.1201,345Trace JSON
GPT-6 SolNot completed7.0507.0177,327.0Trace JSON
Claude Fable 5.1Completed8128.343,741Trace JSON
Gemini 3.8 FlashNot completed8.036.962,916.0Trace JSON
Qwen 3.8 MaxNot completed8.0582.9105,089.0Trace JSON
Qwen 3.5 397B-A17BNot completed8.079.859,840.0Trace JSON
Qwen 3.5 27BNot completed8.077.963,549.0Trace JSON
Claude Opus 5.5Not completed8.0180.636,159.0Trace JSON
DeepSeek 4.1 Flash-1017Not completed8.062.369,168.0Trace JSON
Gemini 4 ArgonNot completed8370.471,987Trace JSON

Watch and inspect this comparison →

VIMA-Bench · Manipulation

Sweep Without Touching

Task: sweep without touching

4 of 10 recorded models completed this task. Compare actions, feedback and resource use.

Select a decision round to inspect the executed actions, returned feedback and native completion check.

ModelOutcomeModel callsTime (s)TokensEvidence
GPT-6 AstraCompleted796.859,118Trace JSON
GPT-6 SolNot completed16.0167.6242,580.0Trace JSON
Claude Fable 5.1Completed11401.2159,714Trace JSON
Gemini 3.8 FlashNot completed16.0216.7440,383.0Trace JSON
Qwen 3.8 MaxNot completed16.02,139.9429,522.0Trace JSON
Qwen 3.5 397B-A17BNot completed16.0168.9393,377.0Trace JSON
Qwen 3.5 27BNot completed16.0179.5376,815.0Trace JSON
Claude Opus 5.5Completed13229.4185,656Trace JSON
DeepSeek 4.1 Flash-1017Completed13246.4204,104Trace JSON
Gemini 4 ArgonNot completed16704.3433,521Trace JSON

Watch and inspect this comparison →

VIMA-Bench · Manipulation

Rearrange, Then Restore

Task: rearrange then restore

6 of 10 recorded models completed this task. Compare actions, feedback and resource use.

Select a decision round to inspect the executed actions, returned feedback and native completion check.

ModelOutcomeModel callsTime (s)TokensEvidence
GPT-6 AstraCompleted5.076.229,392.0Trace JSON
GPT-6 SolCompleted9262.193,800Trace JSON
Claude Fable 5.1Completed6.0206.862,849.0Trace JSON
Gemini 3.8 FlashCompleted16300.6308,096Trace JSON
Qwen 3.8 MaxNot completed16.01,790.6307,028.0Trace JSON
Qwen 3.5 397B-A17BNot completed16.0170.0259,246.0Trace JSON
Qwen 3.5 27BNot completed16.0187.4299,495.0Trace JSON
Claude Opus 5.5Completed7.0321.091,739.0Trace JSON
DeepSeek 4.1 Flash-1017Not completed16196.9196,872Trace JSON
Gemini 4 ArgonCompleted10382.0163,563Trace JSON

Watch and inspect this comparison →

VirtualHome · Planning

Pick Up the Phone

Task: Pick up phone: walk to living room , find phone ,pick up phone 3

7 of 10 recorded models completed this task. Compare actions, feedback and resource use.

Select a decision round to inspect the executed actions, returned feedback and native completion check.

ModelOutcomeModel callsTime (s)TokensEvidence
GPT-6 AstraCompleted3.0200.070,119.0Trace JSON
GPT-6 SolCompleted7.0600.9296,976.0Trace JSON
Claude Fable 5.1Completed6274.185,053Trace JSON
Gemini 3.8 FlashCompleted6.0223.7121,795.0Trace JSON
Qwen 3.8 MaxCompleted7642.2189,492Trace JSON
Qwen 3.5 397B-A17BNot completed8.0359.5211,384.0Trace JSON
Qwen 3.5 27BNot completed8.0393.9196,302.0Trace JSON
Claude Opus 5.5Not completed8.0286.329,457.0Trace JSON
DeepSeek 4.1 Flash-1017Completed8318.2203,491Trace JSON
Gemini 4 ArgonCompleted4546.653,387Trace JSON

Watch and inspect this comparison →

VirtualHome · Planning

Watch TV

Task: Watch TV: I like to set on my sofa.Turn on the tv. And watch Perry Mason. Monday's through Friday's.

7 of 10 recorded models completed this task. Compare actions, feedback and resource use.

Select a decision round to inspect the executed actions, returned feedback and native completion check.

ModelOutcomeModel callsTime (s)TokensEvidence
GPT-6 AstraCompleted4.0361.8110,665.0Trace JSON
GPT-6 SolCompleted7488.5305,218Trace JSON
Claude Fable 5.1Completed4.0186.549,300.0Trace JSON
Gemini 3.8 FlashCompleted8249.7230,095Trace JSON
Qwen 3.8 MaxCompleted6.0422.1142,038.0Trace JSON
Qwen 3.5 397B-A17BNot completed8.0271.7208,925.0Trace JSON
Qwen 3.5 27BNot completed8.0439.1222,022.0Trace JSON
Claude Opus 5.5Completed7.0211.05,198.0Trace JSON
DeepSeek 4.1 Flash-1017Not completed8282.4203,057Trace JSON
Gemini 4 ArgonCompleted81,188.3232,822Trace JSON

Watch and inspect this comparison →

VirtualHome · Planning

Turn On the Bedroom Light

Task: Turn on light: I walk towards to the bedroom, and I open the door of the bedroom, and I switch on the lights inside the bedroom, the switch is beside the door.

8 of 10 recorded models completed this task. Compare actions, feedback and resource use.

Select a decision round to inspect the executed actions, returned feedback and native completion check.

ModelOutcomeModel callsTime (s)TokensEvidence
GPT-6 AstraCompleted3.0242.172,168.0Trace JSON
GPT-6 SolCompleted4.0335.5115,240.0Trace JSON
Claude Fable 5.1Completed3.0212.927,722.0Trace JSON
Gemini 3.8 FlashCompleted4.0211.656,099.0Trace JSON
Qwen 3.8 MaxCompleted5.0550.8107,838.0Trace JSON
Qwen 3.5 397B-A17BNot completed8.0410.0220,988.0Trace JSON
Qwen 3.5 27BNot completed8.0495.0215,219.0Trace JSON
Claude Opus 5.5Completed8522.4185,872Trace JSON
DeepSeek 4.1 Flash-1017Completed7288.5170,216Trace JSON
Gemini 4 ArgonCompleted61,047.5127,575Trace JSON

Watch and inspect this comparison →

VLABench · Manipulation

Put the Rose in the Vase

Task: Insert the rose into the vase_seen.

2 of 10 recorded models completed this task. Compare actions, feedback and resource use.

Select a decision round to inspect the executed actions, returned feedback and native completion check.

ModelOutcomeModel callsTime (s)TokensEvidence
GPT-6 AstraNot completed8.0197.2142,387.0Trace JSON
GPT-6 SolNot completed7.01,374.2264,224.0Trace JSON
Claude Fable 5.1Completed12332.8268,946Trace JSON
Gemini 3.8 FlashNot completed16.0416.1508,090.0Trace JSON
Qwen 3.8 MaxNot completed10.01,298.3547,359.0Trace JSON
Qwen 3.5 397B-A17BNot completed16.0242.7219,652.0Trace JSON
Qwen 3.5 27BNot completed15.0371.3569,158.0Trace JSON
Claude Opus 5.5Completed12.0463.780,483.0Trace JSON
DeepSeek 4.1 Flash-1017Not completed16171.4312,092Trace JSON
Gemini 4 ArgonNot completed14855.1599,620Trace JSON

Watch and inspect this comparison →

VLABench · Manipulation

Add Salad Dressing

Task: Add salad_dressing to the dish

5 of 10 recorded models completed this task. Compare actions, feedback and resource use.

Select a decision round to inspect the executed actions, returned feedback and native completion check.

ModelOutcomeModel callsTime (s)TokensEvidence
GPT-6 AstraCompleted7470.9199,240Trace JSON
GPT-6 SolCompleted7319.4164,040Trace JSON
Claude Fable 5.1Completed4.0120.778,032.0Trace JSON
Gemini 3.8 FlashCompleted9138.4328,477Trace JSON
Qwen 3.8 MaxNot completed14.0957.2593,283.0Trace JSON
Qwen 3.5 397B-A17BNot completed14.0155.3557,034.0Trace JSON
Qwen 3.5 27BNot completed13.0360.9598,823.0Trace JSON
Claude Opus 5.5Completed8662.2119,747Trace JSON
DeepSeek 4.1 Flash-1017Not completed16186.8510,934Trace JSON
Gemini 4 ArgonNot completed13288.8601,737Trace JSON

Watch and inspect this comparison →

Evidence and Timing

Source hashes tie the recorded actions and feedback to each result. Video playback and episode wall time use separate clocks. Native calls, model responses and environment steps are distinct counters.

Download episode metrics · Gemini 4 Argon log verification