Model Selection & Evaluation

Choosing a VLM for Robot Tasks

Start with the capability your task needs, then check whether the agent can satisfy the complete goal within your observation, tool, and time constraints.

Research guide updated 8 November 2026; results correspond to arXiv v3, 23 October 2026. Ten models, five domains, and the completed Planning set. Read all 80 comparison entries.

Which Frontier VLM Performs Best on Robot Tasks?

GPT-6 Astra leads Task Planning (80.3%) and rotation estimation. Gemini 4 Argon leads Spatial Pass@1 (71.6%) and region overlap (0.356 box–mask IoU; 0.441 polygon IoU). Claude Opus 5.5 has the highest reported Manipulation success (47.8%). Even the strongest Manipulation score leaves more than half of the assigned tasks incomplete. Table 2.

Strengths differ across domains. Gemini 3.8 Flash has the lowest numerical Spatial Reasoning error (AbsRel 0.161); Qwen 3.8 Max nearly matches Astra on Spatial Pass@1 (68.9% versus 69.4%). DeepSeek 4.1 Flash-1017 reaches 51.6% Spatial Pass@1, 31.2% Planning, and 20.9% Manipulation, with substantial differences across sources. Full ten-model comparison.

GPT-6 Sol reaches 44.6% Planning and 19.8% Manipulation; Fable 5.1 reaches 63.7% and 24.7%. Qwen 3.5 397B-A17B outperforms Qwen 3.5 27B on Spatial Pass@1 (50.6% versus 48.0%), while the smaller model has higher Manipulation success (11.0% versus 9.9%). Capability profiles.

How Should Means, Run Variation and Domain Scores Be Read?

In Table 2, a value written as 17.2 ± 6.7 means a mean rotation error of 17.2 degrees and a sample standard deviation across runs of 6.7 degrees. The number after ± is not the mean error or a confidence interval. The worked example below separates these fields for GPT-6 Astra; it is not a new model ranking.

Astra · Paper v3, Table 2 · means and variation in separate columns
Domain and metricMeanRun SDUnitRuns
Geometry · Rotation error17.26.7degrees5
Geometry · Translation error106.936.0cm5
Spatial Reasoning · Pass@169.41.0%5
Task Planning · Pass@180.3Not estimated%1

Rotation and translation here combine Map-free and GeoProbe pose cases; they are not GeoProbe-only scores. Spatial Pass@1 and Planning Pass@1 use different task populations: the 80.3% Planning result must not be reported as Spatial Reasoning accuracy. All models, separate mean/SD fields and units · CSV · Source-specific complementary metrics · Original Table 2.

Why Can Precise Local Actions Still Leave Tasks Unfinished?

Geometric estimates, contact points, and near-correct trajectories describe intermediate competence. Complete tasks also require the right endpoint, ordered subgoals, and the final environment state—all within a finite budget. Geometric and spatial diagnostics.

All ten agents complete 0/5 BEHAVIOR-1K tasks under ceilings of 180 minutes per trial, 24 model rounds, 256 tool calls, 640k tokens, and 1,000 control steps. All ten also complete 0/8 HumanCLAW sitting tasks, with 180 minutes, 101 code rounds, 100 environment steps, 400 primitive calls, and 2M tokens (11,000-second outer timeout). Runs can end without success before the time limit. Opus 5.5 completes 3/13 HumanCLAW tasks and DeepSeek-1017 2/13; their successes are navigation tasks. Gemini 4 Argon also completes 2/13 HumanCLAW tasks, both navigation. Planning and Manipulation results.

The evaluation therefore separates local accuracy from complete-task success. A minimal harness leaves interpretation, reasoning, and action selection to the model. Execution traces separate restarting code from completing a task: 62 of 65 CALVIN episodes with execution errors later run code without an error, but only three subsequently complete all five goals. The fixed-input review study similarly separates improvement in an intermediate score from satisfying a success criterion. Feedback and recovery.

Which Model Versions Were Evaluated?

Names and configured API identifiers follow Table A5. DeepSeek-1017 (DS-1017 in figures) denotes DeepSeek-V4.1-Flash-1017, accessed through deepseek-flash on 18–19 October 2026, after the 17 October update.

Ten evaluated models and their configured identifiers
ModelPaper AliasConfigured Identifier
GPT-6 AstraAstragpt-6-astra
GPT-6 SolSolgpt-6-sol
Claude Fable 5.1Fableclaude-fable-5-1
Qwen 3.8 MaxQwen-Maxqwen3.8-max
Gemini 3.8 FlashGemini 3.8gemini-3.8-flash
Qwen 3.5 397B-A17BQwen-397Bqwen3.5-397b-a17b
Qwen 3.5 27BQwen-27Bqwen3.5-27b
Claude Opus 5.5Opus 5.5claude-opus-5-5
DeepSeek-V4.1-Flash-1017DeepSeek-1017deepseek-flash
Gemini 4 ArgonGemini 4gemini-4-argon

How Were the Agents Evaluated?

What Does the Robot-Agent Harness Supply?

The harness preserves source-native observations, operations and success criteria while leaving interpretation and action selection to the model. Geometry, Spatial Reasoning, Affordance and Planning use a persistent Python runtime; Manipulation follows its native agent loop. Variables and environment state persist between code rounds. Images produced during a round reach the model in its next request. Paper v3 · Appendix B.1.

Native capability includes the disclosed source helpers. The baseline adds no specialist perception or control modules beyond those interfaces; it does not remove every source-provided helper. Private scoring annotations are unavailable to the agent. Execution errors may be followed by another round when budget remains, and earlier state changes are not automatically undone. Task completion still depends on the native verifier.

For robot harness or agentic robotics comparisons, keep observation schedules and limits attached to each result. This framework and its within-episode feedback do not establish recursive self-improvement or a demonstration-conditioned in-context learning gain. Research relevance and boundaries · Execution recovery versus task completion · Reading the retained traces.

Shared evaluation framework. All ten models use the same task inventory, source-native interfaces, RGB observations and native success criteria. Manipulation retains the observation schedule and resource limits of each recorded batch.

Recorded execution settings. All ten retained Astra CALVIN episodes use 20 rounds, 640k total tokens and 60 minutes. The corresponding Opus 5.5, DeepSeek-1017 and Gemini 4 Argon reports use 40 rounds, 1.28M tokens and 120 minutes with automatic RGB feedback. Verified reports and configuration audit ↗

Repetitions. Non-tracing Geometry uses five runs, except Fable (three). Spatial Reasoning and box/point Affordance use five, except Fable (three). All ten agents use one run per assignment for polygon grounding, tracing, Planning, and Manipulation. Repeated metrics report the mean and sample SD across runs. Each run averages valid continuous estimates; success rates retain failed and missing assignments. Scoring and repetition accounting.

Task sets. Planning covers all 157 cases, including HumanCLAW: Opus 5.5 completes 76/157 (48.4%), DeepSeek-1017 49/157 (31.2%), and Gemini 4 Argon 110/157 (70.1%). Manipulation counts 182 binary assignments; Opus 5.5 completes 87 (47.8%), DeepSeek-1017 38 (20.9%), and Gemini 4 Argon 80 (44.0%). The 183-case inventory also includes one continuous-reward task. Box–mask and polygon IoU use the same 70 Affordance cases under separate output protocols. Invalid or absent polygons receive zero. Polygon grounding.

Use the evaluation harness, 1,000-case dataset, and model configuration tables to reproduce the setup. Download the ten-model comparison CSV , all-model complementary metrics, or source means and SDs; inspect individual predictions and recordings in the case gallery.

Research topics and literature-review guide · GeoProbe methods · Findings and source records · Download BibTeX

Huang et al., Are Frontier VLM Agents Ready to Be Robot Generalists? An Empirical Study with the Embodied Agent Arena, 2026. arXiv:2610.00854 · BibTeX