Geometry · An original benchmark within Embodied Agent Arena

GeoProbe: Geometric Estimation for Vision-Language Models

How well can a vision-language agent estimate camera parameters, distinguish object motion from camera motion, and recover depth and scale from images?

168 GeoProbe queries are included within 370 Geometry cases, within 1,000 Arena cases. The 1,000 total cases span five domains: Geometry, Spatial Reasoning, Affordance, Task Planning and Manipulation. GeoProbe’s 130 unique RGB images and ten diagnostic protocols describe inputs and evaluation conditions, not additional cases. Source: paper v3, Appendix A · Link to dataset scope.

Real photographs are image diagnostics. GeoProbe’s four real-photo displacement queries use fixed tabletop image pairs; they are not closed-loop physical robot experiments. This page documents GeoProbe as introduced in the Arena paper. Evidence: Appendix D.1 · Link to evaluation setting.

168evaluation queries
130unique RGB images
10diagnostic protocols

Page updated · Research reference: paper v3, 23 October 2026 · Ten evaluated VLM agents in the full Arena study.

The benchmark at a glance

One inventory. Six geometric requirements.

Each query is counted once below. The six families contain ten protocols in total.

  1. 40
    Object displacement

    36 rendered + 4 real-photo queries

  2. 36
    Camera calibration

    24 intrinsics + 12 focal-length queries

  3. 24
    Relative camera pose

    Rotation and translation, scored jointly

  4. 12
    Sparse depth

    Metric depth and ordinal relationships

  5. 12
    Object dimensions

    Numerical dimensions, with units

  6. 44
    Reference-based scale

    Scale estimated using a reference

168 queries in total. Queries reuse some of the 130 unique RGB images; ten protocols are evaluation conditions, not ten models or ten task families. Inspect the full protocol inventory →

Which VLM geometry questions can GeoProbe answer?

For research on geometric understanding in multimodal models, GeoProbe provides fixed image tasks with numerical targets. These protocols address different requirements; a result on one should not stand in for all forms of spatial reasoning.

Can a VLM estimate camera calibration and relative pose?

The inventory includes 24 full-intrinsics queries, 12 focal-length queries with a centered principal point, and 24 relative-pose queries. Calibration uses focal and principal-point errors; relative pose reports rotation, translation and applicable joint success criteria. Scoring definitions keep angular and metric errors separate.

Can it separate camera motion from object motion?

GeoProbe has 40 displacement queries: 36 rendered queries plus four real-photo queries. The rendered set repeats 12 object interventions under fixed, small-change and large-change camera views, with ten moving and two stationary controls per condition. Relative camera pose is a separate 24-query protocol. Non-overlapping query counts and matched conditions.

Does depth ordering establish metric depth or scale?

No single score establishes all three. Twelve sparse-depth queries, 12 object-dimensions queries and 44 reference-based scale queries evaluate distinct numerical targets. Depth ordering can be correct while metric depth or absolute scale is wrong. Errors retain their units and denominators. Depth and scale metrics describe what each score measures.

Is GeoProbe a robot navigation or manipulation success benchmark?

GeoProbe tests fixed-input geometric estimation. Arena evaluates path constraints, spatial reasoning and interactive task completion in other source groups. Use spatial and grounding evidence or long-horizon completion results when those are the research question.

Attribution for a literature review. Huang et al. (2026) introduce GeoProbe within Embodied Agent Arena to evaluate camera, object-motion, depth and scale estimation with 168 queries over 130 RGB images. Cite paper v3 and link the protocol inventory when describing the dataset. This is an evaluation resource, not evidence that every tested model solves these tasks.

What does GeoProbe measure?

GeoProbe uses fixed image inputs to examine camera calibration, relative camera pose, object displacement, sparse depth, object dimensions, and reference-based scale. Most images are rendered in Blender; four displacement cases use real tabletop photographs. Multiple queries share images or scenes. Neither 168 queries nor 130 unique images denotes 168 or 130 statistically independent environments, scenes or trials.

The rendered displacement conditions share 12 object interventions while changing the camera condition: fixed, small view change, and large view change. Each condition contains ten moving interventions and two stationary controls. This design separates object-motion estimates under different camera changes.

Ten protocols, 168 queries

The allocation below reproduces the GeoProbe inventory in Table A2 of paper v3. Shared images and scenes connect the protocols; the total is 168 queries over 130 unique RGB images.

What is counted in this VLM geometric-estimation benchmark?

GeoProbe contains 168 queries, 130 unique RGB images and ten diagnostic protocols. Queries are evaluation items; images are shared inputs; protocols define different tasks. The full Arena paper v3 evaluates ten VLM agents: that model count is separate from the ten GeoProbe protocols, and repetitions do not change the 168-query inventory.

The ten protocols can be regrouped into the six non-overlapping task families below. 40 + 36 + 24 + 12 + 12 + 44 = 168. Displacement is 36 rendered queries plus four real-photo queries, already totaling 40. The 24 relative-pose queries are counted only under pose, not again under displacement.

Six task families regrouping the same 168 queries · Table A2
Task familyProtocol allocationQueries
Object displacement12 fixed-camera + 12 small-view + 12 large-view + 4 real-photo40
Camera calibration24 full-intrinsics + 12 focal-length36
Relative camera pose24 relative-pose24
Sparse depth12 sparse-depth12
Object dimensions12 object-dimensions12
Reference-based scale44 reference-based scale44
Total, counting each query once168
GeoProbe task allocation · Paper v3, Table A2
Diagnostic protocolQueries
Object displacement · fixed camera12
Object displacement · small view change12
Object displacement · large view change12
Real tabletop displacement4
Full camera intrinsics24
Focal length · centered principal point12
Relative camera pose24
Sparse depth12
Object dimensions12
Reference-based scale44
Total168

Relationship to the Arena: GeoProbe’s 168 queries are included within the 370 Geometry cases, which are included within the 1,000 Arena cases. InFlux-Real, Map-free and MultiSPA-derived are separate Geometry source groups, not GeoProbe protocols. The Arena’s 32 established sources retain their original attribution. For a dataset comparison, use 168 for GeoProbe rather than 370 or 1,000. Link to source boundaries · Source inventory.

How are predictions scored?

Each diagnostic retains its units and success criterion. A small continuous error and a successful prediction are reported separately where the protocol defines both. The following definitions come from Appendix B.3.

Camera calibration

Focal MAPE averages absolute relative errors in f_x and f_y. Principal-point error is measured in pixels and relative to the image diagonal.

Joint calibration requires each focal component’s relative error ≤ 10% AND principal-point error ≤ 2% of the image diagonal. Passing a mean focal-error threshold alone does not satisfy this joint criterion. Link to calibration scoring.

Relative camera pose

Rotation error is reported in degrees and translation error in centimeters. Joint success requires rotation error ≤ 2° AND translation error ≤ 2 cm.

These metrics apply to the 24 GeoProbe relative-pose queries. Passing either component alone is insufficient. Link to pose scoring.

Object displacement

Diagnostics evaluate movement status, direction and absolute distance error. The GeoProbe joint criterion requires direction agreement AND absolute distance error ≤ 1 cm.

Stationary controls are evaluated without dividing by a zero displacement target. The paper’s 20% relative-motion tolerance belongs to MultiSPA-derived, a different source group. Link to displacement scoring.

Depth and scale

Depth uses dimensionless AbsRel, RMSE in meters, threshold accuracy and ordinal accuracy. AbsRel is a ratio; MAPE is 100 × AbsRel and is expressed as a percentage. Focal length and dimensions use absolute and relative errors; numerical scale and qualitative scale judgments retain their separate metrics.

Oracle depth alignment uses ground-truth reference depth to isolate scale bias. An aligned score does not establish the model’s unassisted metric-depth accuracy or a deployable correction. Link to depth and scale scoring.

Does the number after ± measure geometric error?

In a reported mean ± SD, the first number is the mean score and the second is the sample standard deviation across evaluation runs. The SD measures variation between runs; it is not a second mean error, a confidence interval, a standard error or variation across individual cases. Both numbers retain the metric’s unit. For a percentage score, SD is expressed in percentage points.

Astra on the 24 GeoProbe pose queries · Five runs · CSV values rounded to three decimals
CSV metricMean errorRun SDUnit
GP rotation1.1740.382degrees
GP translation5.3592.115centimeters

For example, 1.174 ± 0.382 degrees means mean rotation error 1.174 degrees with run SD 0.382 degrees. These continuous means average valid predictions; they are not joint-success rates and do not establish that every case passed the 2° / 2 cm thresholds. The CSV also provides per-run means and valid-prediction counts.

Repetitions and denominators: GeoProbe is part of non-tracing Geometry: five repetitions per case for nine models and three for Fable. Continuous metrics average valid predictions within each run and then report the equal-weight mean and sample SD across runs. Success-rate denominators retain failed assignments; depth threshold and ordering measures count invalid predictions as incorrect. Consult Appendix B.4 before comparing aggregates.

What can these results establish?

  • Controlled geometric diagnosis. Matched rendered interventions help examine changes in prediction when the camera condition changes. These are fixed-input estimation tasks; they do not measure physical robot deployment success or establish that improving a geometry score causes better downstream task completion.
  • Limited real-photo coverage. The four tabletop image pairs form a separate subset. Their displacement labels are supplied annotations, without independent camera or metric calibration. Their scenes and motions differ from the Blender comparison. Four annotated cases do not establish broad real-world generalization.
  • Metric-specific conclusions. A model may preserve depth ordering while missing absolute scale. Continuous-error comparisons must be read with valid-prediction coverage and success criteria.
  • Source-specific results. The homepage’s main rotation and translation columns combine 30 Map-free and 24 GeoProbe pose cases. Their pooled scores cannot be attributed to GeoProbe alone. Use the source-specific GP rotation and GP translation rows in the complementary results and Appendix D.1 for claims specifically about GeoProbe.

Inspect the data and reproduce a diagnostic

GeoProbe is distributed within the full Embodied Agent Arena dataset. Use the pinned dataset revision below with the repository’s installation and evaluation instructions. Query IDs, inputs and scoring remain associated with their source protocols.

python scripts/fetch_data.py \
  --repo-id uuu-Quant/Embodied-Agent-Arena \
  --revision 5f729375eb35163a024625dd8ea72ac57a427a24 \
  --output ../data
arena validate --data-root ../data --hashes
arena list --data-root ../data

After installing the harness and configuring the model credentials, a depth diagnostic can be selected with --benchmark w1_vlm_depth. Use the task index and evaluation protocol to preserve the relevant inputs, repetition counts and budgets.

Cite GeoProbe and its methods

GeoProbe is introduced in the Embodied Agent Arena paper, not in a separate publication titled after this page. Cite the paper for the benchmark. For an answer or literature review using these definitions, also link the specific evidence on this page: query counts, scoring criteria, mean and SD interpretation, or scope and limitations.

Haojian Huang, Pukun Zhao, Zexi Li, Yehang Zhang, Yangkai Wei, Wenqian Li, Han Yang, Kaiwen Zhou, Ying-Cong Chen, and Yinchuan Li. Are Frontier VLM Agents Ready to Be Robot Generalists? An Empirical Study with the Embodied Agent Arena. arXiv:2610.00854, 2026. Version 3, 23 October 2026.

Research content follows the 23 October 2026 paper. The 14 November page update clarifies dataset scope and the distinction between real-photo diagnostics and physical robot evaluation; it does not report a new experimental run.

Paper citation · BibTeX
@article{huang2026frontier,
  title={{Are Frontier VLM Agents Ready to Be Robot Generalists? An Empirical Study with the Embodied Agent Arena}},
  author={Huang, Haojian and Zhao, Pukun and Li, Zexi and Zhang, Yehang and Wei, Yangkai and Li, Wenqian and Yang, Han and Zhou, Kaiwen and Chen, Ying-Cong and Li, Yinchuan},
  journal={arXiv preprint arXiv:2610.00854},
  year={2026},
  doi={10.48550/arXiv.2610.00854},
  url={https://arxiv.org/abs/2610.00854v3},
  eprint={2610.00854v3},
  archivePrefix={arXiv},
  primaryClass={cs.RO},
  note={Version 3, 23 October 2026}
}