VIALS-AA: different paths to scientific image reasoning

Overview

Example trajectories: different paths through scientific images

Short inspection and explicit measurement can both work

Interpreting images without native vision

Tool usage differs vastly between models

We classified tool calls across all 19 models in our original test set, covering 9,177 rollouts. Most calls were about visual evidence. 38.3% of tool calls were viewing images, while 47.3% prepared or measured them, creating crops, enhancing views, and extracting measurements from pixels. Image measurement accounted for just 5–6% of tool calls for GPT-6 Astra (max) and GPT-6.1 Sol (max), compared with 25–27% for Claude Opus 5.5 (max) and Claude Sonnet 5.5 (max).

VIALS-AA: Tool Calls per Task

Mean tool calls per task, averaged over three independent repeats

No publicly available API

Calls that finish or abandon the task are excluded from tool-call counts. Per-task values are averaged across recorded repeats, with up to 483 observations per model. Tool calls per turn is the total number of counted tool calls divided by the total number of completed model turns. Repeats without a recorded turn count, such as an immediate safety block, are excluded.

How we categorize tool calls

One primary category per call, based on its main activity.

Image viewing comes from the view_image tool; the other activity groups classify code_exec calls. Other includes file and environment inspection, answer writing, calculation, OCR, and unclear activity. OCR accounts for 12.4% of Other calls; Gemini 3.8 Flash (high) accounts for 21% of OCR calls.

Not every code execution does analysis

Gemini 3.8 Flash (high) accounts for 86.7% of the cohort’s calls classified as commentary/no-op. This represents 5.6% of its tool calls, or about 3.8 of its 67.5 calls per task.

commentary/no-op tool calls cover static narration, comments, or printed text without new computation or file analysis. In Protein spot 22, a commentary/no-op tool call prints an interpretation of the spot labels. The code execution tool acts as a notepad: the model writes out its interpretation and the text returns in the next model request.

More intelligent models tend to call fewer tools

Models solve similar numbers of tasks, but top scorers are far more consistent

Gemini 4 Carbon (high) leads in three of eight domains

Cost per Task varies by nearly 100× across models

VIALS-AA resources

注意转载授权 · 阅读原文