Supported

How the glm/glm-5.3 row was checked (2026-10-14): each of its three harnesses (mini-swe-agent 2.4.6, Claude Code 2.1.285, dimcode 0.1.0-next.7) ran whole trials through the model gateway, on Linux arm64 under bubblewrap, against a scripted local stand-in for the plan's endpoint. With GLM-5.3 itself they were checked only up to the plan's usage-limit error (HTTP 429), so no trial on the plan has reached a result yet, and how these pairs do on a task is untested.

How the jev row was checked: first (2026-10-20), whole trials on nav tasks, on macOS (Apple silicon) under Seatbelt and on Linux x86_64 without a sandbox, against a scripted local stand-in for TypeSafe's System One endpoint that answers by applying the questions' criteria literally. Then on Jev itself (2026-10-21), with Robo Use 0.6.0 on Linux x86_64 without the sandbox (ROBOUSE_SANDBOX=0): typesafe/jev-latest, which answered as Jev 1.13.0, ran each of the 40 nav tasks once. 13 scored 1; the other 27 ran out of their 1,200-step budget, with Jev never answering finished. The 40 trials cost $0.76. An audit of each trial found that every request the episode server got, apart from the runner's own status check, was one the harness's client sent from Jev's answers. One try per task is not a measured success rate.

How the Gemini 4 Argon and Glimpse rows were checked: first (2026-10-20), each pair ran a whole trial (arc-gravity-vision) on Linux x86_64 without the sandbox (ROBOUSE_SANDBOX=0), against scripted local stand-ins for the Gemini API and for OpenRouter's chat completions endpoint (tests/model_standin.py), with Gemini CLI 0.64.0, Claude Code 2.1.304, mini-swe-agent 2.4.6 and dimcode 0.1.0-next.7; Claude Code on gemini/gemini-4-argon also ran under Seatbelt on macOS. The stand-ins check the plumbing: the routes, the signatures Gemini puts on tool calls, images in tool results, token counts and costs. Then on Gemini 4 Argon itself (2026-10-21), with Robo Use 0.6.0 on Linux x86_64, again without the sandbox, one try per task: Gemini CLI 0.64.0 ran five trials (one-demo-box-mirrored, arc-gravity-vision, libero-pro-spatial-swap-1, one-demo-drawer-left and libero-pro-object-swap-3), and Claude Code 2.1.304 on gemini/gemini-4-argon one (arc-gravity-vision). All six scored 1, for $1.18 in all at the price table's rates. An audit of every tool call found no file named outside the trial's workspace, no network access and no request to the episode server from outside a call that ran robo. Six trials, one per task, check these pairs with the model; they do not measure a success rate. Kinex has run on gemini/gemini-4-argon too (below). No request has reached OpenRouter, so how Glimpse and openrouter/google/gemini-4-argon do on a task is untested, as is how mini-swe-agent and dimcode do on Gemini 4 Argon, and Gemini CLI has not run under the sandbox.

How the aspire row was checked (2026-10-20): aspire ran whole trials on five capx tasks with Claude Code 2.1.304, on Linux x86_64 without a sandbox, through the model gateway to a scripted local stand-in model that answers with fixed tool calls. That checks the practice episodes, the scored episode's lock, Stage 2 and the skill library. With claude-opus-5-5, it ran five trials on four tasks with smaller practice bands than ASPIRE's and a one-hour limit per trial: capx-cube-lifting-s5, capx-cube-stack-s5 and capx-nut-assembly-s5 scored 1 (after a first capx-cube-stack-s5 trial scored 0 because Claude Code stopped ASPIRE's worker early, fixed since), and libero-pro-10-task-6 scored 0 (practice time ran out before a program was selected). Under the macOS sandbox, whole aspire trials have run only in the test suite, with a scripted agent on a light task. aspire-codex has not run a whole trial, and no capx-libero trial has run. Five trials check the harness with a model; how these pairs do on a suite is untested.

How the kinex row was checked: first (2026-10-20), Kinex 1.0.0 ran whole trials through the model gateway against a scripted local stand-in for the plan's endpoint, which answered with fixed Kinex tool calls, on Linux x86_64 without the sandbox (that machine has no bubblewrap). Through the Kinex devices, those calls solved arc-gravity and libero-spatial-0, and on arc-gravity-vision moved the hand with Kinex's servo to a pose and placed it in the front camera's frame from the camera's calibration. Robo Use's side of the harness (the bridge, the device drivers and Kinex's Python environment, with a stand-in for Kinex itself) ran under the macOS sandbox. Then with a real model (2026-10-21): Kinex 1.0.0 on Gemini 4 Argon (gemini/gemini-4-argon) through the model gateway, with Robo Use 0.6.0 on Linux x86_64, again without the sandbox, ran four trials, one try per task. arc-gravity, libero-spatial-0, libero-object-0 and libero-goal-8 all scored 1, for $0.40 in all. An audit of each trial's system calls and Kinex session found no file of the task, its reference solution or another trial opened, and no request to the episode server but the bridge's and the agent's robo commands. Four trials check the harness with a model; they do not measure a success rate.

• Tasks: BenchFlow task folders (task.md, schema 1.3).
• Trials: BenchFlow's trial layout (config.json, result.json, verifier/reward.txt), with the agent's steps in ATIF v1.7 (agent/trajectory.json).
• Trial labels: the input and output files of NVIDIA's Flora scripts (Flora-Rewarder's terminal_reward and dense_reward at revision 7de68d2, the captioning pipeline's infer.py at f736ca3), through robouse flora export and import; checked against the scripts' dry-run and validation modes, a stand-in, and the models themselves on an H100 (the Rewarder on 21 trials, the captioning pipeline on 14) (Annotate trials with Flora).