Leaderboard
Solve rate per model, one trial per task, with 95% intervals; agent harnesses and VLA policies ran different tasks, so each is ranked on its own.
No image input (GLM-5.3 · mini-swe-agent, Kimi K3 · mini-swe-agent). The models never received camera images as input. On vision tasks they read the saved camera frames with Python they wrote, using the camera matrices from robo info where the task provides them.
Opus 5.5 · Claude Code ran 290 of its 352 tasks on Oct 13, 2026, on Linux with Robo Use 0.4.0. The other agent pairs, and its own 62 Meta-World and Gymnasium-Robotics tasks, ran Sep 27 to 29, 2026 on macOS. The Claude Code version, time limits and step budgets were the same, with one exception: one BEHAVIOR-1K task allowed 85 steps instead of 90. The task text and observations were also the same, except on 32 tasks corrected in between (20 LIBERO, 6 RoboCasa, 5 BEHAVIOR-1K, 1 DexJoCo); Opus 5.5 · Claude Code ran those 32 in the corrected form. Without them the order is unchanged (Opus 5.5 · Claude Code 92%, GPT-6 Astra · Codex 86%). On 13 RoboHarm and Drone tasks, Claude Code's refusal fallback answered the first run with Opus 4.8; those trials are not counted, and all 13 were rerun with CLAUDE_CODE_DISABLE_REFUSAL_FALLBACK=1. 3 reruns count; on the other 10 (9 RoboHarm, 1 Drone) Opus 5.5's safeguards refused the session on every attempt, so they are not scored. Kimi K3 · mini-swe-agent: 2 solved trials were removed after an audit and their tasks are not counted (on libero-goal-4 the agent replayed the task's reference demonstration; on libero-close-drawer it read the task's verifier and a stub of its reference solution from a shared folder during its episode, though they held nothing beyond its own observations). GLM-5.3 · mini-swe-agent: 2 solved trials were removed after an audit and their tasks are not counted (on metaworld-hammer-vision the agent downloaded Meta-World's source from GitHub and took the hidden object positions from it; on robosuite-twoarm-peginhole-panda it read robosuite's source, left in a shared folder by another trial).