Robo Use is open source

The agent is the policy

Robo Use turns robot benchmarks into tasks for coding agents. The agent gets the task as its prompt, a fresh workspace and a robot it can reach only through the robo command. It looks with robo observe, moves with robo act, robo move-to or a skill such as robo skill grasp, and ends with robo done or robo give-up.

Each command is a request to an episode server that the runner starts for the trial. The server owns the simulator: it applies each action, counts every step against the task’s budget and records the video. After robo done it holds the robot still for about 10 steps and judges the final physical state. What the agent says it did does not count.

Any program that can run shell commands can be the policy this way. Robo Use ships harnesses for coding agents, and a client for vision-language-action (VLA) policies served elsewhere, which drives the same robots through the same requests.

What a trial leaves behind

Every trial is a folder. It holds the video (artifacts/recording.mp4), the agent’s trajectory (agent/trajectory.json), every robo request with its response (episode/trace.jsonl), the server’s verdict (episode/result.json) and the reward (verifier/reward.txt). The server signs its verdict with a key the agent never sees, and the runner scores only a signed verdict.

Every task ships with a reference solution, a script that must score 1, and a no-op control, which ends the episode without moving and must score 0. The release gate checks both on every task in the package.

What is in 0.6.0

Tasks come from Meta-World, LIBERO, robosuite, RoboCasa, Gymnasium-Robotics, dm_control, MyoSuite, DexJoCo and other benchmarks, and from Robo Use’s own suites: rule puzzles on a table, and mobile manipulators, quadrupeds, humanoids, dexterous hands and drones built on MuJoCo Menagerie robots. What follows is new in this release: NVIDIA’s ASPIRE as a harness and three new suites, each with real trials, a first run of NVIDIA’s Flora on those trials, which ships in 0.6.1, and the new harnesses and models.

ASPIRE: practice first, then one scored run

ASPIRE, from NVIDIA’s GEAR lab (paper), is a workflow for coding agents that write robot programs. For each task a worker subagent explores the scene, writes a first program, runs every version on a band of development seeds and fixes the largest failure first. A verifier subagent then tries to solve the task again from the skill library and the worker’s proposed entries alone, without the worker’s code. Only entries the verifier can solve the task from go into the library, which later tasks read.

Robo Use runs that workflow inside a trial with two harnesses, aspire with Claude Code and aspire-codex with Codex, which follow ASPIRE’s own skill files, shipped unchanged. A trial still has one task and one scored episode, so the harness splits it into practice and one frozen run.

1. Practice. The agent runs its programs as ASPIRE’s skills say, but each run is a practice episode: the task’s own episode server on another seed. Seeds that would show the scored scene are refused.
2. The frozen run. At the practice deadline, or when the agent exits, the harness stops every agent process, copies the selected program out and runs it once on the scored episode, then robo done. With no program, it gives the episode up, which scores 0.
3. The lock. The scored episode answers only requests that carry a token the agent never gets. Until the frozen run opens it, only the runner’s status check and give-up get through, and one refused request leaves the trial unscored.

With a library folder set, each trial starts from it, and the entries its own verifier verified go back into it. A later trial of the same task on another scene, with the same harness and model, runs the stored program without practising again.

In a pilot we ran Claude Code with Opus 5.5 on three CaP-X tasks, where programs get object poses from the simulator, and on one LIBERO-Pro task, where they get only camera images. Trials ran one at a time, with a one-hour limit each and one skill library shared by all of them. The seed bands were smaller than ASPIRE’s: 8 development seeds and 2 unseen-check seeds, where ASPIRE’s own Robosuite bands have 25 and 5.

Cube lifting and nut assembly scored 1.0; the video above is nut assembly’s scored episode. The first cube-stacking trial scored 0 because of the harness, not the model. In print mode Claude Code stops background work 10 minutes after the last turn, and ASPIRE’s coordinator waits on a background worker, so the worker was stopped mid-practice before it selected a program. The harness now has Claude Code wait until the practice deadline, and the rerun scored 1.0. The LIBERO-Pro task asks for the red mug on the plate, then the chocolate pudding to its right, from camera images alone. The worker ran 35 practice episodes, 7 of them successful, but had selected no program when the practice deadline stopped it.

The pilot ran without a sandbox, on a machine without bubblewrap, and before the lock reached its final form, so each trial was audited from its folder. In every trial the first request to reach the scored episode was the harness’s own, opening the lock.

Five trials on four tasks check the harness with a real model. They do not measure how ASPIRE with Claude Code does on a suite. Robo Use also runs ASPIRE through each task’s own interface: on libero-pro, programs send LIBERO’s end-effector deltas and see RGB images, where ASPIRE’s own setup sends joint positions and perceives with RGB-D images, SAM3 and Contact-GraspNet. So a libero-pro score here is not comparable with ASPIRE’s own LIBERO-PRO results.

A trial on aspire gets at least four hours unless you set a timeout, and aspire-codex has not yet run a whole trial. Run agents and VLAs lists the settings: the seed bands, the practice budget, the persistent library, and ASPIRE’s Evolutionary Search in place of its fix loop.

LIBERO-PRO: LIBERO, perturbed

LIBERO-PRO perturbs LIBERO’s tasks: objects swap places, or a familiar scene gets a new goal. The libero-pro suite holds 73 of its position-swap and new-goal tasks: 73 of the 80 that NVIDIA’s ASPIRE reports on, the ones a scripted reference solution solves. The agent gets two camera images of 256 by 256 pixels and the robot’s own state, not where the objects are.

Opus solved the position-swap task from LIBERO-Spatial, the black bowl onto the plate, in 165 of its 1,000 steps. To place objects it wrote small numpy scripts that turn pixels into table coordinates with the camera matrices from robo info. It solved neither of the two long-horizon tasks from LIBERO-10: it ended one with robo done short of the goal and gave up the other. Gemini CLI with Gemini 4 Argon solved the same position swap in 182 steps, and an object swap in 199 (below).

one-demo: one demonstration, a different object

The one-demo suite (16 tasks) shows the agent one demonstration of a skill on one object, as robo commands, replies and frames, and asks for the same skill on a different object: open a hinged box, take a lid off, stand a bottle up, pull a drawer open. Replaying the demonstration, as it is or shifted to the new object, scores 0 on every task.

Opus solved all three tasks it was given. The box was hinged on the other side, so it computed the lid’s arc and moved the gripper through it. The bottle lay the other way round, and it tipped it up about its base through nine waypoints it computed. The drawer faced left instead of to the front. The suite cannot yet say how much the demonstration helped, because the goal is also stated in words and the new object is given as numbers.

The libero-pro and one-demo trials ran one at a time, on the task folders that shipped in 0.6.0, on a Linux machine without bubblewrap, so Robo Use’s sandbox was off. Each trial was audited before its result or footage was used: a script listed the files the agent named outside its workspace, any network access, and anything other than robo that could reach the simulator, and then every command was read. All six were clean, and every trial is reported here, the two failures included.

chemlab: laboratory instruments

The chemlab suite (7 tasks) operates six laboratory instruments from MBZUAI’s ChemLab-Assets with a floating Franka Hand, MuJoCo Menagerie’s model with its own meshes, finger pads and joints, each instrument with its own behaviour runtime running alongside. The tasks: raise a fume hood’s sash to its working stop, close a drying oven’s door, open an incubated shaker’s hood, close an analytical balance’s draft shield, open a moisture analyzer’s lid, and set a hotplate to 100 °C or its stirrer to 500 rpm. The instrument files allow only non-commercial use, so the suite runs from a checkout of the repository and is not in the package. The Franka Hand replaces the simpler two-finger gripper of 0.6.0 and comes in the next release; the runs below are on that version.

Claude Code with Opus 5.5 solved all seven on its first try, for $1.30 in all. On the fume hood it raised the sash with a single robo act repeated 25 times. On the drying oven it wrote a short script that pushes the door a little further along its arc, and ran it eleven times as the door swung shut. On the hotplate it closed the hand on the temperature knob and turned it with one robo act repeated 30 times.

These trials ran one at a time on the version with the Franka Hand, on a machine without bubblewrap, each audited from its folder before its result or footage was used. All seven were clean: no agent touched a file outside its workspace except one scratch script it wrote itself, none used the network, and every request in each episode’s trace matches a robo command the agent ran.

NVIDIA’s Flora on Robo Use trials

Flora, also from NVIDIA’s GEAR lab, is a suite of open video models that NVIDIA uses to label the robot episodes its GR00T models train on. Flora-Rewarder scores an episode from 1 (complete failure) to 5 (complete success) and labels every half second as normal, suboptimal or failure. Every Robo Use trial already has the physics simulator’s verdict, so the trials make a labelled test set for a judge that only watches the video. robouse flora export writes the input files of NVIDIA’s own inference scripts, and robouse flora import reads their labels back into the trial folders. Both ship in 0.6.1.

We ran Flora-Rewarder, unmodified, on 21 of the Claude Code trials with Opus 5.5: those of the launch run, the ASPIRE pilot and chemlab’s first runs, made with its earlier gripper. Counting a 5 as solved, its score matched the simulator’s verdict on 11. The simulator solved 19; Flora gave 9 of those a 5, and gave both failed trials a 3. By group, it agreed on 9 of the 11 launch-run trials, 2 of the 7 chemlab trials and none of the 3 ASPIRE trials.

Where Flora disagreed, its reasons were about what the frames showed or did not show: a lift seen from above, a bowl still in the gripper when the episode ended, a hotplate dial whose setting the last frame does not show. Two failed trials are too few to say how well it catches failures. Scoring 25 recordings, labelling every half second and captioning 14 of them took 55 minutes of H100 time and cost $5.23. Flora’s weights and scripts are under NVIDIA’s license for research and evaluation use, and Robo Use includes none of them: you download them on your own GPU machine.

Gemini CLI, Kinex and Jev: their first real runs

Three harnesses are new in 0.6.0. Gemini CLI runs Google’s agent headless with Gemini 4 Argon. Kinex, a robotics coding agent built on the Pi SDK, drives the robot as Kinex devices through a bridge that speaks RoboEnv’s protocol. Jev, TypeSafe’s System One model, drives the nav suite with typed decisions, one every half second of simulated time, in the protocol of Dimensional’s “Can Jev Nav?” study. Until release they had met only scripted stand-ins for their endpoints. Each has now run on the 0.6.0 release, one try per task, and every trial was audited from its folder before its result or footage was used.

Gemini CLI solved all five tasks it was given, for $1.06 in all, and Kinex all four, for $0.40. On the LIBERO-PRO object swap, Gemini found the bottle cap’s pixels in its wrist camera image with numpy and projected candidate gripper positions into that camera, using the projection matrix robo info reports, to choose where to close. Kinex’s first grasp of the black bowl, at its centre, closed on nothing. It read the robot’s camera through its Kinex camera device, came back down on the bowl’s rim and set the bowl on the plate at step 220 of 280, in seven tool calls.

Jev (jev-1.13.0) solved 13 of the 40 nav tasks, for $0.76 in all. To find a wardrobe it drove a mobile robot from the kitchen through the living room into the bedroom, in 71 decisions. Each decision was four typed answers (drive forward or back, slide, turn, continue or finished), with a probability on every option. It answered “finished” with probability 1.00 beside the wardrobe after 35 simulated seconds and 11.5 m, with no collisions, and the run cost $0.0078. All 27 failures ran out of their step budget; Jev never answered “finished” early. Where the straight line to the target was blocked, it often answered with the open side’s name rather than its bearing, and the robot went back and forth. The nav suite’s own scripted controller, which applies the steering rule exactly, solves 19 of the 40.

Glimpse, Elorian’s visual reasoning model, runs through OpenRouter with Claude Code, mini-swe-agent and dimcode, and has not run on a task yet. Gemini 4 Argon also runs through the model gateway with Claude Code, mini-swe-agent and dimcode: with Claude Code it solved ARC gravity from camera images for $0.12. Robo Use also runs Claude Code, Codex, mini-swe-agent, dimcode, Show-Harness, Dream Labs’ Dream-1 and VLA policies.

What an agent can do to its own score

A coding agent that drives a robot runs as your user, on the same machine as the programs that score it. While reviewing 0.6.0 we found seven ways an agent could change its own score or a later trial’s. Six are fixed in 0.6.0. The first four were in code new to this release, and were fixed before it shipped.

• A lock that trusted a label. The first version of the ASPIRE lock let through any request marked as the runner’s, a mark the agent could set itself. Every request now needs the token.
• Modules planted on an import path. The harness process that holds the token had a workspace folder the agent can write on its import path. It now runs with python -I and keeps the workspace off that path.
• A link swapped in during a copy. The copy out of the workspace after practice checked each path and then read it, so a file swapped for a symbolic link in between would have copied an outside file into the shared library. The copy no longer follows links.
• A scene file agents could read. On macOS the LIBERO backend wrote each scene file, which names every object’s start region, to a temporary folder the sandbox did not deny. It now goes into Robo Use’s cache, which the sandbox denies.
• Files a later trial loads. In a source checkout, agents could change the robot models, textures and walking policy that later trials load, and the walking policy is a torch file, which runs code when it is unpickled. The sandbox now denies them. This gap was in earlier releases.
• A fake error result. On Linux an agent could write an error event into its harness’s output and turn an unfinished, scored trial into one that runs again. Now only the harness program’s own final error event counts. This gap was also in earlier releases.

The seventh is open. The agent can end its own harness process, and a harness that exits with an error before the episode ends is an infrastructure failure, which is not scored and which --retries and --resume run again. So an agent about to fail can get another try. A fix is proposed; until it ships, look at every trial that ended with an exception or was retried before you publish scores, as Run a suite says.

Model harnesses run under the operating system’s sandbox, Seatbelt on macOS and bubblewrap on Linux. It hides the task folder, the installed package and your credentials from the agent, but it is a deny list, not a container. Run agents and VLAs says what it does not hide; run agents you do not trust in a disposable VM or container.

Start with a reference solution

Robo Use runs on Python 3.11 to 3.13, on macOS or Linux.

python3.12 -m venv .venv && source .venv/bin/activate
pip install 'robouse[sim]'
robouse doctor

robouse doctor checks that MuJoCo can render offscreen and reports the simulators and harnesses this machine can run. On Linux without a display, it says what to install if MuJoCo cannot render.

Then run a task’s reference solution, which needs no model and no API key:

robouse run --task arc-gravity --harness oracle --out runs/quickstart

In arc-gravity the agent sees grids of blocks before and after a rule is applied, works out the rule and moves the real blocks with a gripper. The command prints the reward and writes the trial folder under runs/quickstart/. With --harness noop it runs the no-op control, which scores 0.

To put an agent in charge, give Claude Code a token from claude setup-token and change the harness:

export CLAUDE_CODE_OAUTH_TOKEN=...
robouse run --task arc-gravity --harness claude-code --out runs/quickstart

The Quickstart walks through a trial folder, Suites says what each suite needs beyond the install, and Run agents and VLAs covers the other harnesses and models. The package is on PyPI.