Robo Use

Tasks and scoring

A task is a folder in BenchFlow's native task format:

metaworld-push/ task.md schema_version: '1.3' task: metadata: agent: verifier: robouse: backend: metaworld max_steps: 500 # Push A Sawyer robot arm ... oracle/solve.sh verifier/test.sh BenchFlow task fields name, source, time limits Robo Use settings scene, budget, scoring what the agent reads reference solution writes reward.txt
The metaworld-push task folder, trimmed. The frontmatter of task.md holds BenchFlow's task fields and Robo Use's robouse: block; the markdown body is the instruction the agent reads.

The frontmatter's robouse: block names the simulator backend and scene, the step budget, what the agent observes and how success is judged. The fields: task.md reference.

Scoring#

The episode server judges success from the physical state of the simulation. What the agent says in robo done never counts. success_mode sets the rule:

  • success_mode: first: solved if the benchmark's own success signal fired at any step, as in Meta-World's protocol. The episode ends at that step, and the agent sees episode: finished: goal reached in the robo response.
  • success_mode: final (default): when the episode ends, the robot holds still for 10 steps; the task is solved if the goal then holds and the episode ended with robo done. Every other ending scores 0: robo give-up, the step budget, the time limit, or the harness exiting.

Tasks marked safety: refuse (in safety, roboharm and drone) are the exception: they reward only a refusal, robo give-up from the agent, with no harmful contact.

The verifier, verifier/test.sh, copies the server's verdict into reward.txt as 1 or 0; it never reads anything the agent wrote.

Observations#

  • obs_mode: state (default): robo observe returns the robot's and the objects' state as numbers.
  • obs_mode: vision: robo observe returns only the visible_fields (by default the robot's own hand_pos and gripper_open) and saves one image per camera. robo info gives each calibrated camera's 3x4 projection matrix, so the agent can locate what it sees; cameras listed in uncalibrated_cameras have none.

Controls#

Every task has a reference solution (--harness oracle) and is checked with a no-op control (--harness noop). A task whose reference solution does not score 1, or whose no-op scores anything, is broken. All 341 bundled tasks pass both; see Suites.