benchflow/robouse-menagerie
Two-arm tasks for a dual Franka Panda and ALOHA 2.
bench eval run \
-d benchflow/robouse-menagerie@0.1 \
--registry https://robouse.ai/hub/registry.json \
--agent oracleAbout this dataset
- Embodiment
- Dual Franka Panda, ALOHA 2
- Simulator
- MuJoCo Menagerie models
- Requirements
- CPU only · 1.2 GB image ·
bench>=0.7.4,<0.8 - Licence
- Task packages Apache-2.0
@misc{robouse_benchflow_robouse_menagerie_0_1,
title = {Robo Use benchflow/robouse-menagerie 0.1},
author = {{BenchFlow}},
year = {2026},
howpublished = {\url{https://robouse.ai/hub/benchflow/robouse-menagerie/}},
note = {BenchFlow dataset, tag benchflow/robouse-menagerie-v0.1, commit 9e672aa1e7f0}
}15 tasks
Embodiment
Task type
Outcome
Size
Step budget
menagerie-aloha-clutter-retrievalIn the right bowl a red cylinder (target, 2.4 cm across, 2.8 cm tall) stands in a ring of four violet cylinders of the same size (neighbour_0 ..…In the right bowl a red cylinder (target, 2.4 cm across, 2.8 cm tall) stands in a ring of four violet cylinders of the same size (neighbour_0 .. neighbour_3, 4.5 cm from it, centre to centre). Lift the red cylinder out without disturbing the violet ones and set it upright in the yellow tray in the middle of the desk.
ALOHA 2800 stepsjudged on the final state
menagerie-aloha-gentle-insertionAn orange peg (a rod 1.6 cm across and 7.5 cm long with a dark collar at its top end) stands in the left bowl. Seat it upright in the square…An orange peg (a rod 1.6 cm across and 7.5 cm long with a dark collar at its top end) stands in the left bowl. Seat it upright in the square socket in the middle of the desk (aperture 2.3 cm square, 3 cm deep) without ramming it: the contact force between the peg and the socket is monitored during the whole episode.
ALOHA 21,200 stepsjudged on the final state
menagerie-aloha-opening-fourFour chess pieces start in the two bowls: a light pawn and rook in the left bowl, a dark pawn and rook in the right bowl. Every piece has its own…Four chess pieces start in the two bowls: a light pawn and rook in the left bowl, a dark pawn and rook in the right bowl. Every piece has its own target square outlined in gold on the chessboard (<piece>_target). Place all four upright on their squares, using both arms: each arm must place at least one piece.
ALOHA 22,000 stepsjudged on the final state
menagerie-aloha-pattern-heldoutA 5 x 5 grid of 4 cm cells is marked on a board in the middle of the desk. Three tokens with coloured crowns start in the bowls: red_circle and…A 5 x 5 grid of 4 cm cells is marked on a board in the middle of the desk. Three tokens with coloured crowns start in the bowls: red_circle and green_triangle in the left bowl, blue_square in the right bowl (each 6 cm tall with a 2.6 cm base disc). examples shows how a hidden rule maps an input arrangement of the three labels on the grid to an output arrangement; query is a new input arrangement. Work out the rule (here a composition of several operations), apply it to query, and place each token upright on the cell where the rule sends its label. Two violet cylinders just outside the grid must not be disturbed.
ALOHA 22,000 stepsjudged on the final state
menagerie-aloha-pawn-landingA light chess pawn (6 cm tall; base disc 2.6 cm across, a 1.8 cm stem, round crown) stands in the right bowl (a shallow dish 28 cm across with a…A light chess pawn (6 cm tall; base disc 2.6 cm across, a 1.8 cm stem, round crown) stands in the right bowl (a shallow dish 28 cm across with a 3 cm rim). Move it onto the square outlined in gold on the chessboard, standing upright, and let go.
ALOHA 2600 stepsjudged on the final state
menagerie-aloha-pawn-landing-visionA light chess pawn (6 cm tall; base disc 2.6 cm across, a 1.8 cm stem, round crown) stands in the left bowl (a shallow dish 28 cm across with a 3…A light chess pawn (6 cm tall; base disc 2.6 cm across, a 1.8 cm stem, round crown) stands in the left bowl (a shallow dish 28 cm across with a 3 cm rim). Move it onto the square outlined in gold on the chessboard, standing upright, and let go. Where the pawn is is not given as numbers: find it in the camera images.
ALOHA 2600 stepsjudged on the final statevision-only
menagerie-panda-clutter-retrievalIn the left bowl a red cylinder (target, 2.4 cm across, 2.8 cm tall) stands in a ring of four violet cylinders of the same size (neighbour_0 ..…In the left bowl a red cylinder (target, 2.4 cm across, 2.8 cm tall) stands in a ring of four violet cylinders of the same size (neighbour_0 .. neighbour_3, 4.5 cm from it, centre to centre). Lift the red cylinder out without disturbing the violet ones and set it upright in the yellow tray in the middle of the desk.
Dual Franka Panda800 stepsjudged on the final state
menagerie-panda-clutter-retrieval-visionIn the left bowl a red cylinder (2.4 cm across, 2.8 cm tall) stands in a ring of four violet cylinders of the same size, 4.5 cm from it, centre…In the left bowl a red cylinder (2.4 cm across, 2.8 cm tall) stands in a ring of four violet cylinders of the same size, 4.5 cm from it, centre to centre. Lift the red cylinder out without disturbing the violet ones and set it upright in the yellow tray in the middle of the desk. Where the cylinders are is not given as numbers: find them in the camera images.
Dual Franka Panda800 stepsjudged on the final statevision-only
menagerie-panda-fragile-disposalTwo items stand in the left bowl: a glass vial (7 cm tall, 2.4 cm base, cap on top) and a pawn-shaped metal blank (6 cm tall). The blank is scrap…Two items stand in the left bowl: a glass vial (7 cm tall, 2.4 cm base, cap on top) and a pawn-shaped metal blank (6 cm tall). The blank is scrap and is authorised for disposal: put it in the grey bin at the front of the desk (dropping it in is fine). The vial is fragile: set it down upright on the square outlined in gold on the board. It must never fall freely (whenever no gripper touches it, it must not move downward faster than 0.25 m/s, which rules out releasing it more than about 3 mm above a surface) and it must never go into the bin.
Dual Franka Panda1,000 stepsjudged on the final state
menagerie-panda-fragile-kitThree identical glass vials (vial_0 .. vial_2, 7 cm tall, 2.4 cm base, a 2 cm body up to 4.6 cm, then a thin neck and a cap) stand in the bowls:…Three identical glass vials (vial_0 .. vial_2, 7 cm tall, 2.4 cm base, a 2 cm body up to 4.6 cm, then a thin neck and a cap) stand in the bowls: two in the left bowl, one in the right. Each has a coloured label (red, blue or green) on its underside, hidden against the bowl floor; nothing else tells them apart. A black inspection station in front of the board has a camera looking straight up (--camera inspect). Show each vial's underside to that camera, read its label, and pack the vial upright in the compartment of the same colour on the kit tray (red, blue, green from left to right). Two violet ampoules behind the tray must not be disturbed.
Dual Franka Panda2,500 stepsjudged on the final statevision-only
menagerie-panda-keyed-insertionAn orange keyed peg stands in the right bowl, turned at an angle: its bottom 4 cm is a rectangular shaft 18 x 12 mm, above that a round handle…An orange keyed peg stands in the right bowl, turned at an angle: its bottom 4 cm is a rectangular shaft 18 x 12 mm, above that a round handle (1.4 cm across) up to 7 cm and a ball cap (2 cm) on top, 8.5 cm tall in all. Seat the shaft fully in the rectangular socket in the middle of the board (aperture 21 x 15 mm, long side along world x, 35 mm deep), gently. The shaft only fits when it is square to the aperture (peg yaw near 0 or 180 degrees).
Dual Franka Panda1,500 stepsjudged on the final state
menagerie-panda-pattern-apprenticeA 5 x 5 grid of 4 cm cells is marked on a board in the middle of the desk. Three tokens with coloured crowns start in the bowls: red_circle and…A 5 x 5 grid of 4 cm cells is marked on a board in the middle of the desk. Three tokens with coloured crowns start in the bowls: red_circle and green_triangle in the left bowl, blue_square in the right bowl (each 6 cm tall with a 2.6 cm base disc). examples shows how a hidden rule maps an input arrangement of the three labels on the grid to an output arrangement; query is a new input arrangement. Work out the rule, apply it to query, and place each token upright on the cell where the rule sends its label. Two violet cylinders just outside the grid must not be disturbed.
Dual Franka Panda2,000 stepsjudged on the final state
menagerie-panda-precise-landingAn orange block (2.2 x 2.2 x 5 cm, 30 g, standing upright) is in the left bowl (the bowls are shallow dishes 28 cm across with a 3 cm rim). Carry…An orange block (2.2 x 2.2 x 5 cm, 30 g, standing upright) is in the left bowl (the bowls are shallow dishes 28 cm across with a 3 cm rim). Carry it onto the square outlined in gold on the chessboard, stand it upright and let go.
Dual Franka Panda600 stepsjudged on the final state
menagerie-panda-route-aroundA pawn-shaped payload (6 cm tall, 2.6 cm base) stands on the teal source pad on the left. Move it to the teal delivery pad and stand it upright.…A pawn-shaped payload (6 cm tall, 2.6 cm base) stands on the teal source pad on the left. Move it to the teal delivery pad and stand it upright. A tall red column (6 cm across, 60 cm tall, too tall to pass over) stands between the two pads. It is protected: neither arm, nor the payload, may touch it at any time.
Dual Franka Panda1,000 stepsjudged on the final state
menagerie-panda-two-bowls-to-boardEight chess pieces start in the two bowls: four light pieces in the left bowl (two pawns, a rook, a bishop) and four dark ones in the right bowl,…Eight chess pieces start in the two bowls: four light pieces in the left bowl (two pawns, a rook, a bishop) and four dark ones in the right bowl, 5.2 cm apart. Every piece has its own target square on the chessboard, outlined in gold (<piece>_target). Place all eight upright on their squares. The two arms share one clock, so the two halves of the job can run at the same time.
Dual Franka Panda3,000 stepsjudged on the final state
| Model | Result | Steps | Time | Trial |
|---|---|---|---|---|
| GPT-6 Astra · Codex | Not solved | 3,000 | 13 min | Trial |
| Kimi K3 · Claude Code | Not solved | 3,000 | 12 min | Trial |
| Kimi K3 · mini-swe-agent | Not solved | 2,996 | 16 min | Trial |
| GLM-5.3 · mini-swe-agent | Not solved | 1,083 | 14 min | Trial |
| GLM-5.3 · Claude Code | Not solved | 482 | 60 min | Trial |
| GLM-5.3 · Codex | Not solved | 2,600 | 61 min | Trial |
↑ ↓ move between rows · Enter shows details · O opens the task
Each task is a package in the task format, pinned to robohub commit 9e672aa1e7f0. The verifier and the reference solution are not published.
Agent harnesses
| Rank# | Model · harness | Solve rate | Solved | Tasks run |
|---|---|---|---|---|
| 1 | GPT-6 Astra · Codex | 93% | 14 of 15 | 15 of 15 |
| 2 | Kimi K3 · mini-swe-agent | 80% | 12 of 15 | 15 of 15 |
| 3 | Kimi K3 · Claude Code | 73% | 11 of 15 | 15 of 15 |
| 4 | GLM-5.3 · Claude Code | 67% | 10 of 15 | 15 of 15 |
| 5 | GLM-5.3 · Codex | 67% | 8 of 12 | 15 of 15 · 3 infra |
| 6 | GLM-5.3 · mini-swe-agent | 53% | 8 of 15 | 15 of 15 |
Per task
✓ solved ✗ not solved ! infrastructure failure · not run