Put the block in the drawer and close it

Vision variant

This is the vision version of the task. robo observe does not give object or goal positions: you get only the robot's own state (hand_pos, gripper_open, holding, examples, grid, constraint_violations) and two camera images (front, top), saved on every robo observe and printed as paths. Open the images to see the scene. robo info lists each camera with a 3x4 projection matrix P: for a world point (x, y, z), [u*w, v*w, w] = P @ [x, y, z, 1] gives its pixel (u, v) in that camera's saved image (u to the right, v down, origin at the top-left). With two cameras you can triangulate a point you see in both, or intersect a pixel's ray with a known height. Any field names in the description below that are not in your observation are hidden in this variant.

A two-finger parallel gripper hangs over a table (a floating gripper, no arm). The camera looks at the table from the front: x points to the right, y away from the camera (toward the back of the table), z up; the table top is at z = 0. The fingers close along the y axis.

Goal: Put the blue block in the open drawer, then close the drawer (within 1 cm).

What robo observe shows

• hand_pos: the gripper's tool point (between the fingertips), metres. gripper_open: 0 = closed, 1 = fully open (10 cm gap). holding: names of objects currently touching the fingers.
• objects: every movable object by name, with kind, color, pos (centre, metres), size (full x/y/z extent in its own frame), yaw_deg, half_height (current half extent along z), tilt_deg (0 = upright).
• fixtures: static things by name (pads, bowls, trays, plates, sockets, the drawer cabinet) with their positions and sizes. Pads are flat coloured squares on the table; bowls and trays have a floor_z and a rim_z.
• The drawer: opening (metres pulled out, 0 = closed, max 0.12), handle_pos (a horizontal bar along x, about 4 cm in front of the drawer face; the fingers can close around it from above if the gripper is only partly open, e.g. grip 0.2), interior_center, interior_size, wall_top_z, and the cabinet top height.

Gripper

The grip value is a finger position target, not a hold command: +1 = fully closed, -1 = fully open (10 cm gap), and values in between give a partial opening (finger gap = 10 cm * (1 - G) / 2, so 0 = half open, a 5 cm gap). There is no separate "hold" value: to keep holding an object, keep sending +1 (the fingers then squeeze it). robo move-to X Y Z --grip G applies G on every step of the move, starting with the first, so --grip 1 closes the fingers at the start of the move (they take about 15 steps to close fully); without --grip, the last grip value is kept. robo grip G --steps N holds the hand still for N steps while applying G. The same values apply to the GRIP argument of robo act.

Objects must be released (not touching the fingers) when the episode is judged.