Count with the wrist camera

A two-finger parallel gripper hangs over a table (a floating gripper, no arm; it cannot rotate). x points to the right as seen from the front of the table, y away from the front (toward the back of the table), z up; the table top is at z = 0 and spans x from -0.35 to 0.35 and y from -0.30 to 0.30. The gripper's tool point can reach x in [-0.34, 0.34], y in [-0.27, 0.28], z in [0.012, 0.35]. The fingers close along the y axis; the fully open gap is 10 cm.

Goal: Coloured blocks are scattered over the table. Count them (do not count the black marker cube or the numbered pads) and put the black marker cube onto the grey pad whose number equals the count. The six numbered pads lie in a row along the front edge of the table: pad 1 is centred at x = -0.25, pad 2 at x = -0.15, pad 3 at x = -0.05, pad 4 at x = 0.05, pad 5 at x = 0.15, pad 6 at x = 0.25, all at y = -0.235, each 6 cm square. The marker starts at x = 0.0, y = 0.0. Leave the coloured blocks where they are (each must stay within 2 cm of where it started).

What you can observe

robo observe does not give object or goal positions. It returns only hand_pos, gripper_open, constraint_violations and saves one image per camera (wrist; 320 x 320 pixels), printing the paths. Open the images to see the scene. hand_pos is the gripper's tool point (between the fingertips) in metres, gripper_open is 0 (closed) to 1 (10 cm gap) (there is no contact sensor that says what the fingers hold: a finger opening above zero after closing means something is between them, and the images show what), and constraint_violations lists broken rules (any entry means the task has failed).

The only camera is wrist, an eye-in-hand camera fixed to the gripper and moving with it. It looks straight down; in its image, right is +x and up is +y (the back of the table). It is mounted to one side of the fingers, but its exact offset from the tool point is NOT given, and no projection matrix is given (robo info gives only its image size and field of view). You know hand_pos, and the task description gives the positions of some fixed things, so you can work out how image positions relate to world positions by moving the gripper and watching how the view changes. The higher the gripper, the more of the table you see. There is no other camera.

Gripper

The grip value is a finger position target, not a hold command: +1 = fully closed, -1 = fully open (10 cm gap), and values in between give a partial opening (finger gap = 10 cm * (1 - G) / 2, so 0 = half open, a 5 cm gap). There is no separate "hold" value: to keep holding an object, keep sending +1 (the fingers then squeeze it). robo move-to X Y Z --grip G applies G on every step of the move, starting with the first, so --grip 1 closes the fingers at the start of the move (they take about 15 steps to close fully); without --grip, the last grip value is kept. robo grip G --steps N holds the hand still for N steps while applying G. The same values apply to the GRIP argument of robo act.

Task rules

• Step budget: 550 simulated steps (one step = 20 ms). Wall-clock limit: 20 minutes.
• Objects must be released (not touching the fingers) when the episode is judged, about 10 steps after robo done.