Rule inference: the marker

A two-finger parallel gripper hangs over a table (a floating gripper, no arm). The camera looks at the table from the front: x points to the right, y away from the camera (toward the back of the table), z up; the table top is at z = 0. The fingers close along the y axis.

Goal: Coloured blocks stand on a grid of cells marked on the table. The observation field examples holds demonstrations of a hidden rule: each shows an arrangement of coloured blocks on a grid of the same shape before and after the rule was applied (cells are [row, col], row 0 = back row, col 0 = left column). Work out the rule from the examples and apply it to the blocks in front of you by moving them. Some examples mention extra items (spare blocks off the grid, pads, a marker, block sizes); these correspond to the same kinds of items in your scene. Blocks of the same colour are interchangeable. Only the final arrangement is judged; blocks must be released and resting on the table.

The four grey pads in front of the grid are numbered 1, 2, 3, 4 from left to right (pad 1 is the leftmost).

What robo observe shows

• hand_pos: the gripper's tool point (between the fingertips), metres. gripper_open: 0 = closed, 1 = fully open (10 cm gap). holding: names of objects currently touching the fingers.
• objects: every movable object by name, with kind, color, pos (centre, metres), size (full x/y/z extent in its own frame), yaw_deg, half_height (current half extent along z), tilt_deg (0 = upright), and cell ([row, col] or null when off the grid).
• fixtures: static things by name (pads, bowls, trays, plates, sockets, the drawer cabinet) with their positions and sizes. Pads are flat coloured squares on the table; bowls and trays have a floor_z and a rim_z.
• grid: rows, cols, cell_size and the centre of every cell (cell_centers[row][col]). Row 0 is the back row, col 0 the left column.
• examples: the before/after demonstrations described above.

Gripper

The grip value is a finger position target, not a hold command: +1 = fully closed, -1 = fully open (10 cm gap), and values in between give a partial opening (finger gap = 10 cm * (1 - G) / 2, so 0 = half open, a 5 cm gap). There is no separate "hold" value: to keep holding an object, keep sending +1 (the fingers then squeeze it). robo move-to X Y Z --grip G applies G on every step of the move, starting with the first, so --grip 1 closes the fingers at the start of the move (they take about 15 steps to close fully); without --grip, the last grip value is kept. robo grip G --steps N holds the hand still for N steps while applying G. The same values apply to the GRIP argument of robo act.

Objects must be released (not touching the fingers) when the episode is judged.