With two dexterous hands, press an iPad's passcode 1-2-3 without touching a wrong digit

Two Franka Emika Panda arms stand side by side at a table (table top about 0.93-1.0 m above the floor, raised by a random amount per scene), each with a four-fingered Allegro robot hand (16 joints) mounted on its flange instead of a gripper. The hands are about 25 cm long and 10 cm wide; their fingers are index, middle, ring and thumb. World frame in metres, z up.

Goal: An iPad stands in a stand on the table, its lock screen facing the robots and showing a keypad of digit buttons. Enter the passcode 1, 2, 3 by pressing those digit buttons in that order with the fingertips.

Success: The three digits are entered in order and the screen unlocks, then stays unlocked for 10 steps. A press is a new fingertip contact with a button. Failure: touching any wrong digit button at any time ends any chance of success (task_failed appears), so keep the hands clear of the other buttons. The episode ends as solved the moment this condition is met; robo done before that scores 0.

Task fields in robo observe: ipad_*, stand_*, button_0 .. button_9 (button centre positions; buttons hide once unlocked), digits_entered and unlocked.

Controls. robo act takes 44 numbers: the right arm's 22 (R_DX .. R_F15), then the left arm's 22 (L_DX .. L_F15). Per arm: DX DY DZ move the commanded hand pose by 2 cm per unit along world x, y, z (up to +-8 units = 16 cm per step); RX RY RZ rotate the commanded hand orientation by 0.2 rad per unit about the world x, y, z axes (up to +-3); F0 .. F15 change the commanded finger joint angles by 0.3 rad per unit (up to +-6): F0-F3 index, F4-F7 middle, F8-F11 ring, F12-F15 thumb. In the left hand, DexJoCo orders the three fingers the other way round: L_F0-L_F3 drive the joints DexJoCo names ring, L_F4-L_F7 middle, L_F8-L_F11 index (thumb stays L_F12-L_F15); take a wrist-camera picture to see which physical finger moves. For index, middle and ring, joint 0 spreads the finger sideways and joints 1-3 curl it (positive = flex toward the palm); for the thumb, joint 0 swings it across the palm (opposition) and joints 1-3 rotate and curl it. Commands are targets: the arm's operational-space controller and the finger position servos track them, so the hand lags a fast command and stops where contact blocks it. robo info lists each joint's range (the commanded angle is clipped to it). One step is 20 ms of simulated time. Actions are deltas, so --repeat N applies the same change N times. robo move-to X Y Z moves the the active arm's; the right arm is active at the start and any robo act that moves only one arm makes that arm active (active_arm in robo observe) hand (the point hand_pos) toward a point without changing its orientation or fingers, and robo grip G sets the the active arm's; the right arm is active at the start and any robo act that moves only one arm makes that arm active (active_arm in robo observe) hand to a generic power grasp (G > 0 closes to fraction G of it, G < 0 opens all fingers, 0 keeps them); the grasp preset does not suit every object, so finger-level robo act control is often needed. The DX DY DZ GRIP form in the general instructions below does not apply: every robo act needs all 44 numbers.

Observation. robo observe reports for each arm, prefixed right_ / left_: hand_pos and hand_quat_wxyz (measured flange position and orientation, quaternion w, x, y, z), hand_target_pos and hand_target_quat_wxyz (the commanded pose), finger_joints and finger_joint_targets (16 measured and commanded joint angles in radians, in F0..F15 order); for objects, <name>_pos (body origin), <name>_quat_wxyz, <name>_tilt_deg (angle of the body's z axis from vertical) and <name>_yaw_deg; plus the task fields listed above. robo observe --image saves a picture from the back camera; --camera picks another (back, handcam_rgb_right, handcam_rgb_left). The hand's orientation matters: at the start the palm faces down; look at a picture before you reach.

The step budget is 3000 steps (60 s of simulated time). A human teleoperator needed 336 steps for this scene.