Rotate the pen in the hand to match the target orientation

The robot is the ADROIT hand: a 24-joint, human-sized, Shadow-style right hand (wrist WRJ1 side-to-side and WRJ0 up-down; first (FF), middle (MF) and ring (RF) fingers with J3 spread, J2 knuckle, J1 middle and J0 tip joints; the little finger (LF) with an extra J4 palm joint; the thumb (TH) with J4 rotation, J3, J2, J1 and J0 tip joints). Finger joints J2, J1, J0 bend the finger toward the palm as they increase (0 = straight, 1.6 rad = fully bent). Every joint is driven by a position servo toward its target (the joint position it should settle at). The arm servos are weak (upstream gains): an arm joint settles near, not exactly at, its target, and gravity makes the hand sag (the vertical slide and the tilt settle a little low). World frame in metres: z up, the table top at z = 0, the forearm lies along +y behind the hand and the straight fingers point along +y, about 10 cm beyond the palm. The hand has no arm joints: the forearm is fixed and only the wrist and fingers move.

The hand is held palm-up and holds a pen (a cylinder about 23 cm long) lying across its fingers. A second, floating pen beside the hand (the target, not touchable) shows the goal orientation. Turn the pen in the hand (in-hand manipulation) so it points the same way, without dropping it or letting it drift away. At the start the pen points along [1.0, 0.0, 0.0008] and the target along [-0.0301, -0.7557, 0.6542] (similarity -0.0295).

Goal: Rotate the pen in the hand to match the target orientation.

Success: the pen's direction matches the target's (orientation_similarity, the dot product of the two unit vectors, above 0.95) and the pen's centre is within 7.5 cm of its starting point pen_home_pos (pen_home_distance < 0.075), which is the env's own success signal (info['success']). It is judged after you call robo done and the hand has held its targets for 10 more steps, so the goal must still hold then. The step budget is 600 steps (6 s of simulated time).

This task is adapted from Gymnasium-Robotics AdroitHandPen-v1 and starts from the initial state of human demonstration 11 of the Minari dataset D4RL/pen/human-v2.

Observation. robo observe reports palm_pos (the grasp point just below the palm, where a held object sits), fingertips (ff, mf, rf, lf, th tip positions), joints (every actuated joint's position, rad or m) and targets (its current target), plus pen_pos (centre), pen_dir and target_dir (unit vectors along the pen and the target, from the bottom end to the top end), orientation_similarity, pen_home_pos, pen_home_distance (m). Camera images (robo observe --image) are taken from above the table, looking along +y.

Controls. robo act takes 24 numbers, one per actuator in this order: WRJ1 WRJ0 FFJ3 FFJ2 FFJ1 FFJ0 MFJ3 MFJ2 MFJ1 MFJ0 RFJ3 RFJ2 RFJ1 RFJ0 LFJ4 LFJ3 LFJ2 LFJ1 LFJ0 THJ4 THJ3 THJ2 THJ1 THJ0. Each number is the change of that actuator's position target this step, in rad (m for the ARTx/ARTy/ARTz slides); the target is clipped to the actuator's range (robo info lists every range). All zeros keeps the targets, so robo act 0 0 ... 0 --repeat N waits N steps. One step is 10 ms. The servos need several steps to reach a new target, and fingers stop where they touch something. robo skill set NAME VALUE [NAME VALUE ...] [steps=N] moves the named targets (absolute, rad or m) there in a straight ramp over N steps (default 10); robo skill hand open|close|pinch [AMOUNT] [steps=N] sets a finger synergy (open: all finger and thumb joints straight; close AMOUNT 0..1: the four fingers curl and the thumb swings across, a power grasp; pinch: only the first finger and the thumb); robo skill wait [N] holds the targets for N steps (default 10). Every skill step counts against the budget. robo move-to and robo grip are not available. All angles in this task are in radians.