TactileLab computes shear as tactile flow in rigid-body simulation, letting dexterous hands learn object rotation in sim and keep turning it on real hardware

Tactile simulation gains a tangential observation that needs no soft-body solve: every tactile pixel carries a two-dimensional in-plane displacement vector, spatially aligned with the contact depth image to form the policy's high-dimensional tactile input. Rigid-body tactile simulators previously output mainly depth-like signals, suited to normal contact but unable to separate contact states with similar depth and different tangential motion; soft-body simulators captured deformation but had low throughput for large-scale reinforcement learning. In dual-arm object following, a policy using tactile flow alone outperformed contact depth alone, and combining depth with flow sped up learning while reaching final performance similar to flow alone; in in-hand rotation, depth plus flow achieved the longest time-to-fail and lowest fall rate, better than either modality alone both within the training distribution and on out-of-distribution objects.

The pipeline translates pairs of real tactile images into simulated tactile flow and depth fields, so a policy trained only in simulation can read real sensor streams directly. Earlier real-to-sim transfer handled single-frame static contact-depth reconstruction and did not place tangential motion between consecutive frames into the same observation space. Translation was validated on six camera-based tactile sensors: markerless GelSight Mini, DIGIT, 9DTact, GelSight Mini with markers, and flat and hemispherical TacTip; markerless GelSight Mini translational error fell from 1.79 to 0.34 and markerless 9DTact from 0.45 to 0.35, while marker-based sensors were already accurate and changed less after translation.

The PETS-Net policy network feeds positionally encoded joint-angle and action histories together with spatially encoded tactile image histories into a temporal encoder that estimates the teacher policy's latent variable. A standard adaptation module may have limited representational capacity when its input is a long-horizon proprioceptive history plus high-dimensional tactile histories. In ablation, every variant with positional encoding outperformed the version without it, and among temporal encoders the temporal convolutional network reached the highest final reward, with performance largely saturating beyond a 10-frame history.

The demonstrated scope is two contact-rich tasks: dual-arm object following, and in-hand rotation with the Allegro Hand and LEAP Hand; the real-robot rotation experiments use a LEAP Hand with four vision-based tactile fingertips, evaluate each object category from 10 initial grasp configurations, and cap each trial at 30 seconds. Real-to-sim translation was validated on six camera-based tactile sensors, and the hemispherical TacTip was chosen for the downstream real-robot tasks.

Real-to-sim translation assumes the elastomer stays in sticking contact with the object so deformation is dominated by shear; external disturbances or abrupt contact changes can violate this and lead to unrecoverable states, and how to include slip detection in tactile simulation remains an open problem. Tactile flow itself is not restricted to rigid-body simulation and could be derived from surface-point motion in soft-body simulation. Extending to more object shapes will require larger high-dimensional real-to-sim datasets and geometry-aware transfer models.