No tactile glove required: monocular video yields contact, proximity and 3D force, lifting HOCap contact Macro-F1 from 18.0 to 50.5

Hand-object, hand-hand and hand-scene interactions can now be described by one set of dense hand-surface fields, with contact, proximity and 3D vertex force placed on corresponding MANO hand-model vertices from monocular video alone. Earlier work covered only parts of this set: HACO predicted dense contact, InterField estimated hand-object distance, and HOPE and EgoPHI predicted contact with normal pressure or force from images. On HOCap, excluded from training, the temporal variant reaches a contact Macro-F1 of 50.5 against 18.0 for HACO, and its contact Macro-F1 exceeds HACO on DexYCB, HOCap and ARCTIC.

The model encodes a video into per-frame features, a temporal transformer aggregates them across frames, and three learned queries decode contact, proximity and force for both hands at each 10 Hz timestamp. Force is expressed in local tangential and normal components on the hand surface and then analytically aggregated into an object-level wrench, which reduces sensitivity to camera orientation. Training draws on 24 datasets and roughly 50 million frames spanning hand-object, hand-hand and hand-scene interaction; contact and proximity labels come from geometry, and force labels come from object motion for 15 hand-object datasets.

A differentiable rigid-body dynamics layer aggregates predicted vertex forces into an object-level wrench and integrates object motion forward, so motion error propagates back to the force predictions. Object motion constrains only the net force and torque and cannot uniquely determine how force is distributed over the hand surface, so force labels are regularized by geometric, frictional and temporal constraints before supervision. On the DexYCB test split, enabling physics supervision for the temporal variant lowers rollout position error from 5.7 cm to 3.3 cm, rollout angle error from 45.2 to 29.4 degrees, and active-force MAE from 126.4 to 122.7 mN.

Evaluation covers contact and proximity error on HOCap, ARCTIC, DexYCB and MOW, force and rollout ablations on the DexYCB test split, and reconstruction gains for EasyHOI and ArtHOI on DexYCB, HOCap and MOW; HOCap is excluded from training and used for cross-dataset transfer.

Geometry-derived contact is a proxy sensitive to source geometry and alignment, and motion-derived vertex forces are non-unique estimates rather than measurements that depend on mass, friction and environmental-support assumptions; motion-level evaluation is limited to short-horizon, reference-assisted rigid-body rollouts.