Soft grasping gets a six-modality dataset: frozen encoders reach 94.59% post-hoc grasp-outcome accuracy on an independent trial set
Synopsis
Soft robotic grasping lacks multimodal data that coordinates vision, depth, touch, inertial measurement and text, and existing alignment methods apply one strength to every modality pair. SomaAlign first estimates each modality's relative reliability from masked reconstruction, then uses it to set a pair-specific contrastive temperature, and in a third stage tightens intra-sample cross-modal consistency with conflict-aware constraints. Across 30 directed cross-modal retrieval tasks on the Soma dataset the mean Recall@10 is 90.66%, and on the independently collected Soma-Success set, excluded from representation learning, frozen encoders with lightweight probes reach 94.59% post-hoc grasp-outcome classification accuracy.
Interpretation
Multimodal representation learning for soft grasping now comes with a matched data and alignment pipeline: a six-modality grasp dataset aligned by acquisition cycle, and a training framework that scales alignment strength by modality reliability. Earlier visuo-tactile datasets came largely from rigid end-effectors or specific optical tactile sensors, where contact deformation and noise distributions are more constrained, and none paired them with an independently collected grasp-outcome set for testing frozen-encoder reuse. Soma holds 100 objects, 1,000 continuous grasping trajectories and 74,914 cycle-aligned samples spanning remote RGB, hand RGB, depth, tactile pressure, IMU and structured text; Soma-Success independently collected 1,000 trials, retaining 998 after quality control, of which 600 succeeded and 398 failed.
Alignment strength is no longer uniform across modality pairs, so channels such as touch and inertial sensing that are prone to vibration are allowed to align more loosely. Conventional multimodal contrastive learning applies one temperature to all modality pairs, which can over-constrain noisy pairs while underusing more stable visual observations. On a fixed 500-sample test subset, SomaAlign reaches Recall@1, Recall@5 and Recall@10 of 63.04%, 85.55% and 90.66%, against 42.00%, 60.10% and 65.77% for Pairwise InfoNCE and 44.55%, 64.30% and 70.55% for Hub Alignment, which uses remote RGB as the hub, with the same initialization and training budget.
Training proceeds in three stages: estimate reliability, modulate contrastive temperature by that reliability, then refine locally with a conflict-aware geometry-preserving term. Reliability comes from masked-reconstruction difficulty rather than direct sensor-noise annotation; a modality that reconstructs less stably is treated as less reliable. Each pair's temperature is a monotonic mapping of that pair's composite reliability score, so reliable pairs get smaller temperatures and sharper similarity distributions; the third stage anchors to frozen Stage-II reference centers and applies protection only when center drift exceeds a scale-adaptive tolerance.
The learned representations transfer to grasp-outcome judgment on data never used in training, and only the modality encoders are kept for inference. Soma-Success is fully excluded from representation learning, and probes are cross-validated in five folds grouped by object identity, so each trial is predicted by a probe that never saw that object. Frozen encoders with an MLP probe reach 94.59% post-hoc classification accuracy and a linear probe 93.18%; using only pre-grasp and mid-grasp snapshots for early prediction, the linear probe reaches 83.07% and the MLP probe 84.07%, while Hub Alignment's MLP reaches 86.48%.
Perspective
The conclusions rest on two pneumatic soft-gripper configurations, 100 everyday objects and templated structured text, with downstream reuse assessed by frozen encoders plus lightweight supervised probes on Soma-Success.
The reliability score is a proxy for reconstruction difficulty, not a direct measurement of physical sensor noise; structured text annotations are semantically consistent but linguistically limited; Soma-Success shows frozen-encoder reuse with lightweight supervised probes rather than zero-shot generalization, and early-prediction performance also varies with probe architecture.