David Minkwan Kim

PatternDex learns an embodiment-agnostic "interaction pattern" token sequence from human-object demonstrations and, combined with a target robot hand description, decodes wrist motions and contact points that fit that hand to guide a reinforcement learning policy; on 6 unseen demonstrations from the ARCTIC dataset it reaches a 92.8% mean success rate with Allegro hands versus 52.2% for the baseline ObjDex, transfers to Shadow, Paxini, and XHand hands with 84.0%, 73.5%, and 71.6% after fine-tuning alone with a frozen pattern encoder, and opens a real microwave in 7 of 10 trials.

PatternDex learns an embodiment-agnostic "interaction pattern" token sequence from human-object demonstrations and, combined with a target robot hand description, decodes wrist motions and contact points that fit that hand to guide a reinforcement learning policy; on 6 unseen demonstrations from the ARCTIC dataset it reaches a 92.8% mean success rate with Allegro hands versus 52.2% for the baseline ObjDex, transfers to Shadow, Paxini, and XHand hands with 84.0%, 73.5%, and 71.6% after fine-tuning alone with a frozen pattern encoder, and opens a real microwave in 7 of 10 trials.