No bend sensors needed for finger labels: Freebie tracks primate hands in real time with a 16 ms median latency
Synopsis
Finger-movement labels in non-human primates have long come from instrumented manipulanda fitted with bend sensors, which cap the dimensionality of measurable movement. Freebie uses several synchronized cameras and real-time pose estimation to output finger keypoints directly, reaching median end-to-end latencies of 15.8 ms at 30 FPS and 16.2 ms at 60 FPS, with a median absolute correlation of 0.96 against bend-sensor signals. EMG decoders trained on Freebie labels matched those trained on bend-sensor labels in closed-loop control, indicating that training a decoder need not require instrumenting the hand.
Interpretation
Hand-kinematics labels for non-human primates can now be produced in real time by multi-camera markerless tracking and used either to drive a virtual hand directly or as decoder training labels, without an instrumented manipulandum. Nearly all such studies previously captured kinematics with constrained, low-degree-of-freedom instruments, such as bend-sensor manipulanda that hold the hand fixed and track one or two finger groups, limiting the dimensionality of measurable movement. Across 24 of Monkey R's hand-control runs, Freebie's PIP y signal matched the concurrently recorded bend-sensor signal with a median absolute correlation of 0.96, interquartile range 0.95 to 0.97.
The system chains camera acquisition, pose inference and keypoint transmission to the experiment computer into one real-time pipeline, so an estimate leaves before the next frame arrives. Freebie runs on the improv real-time framework, splitting acquisition, inference, serial output, video saving and display into parallel actors that exchange data through a shared in-memory store. Over 690,485 frames from 32 runs, median end-to-end latency was 15.8 ms in the 14 runs at 30 FPS and 16.2 ms in the 18 runs at 60 FPS, with inference taking most of the processing time at a run-mean 14.6 ms.
With Freebie keypoints driving the virtual hand directly, five of six online metrics did not differ significantly from bend-sensor control, and path efficiency was higher under Freebie. An earlier pipeline driven by a derived index joint angle underperformed bend-sensor control on most metrics, with median success rate 0.29 versus 0.94. The comparison used 9 Freebie runs and 15 bend-sensor runs from Monkey R, with median success rate 0.97 versus 0.99 and path efficiency 0.67 versus 0.52, the latter at p = 0.005.
Perspective
The demonstrated scope is one- and two-degree-of-freedom finger target-acquisition tasks performed by two rhesus macaques with the head fixed and the arm restrained; main-text results come from Monkey R's single-camera configuration, while Monkey N used an earlier two-camera version, and 3D tracking during free behavior and functional electrical stimulation characterization is an extension demonstration.
The closed-loop decoder comparison has only four runs per condition, and Monkey N's data come from a different system version, task and rig; all closed-loop decoders mapped forearm EMG rather than cortical activity, so the sensitivity of intracortical decoders to label noise remains to be tested; end-to-end latency runs from the camera callback to packet transmission, excluding exposure, readout and transfer before the callback and the interval to the rendered display, so the delay the monkey actually experienced was likely longer.