A humanoid face stops following fixed gaze rules: a Transformer-VAE cuts seven-joint error below scripted controllers on a two-second benchmark

Gaze on a humanoid face robot is no longer driven joint by joint by fixed rules; a learned action model generates full-head motion from a target eye trajectory. Earlier eye and head controllers relied on hand-scripted or bio-inspired rules, which had limited flexibility across targets and motion styles. On a two-second closed-loop benchmark where every controller receives the same four-dimensional ground-truth eye condition, RGT-Base's seven-joint error is lower than three rule-based controllers and the RNN, VAE-CNN, and Flow Matching learned baselines.

The model separates where to look from how to move: an upstream vision-language model asynchronously selects the referent, and the action head only realizes the target trajectory as lifelike motion. The vision-language model runs below the control rate, so another module must sustain the fixation between its calls. The vision-language model returns a structured referent, FastSAM refines it into a frame-rate fixation projected into the eyeball frame, and an eyeball-trajectory generator unrolls it into the conditioning signal.

The action head is a Transformer-VAE with a latent bottleneck, fine-tuned autoregressively to expose it to the distribution shift caused by its own predictions. A model trained only on teacher-forced reconstruction feeds generated actions back into its history in closed loop, and errors accumulate. Flow Matching has the lowest error under single-step sliding-window evaluation but higher closed-loop error than RGT, a reversal consistent with autoregressive fine-tuning on self-generated histories.

The demonstrated scope is robot-space kinematic fidelity and real-time inference: on a two-second closed-loop benchmark where every controller receives the same ground-truth eye target, RGT-Base reaches the lowest reported seven-joint error, with inference inside the 16.7 ms frame interval. Physical demonstrations show the targeting and control components running together. It suits researchers and engineers working on gaze generation and closed-loop motion generation in human-robot interaction.

The benchmark gives every controller the same ground-truth eye target, so it does not measure whether the vision-language model picks the intended referent or whether the reactive module maintains fixation under occlusion. The error table is one stochastic run without recorded evaluation seeds or confidence intervals; MSE measures joint-space agreement and APD measures stochastic diversity, and neither is an observer rating. The six-participant, non-clinical sample and markerless webcam retargeting constrain generalization to atypical ocular kinematics, different gaze conventions, and other robot joint layouts, and the scored rollouts last two seconds.