Sign language generation splits into two steps: SRVQ picks what to sign, lifting retrieval R@1 to 82.31 on CSL-Daily
Synopsis
Sign language generation turns written sentences into continuous signing motion, which requires deciding which signs to express and then realizing handshape, trajectory and body coordination. Existing pose-based systems and reconstruction-only motion tokenizers entangle those two decisions, leaving the generator without a text-aligned intermediate target. SRVQ builds a hierarchy inside a single residual quantizer: the first codebook is supervised by text and gloss to encode sign content, later residual levels add kinematic detail, and a semantic generator followed by a kinematic generator predicts the codes. On CSL-Daily and Phoenix-2014T the full variant reaches retrieval R@1 of 82.31 and 70.25, versus 51.96 and 24.14 for SOKE, gains of 30.35 and 46.11 points.
Interpretation
Sign language generation is split into two levels, semantics first and kinematics second: the coarsest discrete codes carry which sign content to express, and later residual levels carry handshape, trajectory and body coordination. Earlier pose-based systems and reconstruction-only motion tokenizers learned semantic selection and motion realization within a shared representation, so the generator had no explicit text-aligned intermediate target. On CSL-Daily and Phoenix-2014T, the full variant that predicts both semantic and residual codes reaches retrieval R@1 of 82.31 and 70.25, against 51.96 and 24.14 for SOKE, gains of 30.35 and 46.11 points.
Semantics and kinematic detail share one residual quantizer, without a separate semantic representation or an external planning stage. Prior designs relied on separate semantic spaces, staged training or additional codebooks that are hard to coordinate with reconstruction. The quantizer encodes body, left-hand and right-hand streams separately with 6 quantization levels and code dimension 512; the first codebook is supervised by a text contrastive loss and by cropped segments from frame-level gloss annotations, while later levels carry only reconstruction and velocity-smoothness losses.
Generation uses word-aware conditioning: a semantic generator predicts the first-level semantic codes, a kinematic generator completes the residual codes, and both cross-attend to word-token embeddings augmented with a global sentence token. Earlier systems mostly conditioned on a pooled sentence embedding or a few global text tokens, which can hide local correspondences between words and sign segments. In the full-pipeline conditioning comparison, using both the global sentence embedding and word tokens gives R@1 of 82.31 and 70.25 on CSL-Daily and Phoenix-2014T, versus 75.34 and 41.28 with the global sentence embedding alone.
Perspective
The demonstrated scope is two benchmarks, CSL-Daily (Chinese sign language, 18K training and 1176 test sentences) and Phoenix-2014T (German sign language, 7K training and 642 test sentences), with motion represented as 133-dimensional SMPL-X parameters sampled at 25 fps; quantizer training needs high-quality SMPL-X motion annotations and frame-level gloss, while inference needs no gloss.
Most of the semantic retrieval gain comes from explicit semantic modeling, and the full variant trails the semantic-only variant on most geometric metrics; a 4-frame gloss-span shift or 20% gloss dropout changes CSL-Daily R@1 while Phoenix-2014T R@1 stays nearly unchanged; the authors list cross-lingual settings, explicit modeling of non-manual markers and a Deaf-participant human study as future work.