Skip to main content
Search

# Test-Time Training

4 related articles

Subscribe to updates
  1. arXiv

    Titan-Llama bolts external neural memory onto a frozen Llama: long-document QA accuracy rises from 23.87% to 40.51%

    Titan-Llama converts the first four layers of Llama-3.2-1B-Instruct to blockwise attention and attaches a fixed-size neural memory module to each KV head, using test-time training to compress historical key-value bindings while the backbone stays frozen. Against segmented attention windows alone, accuracy rises from 23.87% to 40.51% on QuALITY and from 14.50% to 40.75% on LongHealth. In the evaluated configurations, prefill is more than 20% faster, decode 37% faster at medium context lengths, and steady-state decode memory 24% lower.

    Massachusetts Institute of TechnologyCornell University
    Sarah Pan（First author）
    Titan-LlamaTest-Time TrainingNeural Memory ModuleLong-context inference
  2. arXiv

    Nonlinear memory stops approximating by blocks: ParaTTT turns test-time training layers into parallel refinement, and its 340M model trains faster than TTT-MLP

    Test-time training layers replace the recurrent state with a small neural network trained online by gradient descent, which is more expressive than a linear memory but forces sequential computation because each token's gradient depends on every earlier update. ParaTTT instead iteratively refines an estimate of the hidden-layer pre-activation gradients, with each refinement built from a forward, a readout and a backward chunkwise-parallel pass, and the exact online recurrence is a fixed point of that iteration. At 4K context, two refinements reach the error of TTT-MLP's block approximation in less time, and 340M and 1.3B models match or exceed Gated DeltaNet and Mamba-2 on perplexity and recall.

    Massachusetts Institute of Technology
    Noel Loo（First author）
    ParaTTTTest-Time TrainingLinear AttentionLanguage modeling
  3. arXiv

    Decision Titan adds test-time training layers to a Decision Transformer and learns long-term dependencies 20x longer than its context window in X-Maze

    The work augments a Decision Transformer with Test-Time Training (TTT) layers, forming Decision Titan, and analyzes its performance and memory mechanism in the X-Maze environment built to test sequential memory: the model learns long-term dependencies with ranges 20x longer than the context window and generalizes to lengths 1.7x the training data, but temporal generalization depends on the time embeddings used and the ability to learn long-term dependencies depends on how the relevant information is encoded.

    Durham University
    Jude E. T. Waide（First author）
    Decision TitanTest-Time TrainingOffline Reinforcement Learning
  4. arXiv

    EvolvingAvatar lets a 3D head learn while it generates, cutting expression-statistics mismatch by up to 11.1% on OOD-Hard

    EvolvingAvatar introduces a causal interactive 3D head generator that adapts at inference time via test-time training on user face video and dyadic audio, using a Dyadic Context Prediction self-supervised objective with persistent fast weights and transient jaw adaptation to produce speaking and listening motion, and releases InterHead-Bench, a 455.95-hour benchmark; experiments show improved conversational motion statistics over strong baselines and improvement as conversations unfold on the hardest out-of-distribution split, reducing mismatch with recorded user-avatar expression statistics by up to 11.1% from the first interval.

    Junjie Chen（First author）
    EvolvingAvatarTest-Time TrainingDyadic Context PredictionInteractive 3D Head GenerationInterHead-Bench

You’re all caught up