# Qwen3-1.7B

This work asks whether raw web video, with no captions and no text loss, can serve as mid-training data for a pretrained language model: frames are encoded into continuous visual tokens and the model learns to predict the next visual token, Qwen3-1.7B is mid-trained on raw clips from YT-Temporal-1B and then given the same image-text instruction tuning as a model without mid-training, yielding 2.9 points higher on video benchmarks and 5.1 points higher on image benchmarks, with text average rising from 48.0 to 48.9, and caption prediction offering no advantage over visual-token prediction.

This work asks whether raw web video, with no captions and no text loss, can serve as mid-training data for a pretrained language model: frames are encoded into continuous visual tokens and the model learns to predict the next visual token, Qwen3-1.7B is mid-trained on raw clips from YT-Temporal-1B and then given the same image-text instruction tuning as a model without mid-training, yielding 2.9 points higher on video benchmarks and 5.1 points higher on image benchmarks, with text average rising from 48.0 to 48.9, and caption prediction offering no advantage over visual-token prediction.

The work introduces Task-Progress Distillation (TPD), an offline approach that pairs each demonstrated action with a short task-stage label so the student selects actions by jointly scoring admissible stage–action pairs; on ALFWorld, a 1.7B student trained with 404 demonstrations reaches 72.4% mean unseen task success with either TPD or action-only supervision, versus 48.3% for a reasoning-trained student using constrained action selection, and explicit stages raise success from 48.0% to 67.7% at 200 demonstrations, with action-only catching up to 76.9% at 808.

The work introduces Task-Progress Distillation (TPD), an offline approach that pairs each demonstrated action with a short task-stage label so the student selects actions by jointly scoring admissible stage–action pairs; on ALFWorld, a 1.7B student trained with 404 demonstrations reaches 72.4% mean unseen task success with either TPD or action-only supervision, versus 48.3% for a reasoning-trained student using constrained action selection, and explicit stages raise success from 48.0% to 67.7% at 200 demonstrations, with action-only catching up to 76.9% at 808.

This work shows mathematically that, in sparse-reward RLVR, omitting or truncating importance-weight correction for enriched rollouts implicitly reweights the defined reward, and decomposes the resulting gradient error into scale, rotation, and variance; the authors develop a sequential Monte Carlo (SMC) weight correction mechanism that, under stability and particle-order assumptions, tempers the exponential compounding of standard correction variance over sample length into additive accumulation, and they apply this analysis to interpret a fine-tune of Qwen3-1.7B on a sparse band of OpenMathReasoning, establishing a concrete mechanism of how enrichment, corrected and uncorrected, helps avoid collapse in sparse domains.

This work shows mathematically that, in sparse-reward RLVR, omitting or truncating importance-weight correction for enriched rollouts implicitly reweights the defined reward, and decomposes the resulting gradient error into scale, rotation, and variance; the authors develop a sequential Monte Carlo (SMC) weight correction mechanism that, under stability and particle-order assumptions, tempers the exponential compounding of standard correction variance over sample length into additive accumulation, and they apply this analysis to interpret a fine-tune of Qwen3-1.7B on a sparse band of OpenMathReasoning, establishing a concrete mechanism of how enrichment, corrected and uncorrected, helps avoid collapse in sparse domains.