Step 5 Preview bets on depth over width: 92 blocks, 1M-token context, million-level agent environments

Long-horizon agentic tasks shift the design burden from explicit reasoning toward implicit reasoning during prefilling, because frequent tool calls create large prefilling contexts and leave little room for full chain-of-thought. Earlier practice relied on explicit multi-hop reasoning at generation time, whereas agentic settings spread multi-hop dependencies everywhere while prefilling information flows only along the temporal and depth axes. Step 5 Preview therefore scales depth directly to 92 transformer blocks, matching the depth of large open-source models such as Qwen3.8-Max and Kimi K3 while using under one-quarter of their total parameters and under one-third of their active parameters.

Full attention layers become the main efficiency bottleneck in both training and inference once context reaches 1M tokens, and sparse attention must balance indexer cost, GPU utilization, and selection granularity. In some existing sparse attention implementations the indexer overhead is non-negligible and can even exceed the cost of the sparse Flash Attention operator itself under extremely long contexts. Sparse GQA compresses indexers for adjacent historical positions into block representations with block size 8, cutting indexer and top-k selection cost by 8x; at B=8 and k=4096 token coverage is 78.06%, close to the 79.09% of the B=1, k=2048 reference configuration.

Data construction is reorganized as an agent-native process in which experts supply high-level principles and priors, agents coordinate construction within those principles, and deterministic operations go to reusable code and tools. Previously, domain experts embedded priors, intuitions, and practices in their own task designs, pipelines, and decisions, so adapting construction to a new domain meant starting over. A unified context layer retains task specifications, pipeline states, environment configurations, intermediate results, and feedback so agents can reuse experience across tasks; quality steering targets difficulty and diversity biases and adds anti-hacking checks, producing million-level verifiable environments spanning scientific reasoning, software engineering, machine learning R&D, and professional work.

RL for long rollouts separates system-intrinsic numerical differences between training and inference from off-policy staleness, aligning exactly where possible and replaying inference computational state where staleness prevents alignment. In online RL the training and inference sides are built from distinct operators with different performance objectives, so tiny numerical differences amplify through MoE dispatch and depth and then couple with off-policy staleness. At staleness zero the training and inference operators are fully aligned with a logprob difference of exactly zero, and training at TP=1 stays bitwise identical to inference at TP=8 with total overhead below 10%; with RL trace replay, long-horizon logprob difference stays below 1e-2.

The report covers the architecture and training design of Step 5 Preview, written for researchers and engineers working on long-context models, agent training, and reinforcement learning infrastructure. The reported depth, parameter ratios, token coverage, and logprob difference come from the design's own observations and apply to long-horizon agentic tasks and context training up to 1M tokens. The sub-1% sample drop rate belongs to the training infrastructure side and is a separate result from those architecture observations.

Scaling the narrow-but-deep design produced numerical explosions and gradient spikes whose underlying cause remains unclear, mitigated only through FP32 hidden states, FP32 RMSNorm scaling, and fine-grained gradient clamping. Autonomous agent synthesis shows systematic biases in difficulty and diversity that require explicit steering signals. The system is still evolving, and limits of context, training infrastructure, and feedback become harder to separate as one improvement exposes bottlenecks elsewhere.