nanochat squeezes GPT-2-grade training onto one node: about two hours and $48 on eight H100s
Synopsis
nanochat is a minimal experimental harness for training large language models on a single GPU node, covering tokenization, pretraining, finetuning, evaluation and inference. It reduces model size to one complexity dial, --depth, the number of layers in the GPT transformer, and derives width, head count, learning-rate adjustments and training horizons automatically. By the repository's accounting, a GPT-2-capability model trains in about two hours for about $48 on an 8XH100 node, against roughly $43,000 for GPT-2 in 2019.
Interpretation
Training a GPT-2-capability language model now fits on a single 8XH100 node, and the result can be talked to over a command line. GPT-2 cost about $43,000 to train in 2019; at roughly $3 per GPU per hour, an 8XH100 node runs about $24 per hour, so two hours is about $48. The repository sets GPT-2 (1.6B) at a CORE score of 0.256525 as the capability bar and keeps a GPT-2 speedrun leaderboard ranked by wall-clock time to beat it.
The whole pipeline sits in one script, running tokenization, pretraining, finetuning, evaluation and inference end to end. Assembling comparable pipelines otherwise means stitching together separate stage code and configuration, whereas runs/speedrun.sh is the reference path to train a GPT-2-grade model and chat with it. The script is designed for an 8XH100 node and takes about 1.5 hours, after which python -m scripts.chat_cli talks to the model from the local virtual environment.
Model size is set by one dial, --depth, the number of layers in the GPT transformer, with the remaining hyperparameters computed automatically. Width, number of heads, learning-rate adjustments, training horizons and weight decays no longer need individual tuning; they follow from the depth setting. The repository notes that GPT-2 capability corresponds to approximately depth 26, and the same rule generates a whole miniseries of compute-optimal models.
Perspective
The repository targets developers and researchers who want to train and chat with a model on a single 8XH100 node, with the speedrun script as the reference path; the trained model is a 4e19 FLOPs capability model, which the repository describes as roughly kindergarten level.
Cost figures are derived from current pricing of about $3 per GPU per hour, and on a spot instance the total can be closer to about $15; development currently focuses on tuning the pretraining stage.