# nanochat

nanochat is a minimal experimental harness for training large language models on a single GPU node, covering tokenization, pretraining, finetuning, evaluation and inference. It reduces model size to one complexity dial, --depth, the number of layers in the GPT transformer, and derives width, head count, learning-rate adjustments and training horizons automatically. By the repository's accounting, a GPT-2-capability model trains in about two hours for about $48 on an 8XH100 node, against roughly $43,000 for GPT-2 in 2019.

nanochat is a minimal experimental harness for training large language models on a single GPU node, covering tokenization, pretraining, finetuning, evaluation and inference. It reduces model size to one complexity dial, --depth, the number of layers in the GPT transformer, and derives width, head count, learning-rate adjustments and training horizons automatically. By the repository's accounting, a GPT-2-capability model trains in about two hours for about $48 on an 8XH100 node, against roughly $43,000 for GPT-2 in 2019.