# Text Entropy Rate Filtering

The work introduces a non-parametric Kontoyiannis entropy rate estimator h_k, computed entirely from raw text with no model of any kind, as a training-data filter; in a six-generation QLoRA collapse experiment on Llama-3.1-8B, h_k-filtering yields +42% unique trigrams, +30% vocabulary, and -19% repetition (all p<0.001), whereas logprob-based filtering provides no significant text-diversity benefit on any metric (p>0.23); h_k is further validated as a cross-domain entropy proxy (beta=0.924, R²=0.746) and collapse detector (rho=+0.454, p<0.0001) across 4 domains, 2 temperatures, 2 generator-scorer model pairs, and 1,520 generated documents.

The work introduces a non-parametric Kontoyiannis entropy rate estimator h_k, computed entirely from raw text with no model of any kind, as a training-data filter; in a six-generation QLoRA collapse experiment on Llama-3.1-8B, h_k-filtering yields +42% unique trigrams, +30% vocabulary, and -19% repetition (all p<0.001), whereas logprob-based filtering provides no significant text-diversity benefit on any metric (p>0.23); h_k is further validated as a cross-domain entropy proxy (beta=0.924, R²=0.746) and collapse detector (rho=+0.454, p<0.0001) across 4 domains, 2 temperatures, 2 generator-scorer model pairs, and 1,520 generated documents.