Information events in language have predictable arrival times: new-word intensity decays, and LLMs introduce new words faster than humans

Information-bearing words in a text can be treated as discrete events arriving along a time axis, and their arrival times can be predicted. Earlier work on how information is distributed in language mostly relied on post-hoc analysis of information-theoretic metrics such as surprisal, leaving the generation process beneath the surface unmodeled. On speech and writing-prompt texts, neural temporal point processes predict the next event's arrival time with lower mean absolute error than conventional temporal point processes and simple neural baselines.

The model fuses the history of event intervals, the absolute position of each event, and a language model's semantic encoding of the preceding text into one history representation, then estimates event intensity from it. Conventional temporal point processes use fixed-form intensity functions, such as a slowly evolving Poisson process, which struggle to express the bursts and long-range correlations of language. A recurrent network that learns the cumulative hazard function and incorporates LLaMA-3.1-8B hidden states achieves the lowest mean absolute error across both event types and both datasets.

The two event types have different intensity shapes: new-word intensity decays over time, while high-surprisal-token intensity stays largely flat. For new-word events, human intensity curves stay consistently below those of LLM texts, meaning LLMs produce new words at higher rates; this gap is not apparent for high-surprisal-token events. Classifying human versus LLM text from intensity-curve features reaches accuracies of 0.945 to 0.980 on speech data and 0.854 to 0.959 on writing-prompt data.

This work is aimed at computational linguistics and psycholinguistics researchers, and examines the arrival times of two event types, new words and high-surprisal tokens, in English speech and writing-prompt texts. The time axis is measured in word or token positions, and models are trained on discrete time steps. Extending the conclusions to other languages, genres, or kinds of information events would require redefining events and retraining.

Mean absolute error is generally higher for high-surprisal-token events, but these events have lower intensity and longer inter-event intervals, which naturally inflates absolute error measured in time steps, so whether they are inherently harder to predict requires an interval-normalized comparison. High-surprisal tokens are defined by a language model's output probabilities, and that estimator shares the same near-lossless access to context as the generating LLMs, so the smaller human-LLM difference on this event type may partly reflect a blind spot of the metric. Word-level and token-level events also occur only at integer points, while the models used are built on a continuous time axis.