Sequence models stop squeezing history into one state: semantic similarity picks the lag, 95.35% on IMDB

A sequence model gains an explicit look-back channel: when the current state updates, it pulls a weighted context from historical states by semantic similarity instead of folding all history into one hidden state. Recurrent networks and LSTMs compress every time step into a single hidden state, transformers attend to all tokens indiscriminately at quadratic cost, and state-space models such as Mamba compress history into a fixed-dimensional state, none of them offering content-addressable memory. On IMDB sentiment classification the model reaches 95.35% accuracy with 2.6M parameters, against 89.90% for LSTM, 83.00% for Mamba, 94.13% for RoBERTa-base and 92.13% for DistilBERT-base in the same table.

How far back to look is decided by meaning: the model computes cosine similarity between the current token embedding and each historical embedding in the window, then applies a temporal decay that suppresses distant history to weight each lag. The delay is no longer a fixed hyperparameter but a quantity that varies with the input; the similarity is normalized through a sigmoid into a semantic confidence, hard selection takes the smallest lag above an input-dependent threshold, and soft selection attends over all lags so end-to-end training stays possible. On a ten-word example sentence, with the maximum lag set to 10, the word "amazing" at position 3 looks back over at most 10 preceding words, and the visualization shows "movie" has the largest influence on "amazing"; classification accuracy generally rises with the maximum lag while fluctuating.

The state recurrence is written as a diagonal linear recurrence whose unrolled form is a closed-form cumulative sum, so no step-by-step loop is needed and each step costs O(1). Recurrent structures previously had to advance sequentially and attention had to compare all token pairs; restricting the state transition matrix to diagonal lets one element-wise cumulative sum compute the whole sequence. On CNN/DailyMail summarization the model reaches 42.63 ROUGE-L with 16.98M parameters, against 38.45 for LSTM, 40.48 for BERT, 29.60 for Mamba, 24.58 for BART CNN-large and 23.54 for T5-base in the same table.

The report covers two text tasks: IMDB sentiment classification using the full 25,000 training and 25,000 test samples, and CNN/DailyMail summarization using 80,000 training samples with 2,000 each for validation and testing. The model is configured with hidden dimension 256, a maximum semantic delay of 30 and 6 fuzzy inference rules, trained in PyTorch on a single Nvidia H100 80G. The authors suggest the framework has potential for image and audio processing and generation, but no such experiments are reported.

The authors state that because the number of benchmark experiments is limited, the model's broader generalizability remains untested, and that the model is also more sensitive to hyperparameter tuning. The theoretical results on existence and uniqueness, boundedness and stability rest on assumptions such as Lipschitz continuity, bounded input, and similarity scores between 0 and 1.