Reward no longer goes only to the nearest action: letting a language model explain outcomes traces delayed credit to its source

In long-horizon decisions, an action's consequence may appear only hundreds or thousands of steps later, and standard reinforcement learning propagates reward by temporal proximity, so actions farther from the outcome receive a weaker learning signal. The prior approach relies on discounted returns, treating temporal closeness as a proxy for relevance; the longer the delay, the more trial and error is needed to average out spurious associations, so delay length translates directly into sample inefficiency. On Crafter, after aggregating technology-achievement rewards into a single terminal reward, vanilla PPO clearly cannot learn in this delayed long-horizon setting, while explanation-based redistribution outperforms PPO across all eight tested combinations of discount factor and episode-length cap.

The framework uses a language model as a backward-looking explainer: given a trajectory and a realized outcome, the model identifies which earlier events plausibly contributed to it and assigns each a weight. Unlike LaRe, which uses an LLM to build a latent-reward representation and learns a decoder mapping it to episodic returns, this approach asks the model which observed events explain the outcome, with attributions restricted to events that occurred in the trajectory. On the hard WordCraft split, the language-model explainer reaches almost the oracle score from an explainer that identifies the true recipe dependencies, while LaRe barely learns and PPO plateaus little above random.

The weights from the explanation are moved onto the cited events by a conservative hierarchical redistribution operator, which recursively explains those events in turn so reward can reach arbitrarily far into the past. Discounted returns use a fixed temporal coefficient to decide where reward lands, whereas here attribution weights depend on the realized outcome and the trajectory evidence, and the shares allocated at every node sum exactly to the amount that node received, leaving the episode return unchanged. In the online KuaiRand setting, an explanation-derived five-step termination rule raises mean evaluation return from 3.773 for vanilla A2C to 6.736, and the full-history extension reaches 13.249, against 15.653 for the oracle gate.

The method targets reinforcement-learning and recommender researchers who need credit assignment under delayed outcomes, in episodic tasks where rewards are sparse and interactions are expensive. It sits as an attribution layer between observing rewards and updating the policy, modifying neither the environment, the policy class, nor the optimization algorithm, so it can combine with other episodic RL methods that consume per-transition rewards. In the online setting the policy still interacts with the original environment; in the offline setting it is applied once to each logged trajectory, and the resulting explanation graphs and redistributed rewards can be cached and reused.

Online KuaiRand evaluation returns come from model-based simulated feedback rather than directly observed engagement from a deployment, so they measure performance in a simulated recommendation environment. The full-history gate reaches a higher cumulative return but its mean reward per step drops to 0.442, below the 0.467 of the vanilla policy, so its gain comes from longer sessions rather than higher reward per recommendation. The industry recommender results come from offline replay against a held-out log and are behavioral proxies. Differences among explanation variants also shift with the evaluation task: in the offline KuaiRand setting the breadth variant achieves the largest mean AUC improvement of 1.99 percentage points, yet its mean NDCG@10 difference is close to zero and its mean MRR difference is slightly negative.