TROPIC swaps summation for maximization in reinforcement learning, beating the strongest on-policy baselines by up to 16 percentage points on four agentic tasks

The paper argues that the classical sum formulation of expected return can only report how often the policy succeeds, not which solution actually worked, and that because probabilities sum to one, reinforcing one solution can make the model forget another that was never shown to be wrong. It grounds this diagnosis in compositional reasoning, where a solution must be assembled from reasoning steps the model produces in separate, often failed attempts but rarely produces together, making expected return a poor fit for such tasks. The argument is presented as a conceptual characterization of the problem and its task setting rather than as an independent experimental measurement.

The paper proposes Tropical Reinforcement Learning, whose core is a change of algebra: instead of adding the probabilities of alternative solutions, it takes their maximum, yielding the tropical semiring; the value of a state then becomes the log-probability of its most likely verified solution, together with an explicit path that can be replayed and reused. Unlike approaches that only change estimators, this changes the algebra of reinforcement learning itself, so the best prefix and the best suffix can be genuinely joined at a shared state even when they come from different rollouts. This is a method-level construction and argument, presented through the algebraic substitution and the definition of state value.

The paper introduces TROPIC, a training algorithm for deterministic, resettable environments with verifiable outcomes. It turns the tropical-semiring value definition into a concrete training procedure and states its applicable conditions as deterministic, resettable, and verifiable-outcome environments. The algorithm design and its applicability conditions are explicitly stated in the paper as a method description.

On four agentic tasks (Sokoban, Countdown, FrozenLake, WebShop), TROPIC outperforms the strongest on-policy baselines by up to 16 percentage points. These comparisons support the claim that changing the algebra of reinforcement learning, not just its estimators, can substantially improve compositional reasoning in language models. Evidence comes from comparisons against the strongest on-policy baselines on four named agentic tasks, with a reported maximum improvement of 16 percentage points; the abstract does not give per-task numbers, random seeds, or statistical test details.

The work targets deterministic, resettable environments with verifiable outcomes, and this setting defines where TROPIC applies; the beneficiaries are language model training pipelines that need to assemble reasoning steps across multiple rollouts to solve tasks compositionally. What can be reused is that a state's value simultaneously yields the log-probability of its most likely verified solution and an explicit path, letting the best prefix and best suffix be joined at a shared state even when they come from different rollouts. As shown in the present text, this capability is tested on four agentic tasks (Sokoban, Countdown, FrozenLake, WebShop), with improvements of up to 16 percentage points over the strongest on-policy baselines.

At the abstract level, per-task scores, random seeds, variance, and statistical tests are not given, so how the maximum 16-percentage-point improvement is distributed across the four tasks remains to be checked in the full text. How the tropical-semiring maximum and the explicit path behave at larger model scales, over longer reasoning chains, and in non-deterministic or non-resettable environments is an open question worth watching. The abstract also does not specify the exact baseline configurations or task scales, so readers concerned with comparison conditions would need to consult the original text.