A shared policy can theoretically match any multi-policy solution, but phase duration, heterogeneity, and per-phase data decide whether single or multiple policies win in practice
Related research and updates
Synopsis
For non-stationary reinforcement-learning problems decomposable into phases, this work first shows that a single shared policy can theoretically achieve the performance of any multi-policy solution, then proposes a regime-based phase decomposition method that uses the duration of transient dynamics relative to the quasi-stationary period to identify which policy performs better, and validates four hypotheses across numerical experiments on different non-stationary RL problems: longer phase durations favor multi-policies, greater heterogeneity between phases increases the burden on a single policy, multi-policies need sufficient data for each phase, and environment-specific transition dynamics between phases affect which policy is preferable.
Figure 3 : Geometric intuition of policy class equivalence and local specialist learning. (a) The common achievable policy space Π sh \Pi^{\mathrm{sh}} encompasses both shared and multi-policy solutions, as established by Proposition 2 . Depending on the alignment of phase-local entry distributions with deployment handoffs, exact multi-policy assembly yields either a specialization gain ( Δ J ¯ > 0 \Delta\bar{J}>0 ) or a handoff mismatch penalty ( Δ J ¯ < 0 \Delta\bar{J}<0 ). (b) Individual phase specialists π ^ m sp \widehat{\pi}_{m}^{\mathrm{sp}} are trained independently within local policy spaces Π m sp \Pi_{m}^{\mathrm{sp}} under phase-local entry distributions. The tuple 𝝅 ^ sp \widehat{\boldsymbol{\pi}}^{\mathrm{sp}} is assembled via the exact gating map Γ \Gamma with zero projection loss.
arXivInterpretation
The paper establishes a theoretical result: when the phase sequence is known and the state is augmented to satisfy the Markovian property, a single shared policy can theoretically achieve the performance of any multi-policy solution. Prior work observed that multi-policy approaches for different phases can outperform a single state-augmented policy shared among phases, but the reasons remained unclear; this work separates that empirical observation from theoretical attainability, indicating the gap does not come from representational capacity itself. This is a theoretical argument, stated in the abstract as 'we first show that the shared policy can theoretically achieve performance of any multi-policy solution'; proof details or theorem numbers are not given in the loaded text.
The paper proposes a regime-based phase decomposition method that uses the duration of transient system dynamics relative to the duration of the quasi-stationary period to identify which policy provides better performance. It turns the single-versus-multiple policy choice from empirical trial and error into a criterion that can be read off the time scales of the phase structure. The method description rests on 'the duration of transient system dynamics relative to the duration of the quasi-stationary period'; the concrete algorithm and thresholds are not expanded in the loaded text.
Numerical experiments on different non-stationary RL problems validate four hypotheses: longer phase durations favor multi-policies; heterogeneity between phases increases the burden on a single policy; multi-policies need sufficient data for each phase; and environment-specific transition dynamics between phases affect which policy is preferable. It attributes the practical advantage of single versus multiple policies to identifiable factors: phase duration, inter-phase heterogeneity, per-phase sample size, and inter-phase transition dynamics. Evidence comes from 'numerical experiments ... with different non-stationary RL problems'; the abstract does not report specific environments, sample sizes, or effect sizes.
The paper states that whether a multi-policy solution outperforms the corresponding single shared policy in practice depends on function approximation, learning and optimization processes, and, for multi-policy solutions, sample efficiency and loss of continuity from one policy to another. It locates the sources of practical divergence from theoretical equivalence in approximation, optimization, sample efficiency, and policy-switching continuity rather than in the number of policies itself. This is a conceptual attribution statement; the abstract provides no ablation data isolating each factor.
Perspective
The result targets non-stationary reinforcement-learning settings where the phase sequence is known, the problem decomposes into phases, and each phase has its own transition probabilities and reward functions; it applies to training scenarios that must decide between a single state-augmented policy and per-phase multi-policies. The criterion is based on the relative duration of transient dynamics and the quasi-stationary period, so users need to be able to characterize phase durations and inter-phase heterogeneity. For multi-policy schemes with sufficient data per phase, and for cases where inter-phase transition dynamics vary by environment, the four hypotheses provide directional guidance.
The loaded text is only the abstract and contains no theorem statements or proofs, no list of experimental environments, no sample sizes, no effect sizes, and no ablation results, so the strength of assumptions behind the theoretical attainability result and the robustness of the four hypotheses across environments cannot be judged. The concrete algorithm of the regime-based phase decomposition method, how transient and quasi-stationary durations are estimated, and how inter-phase heterogeneity is measured are not described in the text; these are open questions a reader would need to confirm in the body. In addition, how the sample efficiency of multi-policies and the loss of continuity across policy switches are quantified, and under which conditions these factors reverse the theoretical equivalence, remain to be addressed in the full paper.