Offline RL drops the moving target: AL-LBLP solves the same linear program with learned multipliers and averages 60.1 across 50 OGBench tasks

The offline RL critic no longer has to regress onto a bootstrapped target that moves with the policy; it solves a linear program defined over the dataset, so no target network or critic ensemble is needed. ALBUM solved the same LBLP but relaxed its constraints into hinge penalties with fixed coefficients, and its guarantees were conditions on coefficients chosen in advance, with thresholds that depend on the problem and are not known beforehand. On the 50 state-based OGBench tasks the same critic averages 55.8 with a Gaussian actor, against 43.4 for ALBUM and 43.8 for FQL, and 60.1 with best-of-n sampling, against 30.3 for IFQL.

The weights on the constraints are learned during training instead of being set by hand. AL-LBLP replaces ALBUM's three fixed hinge coefficients with multipliers learned per constraint family by dual ascent, so the relaxation has a single penalty parameter that sets both the dual step size and the curvature of the penalty on violated rows. In the convex case the fixed points of primal minimization and the dual update are the KKT pairs of LBLP, and at 1M steps the mass identity in terms of the effective multipliers lies between 0.148 and 0.154 on all ten environments against a target of 0.15.

One critic can serve actors of different types because it never queries the policy during training and bootstraps only at dataset transitions. Temporal-difference methods call the policy at every critic update, so with a vision-language-action model as the policy each critic step pays a policy forward pass and depends on actions outside the data. The same critic serves a Gaussian actor, FQL's one-step flow actor and best-of-n sampling at averages of 55.8, 55.7 and 60.1 on the 50 tasks, and training does not collapse up to batch size 8192. The source's contribution list says FQL and FAC fall to 0.0 at batch 2048 to 4096, but Table 9 of the same source gives FQL 4.9 and FAC 0.0 and 0.1 at batch 2048; the two disagree, so that figure is not used here.

The demonstrated setting is the 50 state-based single-task OGBench tasks, 8 seeds, with the final evaluation after 1M gradient steps over 50 episodes; the long-horizon results come from antmaze-giant and humanoidmaze-giant with rollout length raised to 50 and discount to 0.999, reaching 79.8 and 81.5 at batch 4096. The next step stated in the source is to attach an AL-LBLP critic to a vision-language-action model through best-of-n sampling.

The penalty parameter trades early learning speed against long-run stability: at 0.01 the ten-environment mean is 56.1 at 0.5M but falls to 43.9 at 3M and collapses on scene from 91.3 to 8.7, while at 1 it is 54.3 at 1M and 59.2 at 3M. The method is weaker on manipulation than on navigation, averaging 64.7 on the five pixel-based environments against 65.4 for FQL and 76.0 for FAC. The multipliers are scalar per constraint family, an approximation of the per-row method, and the penalty parameter and some hyperparameters are still selected per environment.