Transferable rewards without a flood of expert demos: actively picking dynamics cuts the demonstrations needed
Synopsis
Inverse reinforcement learning aims to recover expert rewards that transfer across environments with differing dynamics, and expert demonstrations are costly, so choosing which dynamics to collect them under matters. A two-phase approach first cycles through the available dynamics to obtain a reliable reward estimate, then actively selects dynamics to reduce uncertainty about the expert reward, yielding sample-complexity guarantees on transfer that hold for any dynamics with a rate independent of the number of available dynamics. Empirically, the method significantly reduces the number of expert demonstrations needed to learn a transferable reward.
Interpretation
Inverse reinforcement learning recovers a reward from expert demonstrations and aims for that reward to transfer across environments with differing dynamics; the new method treats which dynamics to gather demonstrations under as a decision to be made actively. Prior work showed such transfer is possible when demonstrations are gathered across sufficiently different dynamics, but demonstrations are costly, making it critical to select the right dynamics to reduce the expert data required. That comparison comes from the stated condition for transfer in prior work, and the new method folds dynamics selection into learning under that condition.
The method runs in two phases: it first cycles through the available dynamics to obtain a reliable reward estimate, then actively selects dynamics to reduce uncertainty about the expert reward. This two-phase design pairs cycling-based estimation with active selection, where the second phase chooses demonstrations by their effect on uncertainty about the expert reward. The procedure operates over the set of available dynamics and yields sample-complexity guarantees on transfer that hold for any dynamics.
The theoretical result gives sample-complexity guarantees on transfer whose rate does not grow with the number of available dynamics. The guarantee takes the form of a rate independent of the number of available dynamics while holding for any dynamics. The guarantee holds for any dynamics, with a rate independent of the number of available dynamics.
Empirically, the method significantly reduces the number of expert demonstrations needed to learn a transferable reward. What is reduced is the number of expert demonstrations required to learn a transferable reward, matching the stated goal. This conclusion comes from the empirical evaluation of the proposed method; the material gives no specific environments, demonstration counts, or baseline values.
Perspective
The described method targets settings with multiple available dynamics under which expert demonstrations can be gathered; the theoretical guarantee holds for any dynamics, with a rate independent of the number of available dynamics. The material proposes no next experiment or deployment step.
The empirical conclusion is stated as a significant reduction in the number of expert demonstrations needed, and the material gives no specific environments, demonstration counts or baseline values; the conditions of the sample-complexity guarantee are summarized only as holding for any dynamics.