User simulators now write a private review each turn: assistant RL keeps its score and halves the compute

A user simulator now produces, starting from the second user turn, a first-person evaluation of the assistant's reply before generating the next user utterance: an internal thought, a 1-to-5 rating, whether the instruction was followed, a critique-or-praise reaction, and a 2-to-5-word feedback summary. Earlier simulators were trained mainly as utterance generators, so judging whether the assistant satisfied the user required a separate external judge model to estimate that signal from the conversation. On real user-ChatGPT logs from WildChat, the Gemma3-12B simulator's statistical alignment with human turns rises from 0.5135 for the prompted baseline to 0.8635, and its perplexity falls from 13.41 to 3.13.

Those evaluations can serve directly as the reward signal for multi-turn reinforcement learning of assistants, without an external judge model. The external-judge route needs an extra large-model inference per turn, and a judge may correlate poorly with the user's own first-person assessment. Training a Qwen2.5-1.5B assistant with intrinsic rewards from a Gemma3-4B simulator matches or slightly exceeds the external-judge setup on the 1024 multi-turn task checklists of WildBenchV2, while the estimated total compute is about 2.24 times smaller.

The ratings the simulators produce also line up better with real human judgments. Prompted simulators are controllable but have been reported to lack realism and to drift from real user behavior. On the 909 human-assistant conversations of SimulatorArena, Gemma3-12B-LES correlates 0.401 with human ratings against 0.295 for the prompted baseline of the same size.

These simulators target multi-turn environments trained on real user interaction logs, for evaluating assistants and for offline reinforcement learning. Training data come from 135,901 English conversations in WildChat-1M after moderation, deduplication, and removal of WildBench overlap, with evaluation labels generated by Gemini3.1-Pro; assistant-side experiments use Qwen2.5-1.5B, and simulators cover Gemma3 at 4B, 12B, and 27B plus Qwen2.5 at 3B, 7B, and 14B.

The performance edge of intrinsic rewards over an external judge is described as small and tied to the current datasets; a judge that is too small barely helps, as a Qwen2.5-3B judge produced only minimal improvement. Simulators with latent states have lower recall on critiquing behavior than plain utterance-generating SFT simulators, which the authors attribute to greater caution before raising a critique; in the training data, critiques are 14.15% of turns, praise 1.26%, and the remaining 84.59% neutral.