# Sim-to-Real Transfer

12 related articles

Subscribe to updates

arXiv Nov 4, 2026
Humanoid soccer stops juggling separate controllers: SoccerBot unifies four skills on a human motion prior and transfers zero-shot to a real robot
Humanoid soccer robots have either covered only a few skills or skipped motion priors entirely and moved unnaturally. SoccerBot distills large-scale human soccer motion-capture data into a structured latent motion prior, letting a single skill-conditioned policy learn moving, dribbling, trapping and kicking in one latent space. On the Unitree G1, the policy transfers zero-shot to hardware, with simulation success rates of 97.04% for kicking and 89.88% for dribbling, while real-robot kicking direction error rises from 4.09 to 10.41 degrees.
Humanoid soccer robots have either covered only a few skills or skipped motion priors entirely and moved unnaturally. SoccerBot distills large-scale human soccer motion-capture data into a structured latent motion prior, letting a single skill-conditioned policy learn moving, dribbling, trapping and kicking in one latent space. On the Unitree G1, the policy transfers zero-shot to hardware, with simulation success rates of 97.04% for kicking and 89.88% for dribbling, while real-robot kicking direction error rises from 4.09 to 10.41 degrees.
Shanghai Jiao Tong University Beijing Institute for General Artificial Intelligence Peking University
Haoyu Shao（First author）
SoccerBot Latent Motion Prior Humanoid Soccer Hierarchical Reinforcement Learning Sim-to-Real Transfer
arXiv Nov 5, 2026
TactileLab computes shear as tactile flow in rigid-body simulation, letting dexterous hands learn object rotation in sim and keep turning it on real hardware
TactileLab moves tangential motion out of soft-body deformation and generates a dense tactile flow field directly from rigid-body simulation, feeding it to policies alongside contact depth images. Rigid-body tactile simulators previously offered mainly depth-like signals that could not express shear or slip, while soft-body simulators captured deformation but could not sustain large-scale reinforcement learning. On dual-arm following and dexterous in-hand rotation, policies with tactile flow converged faster, and on real hardware the depth-plus-flow policy completed the most rotations across cylindrical, round and cube-shaped objects.
TactileLab moves tangential motion out of soft-body deformation and generates a dense tactile flow field directly from rigid-body simulation, feeding it to policies alongside contact depth images. Rigid-body tactile simulators previously offered mainly depth-like signals that could not express shear or slip, while soft-body simulators captured deformation but could not sustain large-scale reinforcement learning. On dual-arm following and dexterous in-hand rotation, policies with tactile flow converged faster, and on real hardware the depth-plus-flow policy completed the most rotations across cylindrical, round and cube-shaped objects.
Yijiong Lin（First author）
TactileLab Simulated Tactile Flow TacFlowNet PETS-Net Sim-to-Real Transfer
arXiv Nov 5, 2026
Humanoid HALO learns perceptive whole-body motion in simulation, then hits 91% over 360 trials in six unmapped real homes
Humanoids asked to sit down or walk to a named object need paired supervision mapping pixels and free-form language to whole-body joint commands, and that data has so far come from teleoperation or wearable capture, where an hour of data costs at least an hour of human time. HALO moves data generation into simulation, synthesizing egocentric vision-language-motion episodes across a thousand procedurally generated homes, then uses a layered stack that splits a free-form request into subtasks, generates five-second whole-body motion plans, and executes them with a perceptive tracker. Without seeing a real home in training, it transfers zero-shot to six unscanned, unmapped real homes, succeeding in 91% of 360 trials.
Humanoids asked to sit down or walk to a named object need paired supervision mapping pixels and free-form language to whole-body joint commands, and that data has so far come from teleoperation or wearable capture, where an hour of data costs at least an hour of human time. HALO moves data generation into simulation, synthesizing egocentric vision-language-motion episodes across a thousand procedurally generated homes, then uses a layered stack that splits a free-form request into subtasks, generates five-second whole-body motion plans, and executes them with a perceptive tracker. Without seeing a real home in training, it transfers zero-shot to six unscanned, unmapped real homes, succeeding in 91% of 360 trials.
Siyuan Hu（First author）
HALO Humanoid Whole-Body Control Synthetic Data Generation Sim-to-Real Transfer Flow Matching
arXiv Nov 7, 2026
A camera that hunts the wake instead of holding still: an agent finds hydrofoil wakes in a real water tunnel in about half a full scan's time
Particle image velocimetry (PIV) usually fixes the imaging position before the measurement, so if the structure of interest departs from expectation, no later algorithm can recover information that was never captured. This work treats camera placement as a closed-loop decision variable: a DreamerV3 agent trained in Isaac Sim drives a 6-DoF arm using only the velocity field and uncertainty estimated by LIMA to locate, frame and resolve a hydrofoil wake. After zero-shot transfer to a real water tunnel, the agent locates wakes of randomly placed hydrofoils and measures them at higher resolution and lower flow-statistics error than a fixed view, in roughly half the time of a full scan and with far fewer image pairs.
Particle image velocimetry (PIV) usually fixes the imaging position before the measurement, so if the structure of interest departs from expectation, no later algorithm can recover information that was never captured. This work treats camera placement as a closed-loop decision variable: a DreamerV3 agent trained in Isaac Sim drives a 6-DoF arm using only the velocity field and uncertainty estimated by LIMA to locate, frame and resolve a hydrofoil wake. After zero-shot transfer to a real water tunnel, the agent locates wakes of randomly placed hydrofoils and measures them at higher resolution and lower flow-statistics error than a fixed view, in roughly half the time of a full scan and with far fewer image pairs.
Chang Rajani（First author）
Particle Image Velocimetry Reinforcement Learning Sim-to-Real Transfer Latent World Model
arXiv Oct 14, 2026
Humanoid running and jumping need more than motion: Contact-UAN patches the actuator force model with 6.5 minutes of walking data
Humanoid policies learned in simulation can transfer motions to hardware while failing to transfer the force interactions learned in training, which costs torque efficiency, force-tracking accuracy and impact absorption. Contact-UAN keeps the simulator's rigid-body and contact physics and learns only a residual correction on the actuator command channel, using onboard proprioception to infer a stance-foot pose so replay starts with hardware-consistent contact forces. On a Unitree G1, a single residual model trained on 6.5 minutes of real walking data yields 20% lower running torque and 43% lower force-tracking error, and supports stable jumping and landing.
Humanoid policies learned in simulation can transfer motions to hardware while failing to transfer the force interactions learned in training, which costs torque efficiency, force-tracking accuracy and impact absorption. Contact-UAN keeps the simulator's rigid-body and contact physics and learns only a residual correction on the actuator command channel, using onboard proprioception to infer a stance-foot pose so replay starts with hardware-consistent contact forces. On a Unitree G1, a single residual model trained on 6.5 minutes of real walking data yields 20% lower running torque and 43% lower force-tracking error, and supports stable jumping and landing.
Massachusetts Institute of Technology
Nolan Fey（First author）
Contact-UAN Residual World Model Sim-to-Real Transfer Humanoid Control
NVIDIA Technical Blog Oct 8, 2026
Robots take on AI server test trays: busbar assembly tops 95% success, four-cable insertion reaches 90–95%
NVIDIA's Seattle Robotics Lab and the Isaac engineering team handed two GB300 tester-tray tasks to robot arms: busbar assembly and four-connector insertion. Busbar assembly exceeds 95% success but takes 160 seconds against a 124-second target; connector insertion reaches 90–95% at about 40 seconds per cable, also short of deployment requirements. The route was not pure end-to-end learning: classical modular pipelines came first, with learning added only for connector grasping and insertion.
NVIDIA's Seattle Robotics Lab and the Isaac engineering team handed two GB300 tester-tray tasks to robot arms: busbar assembly and four-connector insertion. Busbar assembly exceeds 95% success but takes 160 seconds against a 124-second target; connector insertion reaches 90–95% at about 40 seconds per cable, also short of deployment requirements. The route was not pure end-to-end learning: classical modular pipelines came first, with learning added only for connector grasping and insertion.
NVIDIA
DOPER TALOS GB300 Tester-Tray Assembly Sim-to-Real Transfer Reinforcement Learning
arXiv Oct 25, 2026
Construction robots drop hand-written avoidance rules: LiDAR-only corrections cut bricklaying path irregularity by 39.2%
Construction robots working near people must anticipate human motion, yet existing social-navigation methods rely on hand-crafted interaction rules, labeled demonstrations, or robot-specific control policies. This LiDAR-only framework generates residual targets automatically by constant-velocity extrapolation, models relations among robot, pedestrians and goal with a graph neural network, and uses an extended Kalman filter to turn frame-wise detections into dynamic graphs, outputting planner-agnostic path corrections. In a bricklaying trial it reduced path irregularity by 39.2% and raised minimum time-to-collision; across seven Dynablox sequences it raised average minimum time-to-collision by 38.3% with shorter, smoother paths than the baselines.
Construction robots working near people must anticipate human motion, yet existing social-navigation methods rely on hand-crafted interaction rules, labeled demonstrations, or robot-specific control policies. This LiDAR-only framework generates residual targets automatically by constant-velocity extrapolation, models relations among robot, pedestrians and goal with a graph neural network, and uses an extended Kalman filter to turn frame-wise detections into dynamic graphs, outputting planner-agnostic path corrections. In a bricklaying trial it reduced path irregularity by 39.2% and raised minimum time-to-collision; across seven Dynablox sequences it raised average minimum time-to-collision by 38.3% with shorter, smoother paths than the baselines.
Yilong Chen（First author）
Graph Neural Network Sim-to-Real Transfer Socially Aware Robot Navigation LiDAR-Based Dynamic Object Tracking Human-Robot Interaction
arXiv Oct 7, 2026
AdvSim2Real co-evolves tasks, injections, and a web executor inside a frozen web world model, raising a 4B agent's completion under an unseen frontier adversary by 33.6% relative
AdvSim2Real co-evolves a task curriculum, an injection adversary, and a web executor inside a frozen web world model, where the curriculum is rewarded for tasks the executor solves about half the time and the adversary only for flipping a judged success into a failure; the trained 4B executor raises clean completion on 150 web tasks from 74.89% to 81.33%, completion under three learned adversaries from 48.07% to 57.48%, and completion under the untrained Kimi-K3 from 23.00% to 30.72%.
AdvSim2Real co-evolves a task curriculum, an injection adversary, and a web executor inside a frozen web world model, where the curriculum is rewarded for tasks the executor solves about half the time and the adversary only for flipping a judged success into a failure; the trained 4B executor raises clean completion on 150 web tasks from 74.89% to 81.33%, completion under three learned adversaries from 48.07% to 57.48%, and completion under the untrained Kimi-K3 from 23.00% to 30.72%.
Massachusetts Institute of Technology
Sarim Hashmi（First author）
adversarial training World Model Prompt Injection LLM Agents Sim-to-Real Transfer
arXiv Oct 8, 2026
EmbodiedRSI uses a hypothesis graph and value-of-information experiment selection to lift a frozen VLA to 77.0% on RoboCasa365 and 86.8% on LIBERO-Pro, transferring zero-shot to a real robot at 71.3%
EmbodiedRSI is a self-evolving agentic harness that, around a frozen robot foundation model, maintains code and skill hypotheses in a Hypothesis Graph, uses Value-of-Information Experiment Selection to pick the physical experiments that best distinguish them, and turns each interaction into reusable improvement through Code-Skill Co-Evolution and Reward-Grounded Memory Learning; it reaches 77.0% overall success on RoboCasa365 and 71.3% on Composite-Unseen (best baseline 40.1%), 86.8% overall on LIBERO-Pro, and transfers the simulation-evolved harness zero-shot to a real robot at 71.3% overall success.
EmbodiedRSI is a self-evolving agentic harness that, around a frozen robot foundation model, maintains code and skill hypotheses in a Hypothesis Graph, uses Value-of-Information Experiment Selection to pick the physical experiments that best distinguish them, and turns each interaction into reusable improvement through Code-Skill Co-Evolution and Reward-Grounded Memory Learning; it reaches 77.0% overall success on RoboCasa365 and 71.3% on Composite-Unseen (best baseline 40.1%), 86.8% overall on LIBERO-Pro, and transfers the simulation-evolved harness zero-shot to a real robot at 71.3% overall success.
Columbia University Princeton University
Python Song（First author）
EmbodiedRSI Harness Evolution Self-Evolving Agents Embodied Memory Sim-to-Real Transfer
arXiv Sep 30, 2026
AIMS turns natural-language deployment requests into sim-to-real ISAC configurations via a two-agent framework, improving vehicle detection and beam prediction on DeepSense 6G
The authors propose AIMS, an agentic AI framework for sim-to-real multi-modal integrated sensing and communication (ISAC): given a natural-language deployment request specifying the target task, deployment conditions, and real-data budget, it derives a deployment-specific sim-to-real configuration and coordinates its execution to produce a deployment-specific task model, using a two-agent architecture for scene construction and task learning; on the real-world DeepSense 6G dataset it improves vehicle detection and beam prediction over the considered simulation and fusion baselines, and a separate orchestration benchmark shows improved plan correctness with structured domain knowledge and validation feedback.
The authors propose AIMS, an agentic AI framework for sim-to-real multi-modal integrated sensing and communication (ISAC): given a natural-language deployment request specifying the target task, deployment conditions, and real-data budget, it derives a deployment-specific sim-to-real configuration and coordinates its execution to produce a deployment-specific task model, using a two-agent architecture for scene construction and task learning; on the real-world DeepSense 6G dataset it improves vehicle detection and beam prediction over the considered simulation and fusion baselines, and a separate orchestration benchmark shows improved plan correctness with structured domain knowledge and validation feedback.
Yijie Bian（First author）
AIMS Integrated Sensing and Communication Agentic AI Sim-to-Real Transfer