Industrial robot evaluation stops resetting every episode: HolidayFactory 1.0 scores seven factory tasks on persistent scene state
Synopsis
Industrial robots are usually evaluated in episodes that begin from a reset, while factory work is a chain of jobs whose outcomes shape the next one. HolidayFactory 1.0 combines simulation, world construction and evaluation: Holiday-Sim adds constitutive laws, a semi-implicit primal-dual solver and stateful actuators to Newton's sparse-SDF hydroelastic contact, an agentic pipeline generates and verifies assets and scenes, and 24/7-Bench scores seven industrial tasks without resetting the scene, counting completed, failed and unfinished jobs, throughput, latency and motor energy. Evaluation-generation experiments show that passing executable checks does not establish agreement with task intent.
Interpretation
Industrial robot evaluation moves from a single episode after a reset to repeated jobs whose scene state persists between them, recording completed, failed, timed-out and unfinished jobs, throughput, job latency and motor energy. Existing simulation benchmarks mostly target homes and everyday activities, such as BEHAVIOR-1K, RoboCasa and RoboCasa365, and each episode starts from a reset, so the state left by one job cannot affect the next. 24/7-Bench defines seven industrial tasks covering conveyor loading, flow-rack loading, cart push, jig loading, press tending, valve operation and barcode routing, with job outcomes and run-level verdicts taken from privileged simulator state.
On the physics side, contact, actuation and tactile sensing advance within one state update, so interactions that decide whether a job succeeds, such as seating a part in a tight pocket, holding a grasp or delivering motor effort, can be represented. Representative benchmarks use convex decomposition with sparse point contacts, which can fill narrow features and collapse distributed loads onto a few points, and their actuators are mostly ideal torque or PD drives without internal state. Holiday-Sim extends Newton's sparse-SDF hydroelastic contact with compression hardening, compression-dependent damping and speed-dependent friction, resolves stiff coupled contacts with the semi-implicit primal-dual solver CDMM, and couples motor current, temperature, transmission and LuGre friction states into articulated dynamics.
World construction is done by coding agents, and assets, scene variants and evaluation proposals must pass executable checks, with evaluation proposals confirmed by a person before compilation. Assets from 3D repositories or generative models offer broad semantic and visual coverage but are not verified against a particular task, and scene construction lacked a way to carry task constraints into the generation loop. Verification rejects 47% of 72 assets in the first round, with conformance accounting for 39%; in a 32-task revision study, measurement-grounded revision repairs 11 of 12 first-round failures in one round and reaches 100% final success, against 84% for independent regeneration and 81% for coarse pass/fail feedback.
Perspective
The demonstrated scope is seven industrial tasks in simulation, one wheeled humanoid robot, Friday, and the contact, actuator and tactile models of Holiday-Sim; the contact and solver studies compare only with numerical quadrature of the same pressure law or with another backend, not with hardware measurements. The intended readers are researchers and engineers in robot simulation, benchmark design and industrial manipulation.
The physical models have not been compared with measurements from a physical robot, and their effect on policy rankings is unmeasured; no policies have yet been compared on the seven tasks, so the number of runs needed to separate policies is unknown. The evaluation-generation study covers one task, one scene variant and 15 reused recordings, and the simulated user does not establish the reliability of human confirmation.