Simulating stochastic hybrid systems no longer stops at discrete choices: discretized time intervals let Q-learning pick optimal schedulers on continuous models

Simulation of stochastic hybrid systems now handles time nondeterminism, not only discrete nondeterministic choices. Earlier simulation-based learning covered discrete nondeterministic choices only; time nondeterminism is harder because a single state already contains infinitely many possible choices. On the water tank benchmark the simulated maximum reachability probabilities agree with those of the existing PLT method, and all confidence intervals are consistent with the PLT results except one configuration attributed to insufficient training.

The approach abstracts the continuous model into finite state and action spaces, then lets Q-learning choose the scheduler within them. Q-learning needs finite state and action spaces, so continuous states are binned at a fixed granularity and each action's time interval is discretized at the same granularity. Learning runs on the discrete abstraction while every simulation still executes on the original continuous model; on the running example the simulated values match the analytically computed expected maximum reachability probabilities within the confidence interval width.

Symbolic precomputation yields a reachable-state tree that terminates simulation runs which cannot reach the goal. Guidance does not always pay off: in the original water tank model, stochastic blocking times dominate the reachability probability, so guidance prunes only a small fraction of the search space. In the water tank variant with time nondeterminism and a low-probability underflow objective, guided simulation clearly outperforms unguided simulation for a reasonable number of training runs and gives a reachability probability of at least 45% at granularity 0.05 with one billion training episodes.

The demonstrated scope is the class of stochastic hybrid models called singular automata with random clocks, with the simulator integrated into RealySt and supporting configurable learning parameters, discretization settings and two scheduler classes. Evaluation covers a running example, the original water tank benchmark, and a water tank variant with time nondeterminism and an underflow objective. Future work points to more expressive classes of stochastic hybrid models, more efficient symbolic representations and pruning, and generalizing guided simulation to minimum reachability probabilities.

The benefit of guidance depends strongly on the property: in the original water tank model guidance brings no speed-up because stochastic blocking times dominate the reachability probability, and reach tree construction scales poorly. The underflow variant needs a fine granularity, with the Q-table reaching 12.17 million states at granularity 0.05 and one billion training episodes needed to obtain the estimate of at least 45%, while unguided simulation is infeasible at that scale. One configuration's deviation is attributed to insufficient training, which the authors note exposes a limitation of purely simulation-based approaches.