A LiDAR detector's safety threshold may not hold in a new condition: a test returns applicable, not applicable or not determined
Synopsis
In safety-of-the-intended-functionality analysis under ISO 21448, an uncertainty threshold that keeps a LiDAR detector within a performance target under one condition had no procedure to check whether it still does under another. The threshold is transferred unchanged into the condition of use, and a statistical test decides at a stated error probability whether the target is applicable, not applicable or not determined, together with a lower bound on the independent units needed to show the target is met. On PointPillars ensembles over KITTI and nuScenes, counted in frames, the threshold transferred from nuScenes to KITTI was not applicable on all five splits; counted in recorded sequences, neither dataset held enough units to show any target limit met.
Interpretation
Transferring a threshold across conditions becomes a statistical decision with one of three explicit outcomes: applicable, not applicable or not determined at a stated error probability. Previously a threshold was selected on data of one condition, and its statistical guarantee held only for data distributed like that selection data, whereas safety-of-the-intended-functionality analysis covers every triggering condition. On PointPillars ensembles over KITTI and nuScenes, counted in frames, the threshold transferred from nuScenes to KITTI was not applicable on all five splits, with the same outcome at both exploratory limits.
An applicable outcome does not mean the detector works well, because a threshold can meet the limit by rejecting most correct detections. The false acceptance proportion counts only the share of wrong detections among accepted detections, so a condition with almost no accepted detections meets the limit trivially. At the target limit, every transferred threshold with an applicable outcome had a retention of at most 29 percent on every split, while thresholds selected in their own condition of use reached a median retention of at most 33 percent.
How much data is needed to show a limit is met can be computed before recording, which sets how many independent units each condition requires. The same data counted in frames and counted in recorded sequences give entirely different conclusions, because frames of one continuous recording are not independent. At the target limit, selecting a threshold alone requires 299 independent units and testing it 368; counted in sequences, the 150 nuScenes validation scenes fall short of that number, as do the 20 to 21 KITTI test drives.
Perspective
The results concern the LiDAR object detector as one system element, obtained on PointPillars deep ensembles over KITTI and nuScenes across changes of distance, weather and dataset; far distance and rain are treated only as potential triggering conditions, and the work does not show that they initiate a hazardous behaviour.
Counting frames overstates the evidence because frames of one recording are not independent, while counting sequences leaves too few units, and nuScenes scenes cut from one log are not independent either. The not applicable outcomes in rain appeared only at the exploratory limits, and clear and rain scenes also differ in location and traffic, so these changes accompany the outcomes without being shown to cause them.