Connectome-constrained fly vision model put through stress tests: mechanistic claims shift with intervention and trained solution

Mechanistic conclusions from a connectome-constrained model change with the comparison model, the intervention, the circuit representation and the trained solution, so such claims must be scoped to a specific model family, intervention, representation and trained solution. Connectome-constrained networks were previously used to generate predictions about circuit function and behaviour, with the biological constraint treated as support for mechanistic conclusions; the question here becomes how much a conclusion moves when the comparison and intervention change. The stress test runs on the 50 pretrained flyvis models, task-matched causal CNN controls and replicated interventions, and the dependence appears at both the behavioural and the mechanistic level.

Reverse-phi direction reversal in flyvis is strongly period- and velocity-dependent and also appears in task-matched CNNs, and neither family reproduces the fly's turning response to stationary sawtooth gradients. Reverse-phi was previously treated as a prediction of correlation-type motion detectors and used to separate bright and dark pathway contributions; here it is neither uniform nor unique to the connectome-constrained family. Under the matched three-level reverse-phi stimulus, reversal counts across the 50 flyvis models range from 14 to 46 with velocity, while all five small and all five medium CNN checkpoints reverse at 65.7 degrees per second; the stationary-sawtooth criterion is met by 3 of 50 flyvis models per polarity and by no CNN.

Attribution methods agree with exhaustive single-connection ground truth in effect size only when evaluated under the intervention they were built for, while identification of important connections is far less sensitive to that baseline. Circuit discovery was previously evaluated against planted circuits or faithfulness proxies; here the ground truth is exhaustive single-component intervention, with zero, held-out-mean and counterfactual replacement treated as distinct causal questions. For held-out-mean replacement, matched first-order edge attribution patching reaches pooled AUROC 0.970 and Spearman 0.872, versus 0.933 and 0.126 for its zero-aligned score against the same ground truth; AUROC stays above 0.91 in every row while precision at 20 ranges only from 0.44 to 0.66.

Only 25 of 50 models meet a model-relative T4/T5 tuning criterion, and the connection sets derived from those cell-type solutions survive pruning but mostly fail an added two-sided response-amplitude check. Connectome-constrained models were previously presented through circuit conclusions from a single trained solution; here the same criterion is met unevenly across the released ensemble and shifts with intervention granularity and representation. Greedy cell-type search reaches the criterion within 15 additions in 25 of 50 models after a median of 7 additions; expanding the six first-success type sets and pruning removes 74 to 91 percent of added connections and leaves 13 to 31, yet only models 003 and 005 still meet the threshold after the two-sided amplitude check.

The stress test covers one architecture, flyvis, and its released ensemble of 50 models, with behavioural controls being causal CNNs matched on task, Sintel rendering and target, loss and input lattice, and mechanistic analyses run at both the 604 type-to-type connection classes and the cell-type level where single-component interventions can be computed exhaustively. It suits researchers using or evaluating connectome-constrained models who need to know which qualifications a mechanistic claim carries. The released package does not retain the models' launch-time training split or the exact generator of its packaged task-error scalar, so the common evaluation is descriptive rather than a shared held-out generalization comparison.

Subtracting each frame's spatial mean makes the three-level reverse-phi stimulus equal to a two-level contrast-inverting pattern at half contrast, so the change in reversal counts cannot be assigned to the polarity structure, the halved moving-pattern contrast or the added flicker individually. The T4/T5 tuning criterion is model-relative, its absolute-value denominator saturates and it places no upper bound on amplitude, so meeting it is not physiological sufficiency. Output silencing also removes eight of the decoder's 34 input cell types, so dependence on the recurrent circuit cannot be separated from loss of direct decoder input. The pruned endpoints are greedy, one-step irreducible sets rather than globally minimal circuits, and 455 of 615 pruning steps were numerically ambiguous at the frozen scale.