Nagham Omar

The work introduces the Stability-Aware Generalization Objective (SAGO), which defines LLM generalization as per-instance behavioral stability of the same input under semantically equivalent variations, and evaluates eleven open- and closed-source models across six datasets on four behavioral axes—generation consistency, internal activations, confidence, and response mirroring—finding that every model exceeds a significance threshold on at least one axis, that the axes capture independent failure modes, and that cross-dataset variation can reverse model rankings.

The work introduces the Stability-Aware Generalization Objective (SAGO), which defines LLM generalization as per-instance behavioral stability of the same input under semantically equivalent variations, and evaluates eleven open- and closed-source models across six datasets on four behavioral axes—generation consistency, internal activations, confidence, and response mirroring—finding that every model exceeds a significance threshold on at least one axis, that the axes capture independent failure modes, and that cross-dataset variation can reverse model rankings.

The authors built MIST (the Misleading-Image Stress Test), 200 English sentences each containing a phrase readable either figuratively or literally and shown with an aligned image, a misleading image, or no image; across thirteen VLM judges an aligned image changed 20.5% of labels and a misleading one 19.4%, versus 11.6% when only the ignore-the-image instruction was deleted with the image left in place, and only 37% of the labels that differ between the two images moved toward the sense shown, while agreement with the human annotators was unchanged whether the image was absent, aligned, or misleading.

The authors built MIST (the Misleading-Image Stress Test), 200 English sentences each containing a phrase readable either figuratively or literally and shown with an aligned image, a misleading image, or no image; across thirteen VLM judges an aligned image changed 20.5% of labels and a misleading one 19.4%, versus 11.6% when only the ignore-the-image instruction was deleted with the image left in place, and only 37% of the labels that differ between the two images moved toward the sense shown, while agreement with the human annotators was unchanged whether the image was absent, aligned, or misleading.