St. Edward's University

This work introduces information satisfaction—a query paired with a reader persona—as an axis of summarization evaluation, and through five perturbation tests plus an expert human evaluation finds that traditional metrics such as ROUGE and BERTScore and LLM-as-judge metrics such as Llama-3.3-70B and Prometheus-7B mostly fail basic perturbation checks and agree with reader preferences at near-chance levels, indicating that existing metrics are insufficient measures of how well a summary serves a specific reader's informational needs.

This work introduces information satisfaction—a query paired with a reader persona—as an axis of summarization evaluation, and through five perturbation tests plus an expert human evaluation finds that traditional metrics such as ROUGE and BERTScore and LLM-as-judge metrics such as Llama-3.3-70B and Prometheus-7B mostly fail basic perturbation checks and agree with reader preferences at near-chance levels, indicating that existing metrics are insufficient measures of how well a summary serves a specific reader's informational needs.