EViSE scores summaries on faithfulness, coherence and relevance in one encoder pass, reaching 0.806 overall Spearman on held-out News
Synopsis
EViSE splits summary evaluation into faithfulness, coherence and relevance and returns all three scores from a single non-autoregressive encoder pass, training score regression together with a within-source ordering constraint over candidate summaries. On 80,856 human-reviewed Vietnamese document–summary instances, EViSE-mmBERT reaches an overall Spearman correlation of 0.806 on the held-out News test set and the compact EViSE-ViDeBERTa reaches 0.737; adding the ranking term to the same architecture raises the three criterion Spearman values by 0.045 to 0.061 over regression alone.
Interpretation
Automatic summary evaluation moves from one aggregate score to three criterion scores, all produced in a single encoder forward pass with no reference summary at inference. The widely used ROUGE and BERTScore return one reference-based overlap or semantic-similarity number and do not separate factual support, discourse quality and content selection; prompted large-model judges instead need repeated autoregressive calls. On the held-out News test set of 1,348 documents and 8,088 summaries, EViSE-mmBERT reaches overall Spearman 0.787 and Pearson 0.870, and EViSE-ViDeBERTa reaches 0.721 and 0.827, while on the same 6,740-pair comparable subset ROUGE-Lsum gives 0.459 and 0.580 and BERTScore-F1 gives 0.357 and 0.506.
The training objective ties absolute score regression to relative ordering among summaries of the same source, so the model both tracks human scores and preserves the quality order within one document. With regression alone, the model is only asked to approach each score individually, with no explicit constraint on the order of candidates from the same source. Under the fixed ViDeBERTa configuration, adding the ranking term raises Spearman for faithfulness, coherence and relevance from 0.646, 0.570 and 0.674 to 0.697, 0.615 and 0.735, and lowers MAE from 0.120, 0.121 and 0.127 to 0.104, 0.104 and 0.109.
The ranking loss is normalized within each source–criterion group, averaging over eligible summary pairs inside a group before averaging across all sources and the three criteria, with an empty pair set contributing zero. Averaging all summary pairs together would give a source with more unequal-label pairs a larger ranking weight purely because it contributes more pairs. For two eligible sources with 5 and 14 pairs and ordering errors of 1 and 0, pooling gives 5/19 while an equal-weight source mean gives 1/2; each training micro-batch holds all six candidate summaries of one source.
Perspective
Evaluation covers the held-out News test set of 1,348 documents and a cross-domain benchmark of 900 IT-textbook summaries, with a separate 150-pair blind subset for the compact model; computational cost is measured on one Tesla T4 with FP32 and batch size 4 in a matched profile.
The theoretical mean alignment and exact-label optimality assume freely adjustable prediction coordinates, and shared parameters can couple group offsets with ranking updates; cross-domain coverage is one technical-educational domain, independent blind validation is available only for the compact model, and multilingual effectiveness remains to be evaluated.