Surgical scene captions learn from retrieved cases: GRAC reaches 52.6% exact instrument-action-target match on EndoVis-18

Frame-level descriptions of surgical video no longer rely only on language patterns stored in model parameters; the system first retrieves graphs and captions of similar surgical activities from the training set and generates the current description from them. Earlier scene-graph captioning methods mainly learned a direct mapping from scene representation to natural language, which is hard when annotated training data are limited. On the combined EndoVis-18 benchmark, GRAC reaches a CIDEr-D of 5.88, against 4.23 for SGT and 3.36 for the strongest conventional Transformer baseline; exact triplet-set accuracy is 52.6%, against 32.7% for SGT and 26.2% for the strongest conventional baseline.

Retrieval is driven by surgical activity rather than visual appearance: caption-supervised contrastive learning pulls graphs with similar activities together during training, and inference retrieves three graph-caption pairs by embedding similarity. Directly copying the top-ranked retrieved caption yields a CIDEr-D of 5.33 and exact triplet-set accuracy of 46.1%, below GRAC's 5.88 and 52.6%. Candidates are ranked by similarity, and the highest-ranked scene from each distinct triplet set is kept until three graph-caption pairs are selected; candidates from the query's own surgical sequence are excluded during training to prevent self-retrieval.

Before generation, the model matches instruments between the current graph and each retrieved graph by class, compares their node representations and instrument-tissue edge features, and encodes the differences and agreements into the retrieved caption tokens passed to the decoder. Graph comparison without compatibility supervision gives a CIDEr-D of 5.74, below 5.81 for caption retrieval alone; adding compatibility supervision gives 5.88 and lowers instrument hallucination from 0.049 to 0.037. The drop in instrument hallucination is significant in the paired frame-level analysis (p=0.003), while the increments over caption retrieval in CIDEr-D, exact triplet-set accuracy and triplet F1 are not statistically significant (p=0.054, 0.21 and 0.23 respectively).

Evaluation covers the 392-frame held-out EndoVis-18 benchmark and the 36-frame TORS test set, with annotated scene-graph features as input and single-frame descriptions rather than complete operative reports as output. The authors propose future evaluation on automatically predicted scene graphs and more diverse surgical procedures, with expert assessment of factual accuracy and clinical relevance.

The TORS result is a within-dataset replication after independent training on that dataset, not zero-shot transfer from EndoVis-18, and it does not prove robust generalization to independent clinical populations; its reference captions concentrate on recurring activities and the training set is likewise small. The authors also report that when the top-ranked retrieved caption has the correct instruments but a wrong action, GRAC reproduces that action in 54% to 71% of the examined frames.