Surgical skill scores stop being tied to one rubric: PRA reads text descriptions, lifting macro-F1 by 0.113 over Video Swin-T
Synopsis
Automated surgical skill scoring has depended on a fixed rubric, so a new set of similarly worded assessment items meant retraining. Predictive Rubric Alignment (PRA) instead predicts a rater-weighted combination of the embeddings of each level's written description, then projects that prediction onto a path joining adjacent descriptions to read out a score. On AIxSuture suturing videos, the PRA-item variant with learned item queries reached 0.848 accuracy and 0.829 macro-F1, 0.113 macro-F1 above Video Swin-T, and PRA-cond trained on only two items still gave positive score correlations on unseen JIGSAWS items.
Interpretation
Automated scoring of surgical video can now follow assessment criteria written in natural language, instead of locking its output to the levels or numbers used at training time. Earlier automated methods typically predicted predefined skill levels or regressed numerical scores tied to a fixed rubric, which limited transfer once the wording of assessment items changed. On the 314 five-minute suturing videos of AIxSuture, PRA-item reached 0.848 accuracy and 0.829 macro-F1, 0.113 macro-F1 above Video Swin-T and 0.137 above I3D, and led on accuracy, macro-F1 and every class-wise F1.
The gain concentrates on the intermediate class, trainees whose item scores mostly fall between the described levels. The intermediate class was previously the hardest of the three, with F1 of 0.538 for Video Swin-T and 0.403 for I3D. PRA-item reached 0.703 intermediate-class F1; PRA-cond, which has no item-specific parameters, gained 0.042 macro-F1 and 0.060 intermediate-class F1 over the equally parameter-free BCE path readout.
Scores are no longer obtained by comparing similarities; the model directly predicts a point in the space of description embeddings. Contrastive methods convert cosine similarities into level probabilities through softmax or sigmoid, and the temperature parameter's value affects learning and downstream performance. PRA linearly interpolates the rater weights on adjacent level descriptions into a target vector, supervises the video model to predict it by squared distance, and at inference returns the score of the nearest point on the description path, so no temperature is tuned; after retraining with reset temperatures, the best CE-8 seed at any temperature stayed below the worst PRA-item seed.
PRA-cond, which generates queries from item names, can take a new rubric as text without retraining. PRA-item learns one query vector per item and is therefore tied to the items seen in training. Trained on only the AIxSuture instrument-handling and knowledge-of-specific-procedure items, PRA-cond produced positive score correlations on six unseen JIGSAWS items, and its Grad-CAM overlapped instruments and suture more than the description-matching baseline.
Perspective
The evaluation covers 314 open-suturing videos from AIxSuture and the bench-top da Vinci Suturing and Knot Tying tasks of JIGSAWS, scored on OSATS and modified-OSATS items; the authors note limits from dataset size and rubric diversity and plan to expand training and evaluation across more surgical tasks and rubrics.
In zero-shot transfer PRA-cond is not the most accurate model: its mean correlation is lower than BCE with probability readout and varies more across seeds, and all models rank below a trial-duration baseline; the authors interpret the higher Grad-CAM overlap as a possible effect of the predictive objective preserving stronger dense features.