Three agents argue a Supreme Court case: precedent retrieval lifts criminal petitioner Legal-sim to 0.8921

Courtroom argument generation no longer starts from case facts alone: three independent LLM agents play judge, petitioner and respondent in turn, producing two dialogues, judge with petitioner and judge with respondent. Earlier judgment prediction work reasoned mainly over case facts or used a single-agent model, unlike the three-party argumentative exchange of a real courtroom. Experiments draw on public U.S. Supreme Court documents from 2000 to 2019, sampling 450 documents, 150 each from criminal, civil and constitutional cases, with 100 for training and the rest for testing; argument generation uses GPT-4o and Llama-3.1-70b, and results reported are from GPT-4o.

Few-shot examples for the agents are not chosen at random: each test document first retrieves the training document with the most similar facts, then the utterances closest to the case facts are placed in the prompt. The comparison is a fact-based conversation given only case facts, and a precedent similarity maximization strategy that picks one training document globally per case category. On criminal petitioner arguments in the multi-agent setting, local precedent retrieval scores 0.5682 BERTScore, precedent similarity maximization 0.5267 and fact-based conversation 0.5123; in the same multi-agent setting, local precedent retrieval reaches 0.8921 Legal-sim and 0.4782 BLEURT, against 0.8288 and 0.4381 for fact-based conversation.

The generated arguments are passed into a judgment prediction task to test whether they carry usable legal reasoning rather than surface similarity alone. Zero-shot supplies only case facts, two-shot supplies two randomly chosen training examples, and chain-of-thought asks the model to write reasoning steps before the verdict. Across 450 documents, local precedent retrieval reaches 0.55 judgment prediction accuracy, chain-of-thought 0.53, two-shot 0.50 and zero-shot 0.48; class F1 is 0.52 and 0.58 for local precedent retrieval versus 0.45 and 0.50 for zero-shot.

The demonstrated scope is public U.S. Supreme Court documents from 2000 to 2019, with 450 documents sampled across criminal, civil and constitutional cases, and GPT-4o as the reported generation model; the authors propose extending to broader legal domains, richer legal knowledge and precedents, and human-in-the-loop evaluation.

The authors report that lexical metrics ROUGE-L and METEOR score typically below 0.2, indicating substantial surface-level mismatch between generated and real arguments, while semantic metrics score higher and are read as semantic alignment rather than verbatim reproduction. Expert ratings use a 0 to 4 scale with weighted Cohen's kappa for agreement, and cases whose ratings span more than two points are adjudicated in a consensus session. The results section is internally inconsistent about single- versus multi-agent performance, stating in one place that single-agent models outperform multi-agent ones across all metrics and in another that multi-agent setups consistently enhance performance, so this report does not treat either ranking as settled.