AIMS WORKSHOP · COLM 2026

LegalScope

Measuring Exam-to-Case Transfer
in LLM Legal Reasoning

How much does success on a legal exam tell us
about reasoning through a real case?

Hongyu Wang¹ Rilyn R. Han² Yilun Zhao^2,* Xuandong Zhao³ Arman Cohan²

¹ UC Santa Barbara² Yale University³ UC Berkeley

* Corresponding author

Read the paper GitHub Hugging Face

Downloadable now: 78 source records + benchmark metadata · Data availability and release notes

861Legal-exam questions

276Issue–stance prompts

28Model groups

56De-identified judgments

THE RESEARCH QUESTION

Abstract

Public legal exams are widely used to evaluate legal reasoning in large language models, yet it remains unclear whether exam performance reflects the ability to reason over real-case facts. We introduce LegalScope, a dual-track benchmark that pairs 861 legal-exam questions with 276 lawyer-reviewed issue-stance prompts derived from de-identified Chinese judgments. LegalScope evaluates models under both reference-answer scoring and case-based rubric scoring, enabling direct analysis of exam-to-case transfer.

Across 28 model groups, public-exam scores correlate with real-case scores but do not fully predict them; citation relevance receives lower mean scores than argument validity under both automatic and lawyer evaluation. We further show that automated evaluation aligns strongly with human review on exam answers but less reliably on real-case legal analysis, highlighting the need for expert-grounded case evaluation.

Explore the workshop paper

BENCHMARK

Benchmark overview

Two tracks, evaluated on the same 28 model groups.

PUBLIC EXAMS861

Questions with a reference answer.

Legal-exam questions from four jurisdictions, scored for substantive consistency with the reference answer on a 0–4 scale.

United States: 603
China: 94
United Kingdom: 86
Australia: 78

REAL CASES276

Arguments on both sides of an issue.

138 legal issues from 56 de-identified Chinese judgments. Each issue becomes a supporting and an opposing prompt, preserving a closed factual record.

SupportOppose

Seven categories: Tort, Contract, Criminal, Intellectual Property, Administrative, Civil Procedure and Property.

LegalScope pipeline: source collection and de-identification; two-track evaluation across 28 models; independent human validation; and reliability audits. — **Figure 1.** Benchmark construction and evaluation. Historical controls and fixed-answer audits retain their own evaluation pools.
View full-resolution figure ↗

Citation relevance

Does the selected authority respond to the issue?

Constraint extraction

Does the response follow the stance, facts and task constraints?

Argument validity

Does the conclusion follow from the law and the supplied facts?

The real-case rubric adapts the three evaluation dimensions of CourtReasoner (Han et al., EMNLP 2025) to closed-book Chinese case analysis.

Task examples

Read the input, inspect an answer, and see what the scoring protocol measures.

AUSTRALIA · VICTORIAN BAR · EXTVICBAR01A

Barrister confidentiality

Reference-answer scoring

Facts supplied in the exam

A barrister is representing a celebrity client who issued proceedings for breach of contract as a result of sexual harassment from their employer. During the course of the retainer, the client informs the barrister that the client is considering divorcing their spouse.

The question

(a) Could the barrister disclose the information about the potential divorce to other people? Why or why not? [1 mark]

What LegalScope evaluates

Whether a model reaches a compatible conclusion and explains the decisive legal point: here, the duty of confidentiality and its application to information obtained during the retainer. Evaluation compares substance with the reference; it does not require identical wording.

CHINESE JUDGMENT · PAPER EXAMPLE RV038

Supporting a refusal of re-appraisal

Closed-book · Supporting stance

The given record

In a medical-malpractice dispute, an appraisal had already been issued. The plaintiff challenged it and applied for re-appraisal. The court refused that request and relied on the existing appraisal materials.

The prompt does not supply the detailed appraisal procedures or the specific medical records reviewed.

The model’s task

Write a two-paragraph Chinese legal analysis supporting the court’s refusal to grant re-appraisal. Use only the supplied facts.

First summarize the relevant facts and dispute; then apply the legal authorities. Do not add medical records, appraisal defects, treatment details or outside case information.

Evaluator reference

The paper’s scoring packet identifies Article 40 of the Supreme People’s Court provisions on civil evidence: dissatisfaction with a conclusion alone does not establish grounds for re-appraisal. The answer must connect the applicable threshold to the facts actually supplied.

Abbreviated research example from the current manuscript’s RV038 scoring packet and case study. No original judgment or identifying case details are reproduced.

03 Actual model answer

Qwen3 235B A22B · Stored English answer excerpt

In this case, the original appraisal was made by an appraisal institution with statutory qualifications after lawfully accepting the commission; during the appraisal process, all outpatient and emergency medical records, electrocardiogram materials, and emergency treatment records were reviewed, and the compliance of the diagnosis and treatment conduct, the sufficiency of the risk disclosure, and causation were analyzed item by item, with complete procedures and sufficient basis.

Highlighted assertions were not supplied in the prompt.

Citation relevance4/4

Constraint extraction1/4

Argument validity3/4

Citation · 4: The answer cites relevant appraisal authority.
Constraint · 1: It adds appraisal procedures and medical evidence absent from the closed record.
Argument · 3: Its re-appraisal reasoning remains usable, despite exceeding the supplied facts.

Recorded A/B/C scores from the paper. Highlighting and explanatory labels are added for this web presentation.

Why three scores matter

A plausible legal argument can still rely on invented factual support. Separate scores make that boundary violation visible instead of hiding it inside a single overall rating.

RESULTS

Main results

Read the full results summary

0.817

Exam–case association

Model-level Pearson correlation for automatic scores; Spearman correlation is 0.708. Rankings and variant gains do not transfer uniformly.

66.9 / 72.6

Citation / argument

Mean automatic scores on a 0–100 scale. Lawyers show the same direction descriptively; the pooled lawyer interval crosses zero.

0.910 / 0.312

Exam / case judge agreement

Answer-level Pearson correlations between automatic and human scores. Case scores use the equal-weight mean of two lawyers.

Best in columnSecond best in columnAll 28 model groups · ties share rank

Loading published results…

LegalScope model scores. Best and second-best displayed values are marked separately within each column over the complete roster.
	Automatic evaluation					Human evaluation

All scores are on a 0–100 scale. Automatic scores cover the full tracks. Human scores cover 80 exam items and 10 case prompts; case scores pool two lawyers. Best and second-best marks use the displayed one-decimal values across all 28 groups; equal values share a mark, and filtering does not change their rank. The initial order follows the paper. CSV/Parquet additionally include the two descriptive overall means. Highlights are descriptive, not significance tests. Scores do not equate difficulty across tracks.

Explore the score distributions and transfer plots

Per-model score distributions for 861 exam prompts and 276 case prompts across all 28 groups. — Answer-quality distributions, not repeated-scoring stability.

Automatic exam-to-case scatterplot with Pearson correlation 0.817 and Spearman correlation 0.708. — Automatic exam–case association over 28 model groups.

Human exam-to-case scatterplot with Pearson correlation 0.696 and Spearman correlation 0.690. — Human exam scores and pooled lawyer case scores. This is not an automatic–human agreement plot.

RESOURCES

Paper, code and data

Research resources and current release information.

Paper

Method, full experimental results, rubric calibration and expert validation.

OpenReview 02

Repository

Documentation, original paper figures, aggregate results and workbook helpers.

GitHub 03

Data and source subset

78 licensed source excerpts and selected candidate answers, model results, benchmark composition and dataset card.

Hugging Face

Data availability. Hugging Face provides 78 Victorian Bar source-excerpt records and selected candidate answers under CC BY-NC-ND 4.0, alongside metadata and aggregate results. The four dataset configurations load independently; original CSV and JSONL files are also available. Source excerpts preserve attribution and license notices and are not exact replacements for the paper’s evaluation prompts. The full 861-item exam track, the 276-prompt case dataset, full model responses and lawyer review sheets are not included. Case prompts and answers are being considered for item-level source and privacy review; this is not a blanket prohibition on publishing anonymized research material. These materials cannot rerun the complete evaluation. Source-specific release details · Loading examples, fields and answer types.

CITATION

Citation

AI Measurement Science Workshop
at COLM 2026

@inproceedings{wang2026legalscope,
  title = {{LegalScope}: Measuring Exam-to-Case
           Transfer in {LLM} Legal Reasoning},
  author = {Wang, Hongyu and Han, Rilyn R. and
            Zhao, Yilun and Zhao, Xuandong and
            Cohan, Arman},
  booktitle = {AI Measurement Science Workshop
               at COLM 2026},
  year = {2026},
  url = {https://openreview.net/forum?id=BNx62Wx1ej}
}