LegalScope

Public legal exams are widely used to evaluate legal reasoning in large language models, yet it remains unclear whether exam performance reflects the ability to reason over real-case facts. We introduce LegalScope, a dual-track benchmark that pairs 861 legal-exam questions with 276 prompts derived from deidentified Chinese judgments and reviewed by lawyers. LegalScope evaluates models under both reference-answer scoring and case-based rubric scoring, enabling direct analysis of exam-to-case transfer.

Across 28 model groups, public-exam scores correlate with real-case scores but do not fully predict them; citation relevance receives lower mean scores than argument validity under both automatic and lawyer evaluation. We further show that automated evaluation aligns strongly with human review on exam answers but less reliably on real-case legal analysis, highlighting the need for expert-grounded case evaluation.