Legal · Contract clause extraction

CUAD

9 models · 3 composite metrics

0.9 pts
Claude Opus 4.7's lead over #2
11 pts
spread across all nine models
Overview

What this dataset measures

Clause extraction across 41 legal provision types from real commercial contracts. The Lab uses 150 rows from the prepared CUAD subset.

Contract review is a high-frequency legal AI workflow. This dataset asks whether a model can locate and return the right clause span, not whether it can summarise the agreement.

Where this matters:Contract review, clause extraction, M&A due diligence
Results

Composite ranking across 9 models

Composite score is the unweighted mean of 3 active metrics: Numeric match, Text similarity (F1), LLM judge accuracy.

Groundedness and hallucination proxy sit near 100% for all models on extraction tasks. They add no signal and are excluded from the composite.

Claude Opus 4.7
66.4%
Claude Opus 4.8
65.5%
Claude Fable 5
62.5%
Claude Haiku 4.5
61.4%
Claude Sonnet 4.6
60.3%
GPT-5
58.7%
GPT-5 mini
58.0%
Groq Llama 4 Scout
55.9%
Groq Llama 3.3 70B
55.1%
Metric breakdown

How models differ across scoring dimensions

Scores below are the metrics that feed the composite for this dataset.

Numeric match
Claude Opus 4.781.3%
Claude Opus 4.884.7%
Claude Fable 581.3%
Claude Haiku 4.582.0%
Claude Sonnet 4.682.7%
GPT-584.0%
GPT-5 mini83.3%
Groq Llama 4 Scout74.7%
Groq Llama 3.3 70B76.7%

Unweighted mean across test rows. Measures whether extracted numbers match the reference answer.

Text similarity (F1)
Claude Opus 4.760.6%
Claude Opus 4.857.1%
Claude Fable 555.5%
Claude Haiku 4.552.9%
Claude Sonnet 4.651.0%
GPT-548.1%
GPT-5 mini47.3%
Groq Llama 4 Scout49.0%
Groq Llama 3.3 70B46.7%

Unweighted mean across test rows. Token-level F1 between model output and reference string.

LLM judge accuracy
Claude Opus 4.757.3%
Claude Opus 4.854.7%
Claude Fable 550.7%
Claude Haiku 4.549.3%
Claude Sonnet 4.647.3%
GPT-544.0%
GPT-5 mini43.3%
Groq Llama 4 Scout44.0%
Groq Llama 3.3 70B42.0%

Unweighted mean across test rows. LLM judge scores answer correctness against the reference.

Failure analysis

Where models go wrong

Representative examples of where models struggle on this benchmark. Each pattern shows a common error type drawn from the evaluation set.

Partial extraction

The model identifies that a clause exists but extracts only part of the relevant text, missing conditions, exceptions, or carve-outs that change the legal meaning. The correct answer for this example is NONE: the clause does not exist, but the model returns text it incorrectly treats as responsive. This is a false positive on clause presence, the most operationally dangerous error type in contract review.

Watch for: False positives on clause existence are harder to catch than missed clauses. A reviewer who trusts the extraction may miss the fact that no responsive clause exists. For M&A diligence workflows, validate specifically on negative examples where the correct answer is NONE.

Example row
Notice Period To Terminate Renewal: highlight the parts of the contract that relate to this clause type.
Expected
NONE
Absent clause: false text

When a clause type is absent from the contract, models sometimes return a date, name, or phrase that appears nearby in the document rather than returning no answer. The effective date of the agreement is present; the specific clause being searched for is not. The model confuses document presence with clause presence.

Watch for: This failure produces confident-looking wrong answers. A model returning "1 August 2011" for a missing clause type looks correct to a reviewer who does not check the source. Spot-check negative clause results on a sample before relying on systematic extraction at scale.

Example row
Effective Date: highlight the parts of the contract that relate to this clause type.
Expected
1 August 2011
Wrong clause span

The model locates the correct section of the contract but extracts text from the wrong clause within it. Renewal term provisions often sit adjacent to termination provisions. The model returns the wrong span, producing an answer that is textually plausible but legally incorrect.

Watch for: Adjacent clause confusion is most common in contracts with dense boilerplate sections where multiple related provisions appear in sequence. If your workflow covers renewal, termination, and notice clauses, test all three together on the same contracts, not in isolation.

Example row
Renewal Term: highlight the parts of the contract that relate to this clause type.
Expected
NONE
Takeaways

What this means for deployment

01
Why these results look the way they do

Claude Opus 4.7 leads at 66.4% composite across 150 CUAD tasks, with a fleet average of 60.4%. CUAD tests two distinct capabilities: whether a clause exists (presence detection) and what it says (span extraction). Models that score well on one do not always score well on the other. The composite score masks this split. For contract review deployment, presence accuracy and extraction accuracy should be measured separately.

02
Where the leader stands

Claude Opus 4.7 leads on composite (66.4%) with numeric match at 81.3%, context groundedness at 74.0%, and LLM judge accuracy at 57.3%. The gap between numeric match and LLM judge accuracy is meaningful: the model locates the right part of the contract more reliably than it extracts a clean answer string. For clause presence tasks, it is strong. For precise span extraction, there is meaningful variance.

03
Metric interpretation

Hallucination proxy at 100% means models added no unsupported text beyond the contract content. This is expected for extraction tasks where the answer must come from the document. Text similarity at 60.6% reflects that extracted clause spans vary in length and phrasing even when they cover the correct provision. LLM judge accuracy at lower levels on some models signals that the extracted span is in the right area but not precisely bounded.

Interactive charts

Cost & workflow analysis

Claude Opus 4.7 leads on composite (66.4%); several alternatives trade a few points of accuracy for much lower run cost.