LegalBench (contracts)

The question asks for a Yes or No answer about clause presence. The model answers the wrong question, returning a clause description, a quoted excerpt, or a conditional answer instead of the binary classification required. The correct answer is Yes; the model produces text rather than a classification.

Watch for: Task format compliance is a prerequisite for accuracy on binary classification benchmarks. A model that answers the right question in the wrong format scores zero regardless of whether its underlying reasoning is correct. Test models on format compliance explicitly before evaluating classification accuracy.

The model classifies the clause type incorrectly, returning No when the clause is present or Yes when it is absent. For termination for convenience clauses, which are common in commercial contracts, a false negative means a reviewer misses a material provision. This is not a formatting failure; it is a substantive legal reasoning error.

Watch for: False negatives on high-stakes clause types (termination, indemnification, limitation of liability) carry more operational risk than false positives. If deploying on contract review, measure precision and recall separately by clause type rather than relying on aggregate accuracy.

Most favored nation clauses often appear late in commercial agreements. Models that perform accurately on shorter contracts or on clause questions early in a document degrade on questions whose relevant text appears deep in a long contract. The model loses track of the clause structure as context length increases.

Watch for: Performance on long-context drift correlates strongly with contract length. If your workflow covers enterprise agreements or master service agreements that run to 50+ pages, test specifically on clause questions where the relevant provision appears after page 20. Short-contract benchmark scores do not predict long-contract performance.