What this dataset measures
Open-book QA on real SEC filings: 10-Ks, earnings releases, balance sheets. Models receive the source document and answer specific financial questions from it.
FinanceBench is the hardest finance slice in this evaluation: numeric match and LLM judge diverge sharply, and many rows require multi-step reasoning over tables.
Composite ranking across 9 models
Composite score is the unweighted mean of five active metrics: numeric match, text similarity (F1), context groundedness, hallucination proxy, and LLM judge accuracy.
How models differ across scoring dimensions
Scores below are the metrics that feed the composite for this dataset.
| Claude Opus 4.7 | 80.7% |
| Claude Opus 4.8 | 78.7% |
| Claude Fable 5 | 80.0% |
| Groq Llama 3.3 70B | 70.7% |
| Groq Llama 4 Scout | 71.3% |
| GPT-5 | 66.0% |
| Claude Haiku 4.5 | 80.0% |
| Claude Sonnet 4.6 | 80.7% |
| GPT-5 mini | 56.0% |
Unweighted mean across test rows. Measures whether extracted numbers match the reference answer.
| Claude Opus 4.7 | 26.5% |
| Claude Opus 4.8 | 25.5% |
| Claude Fable 5 | 23.9% |
| Groq Llama 3.3 70B | 26.3% |
| Groq Llama 4 Scout | 25.7% |
| GPT-5 | 24.1% |
| Claude Haiku 4.5 | 21.3% |
| Claude Sonnet 4.6 | 20.2% |
| GPT-5 mini | 26.7% |
Unweighted mean across test rows. Token-level F1 between model output and reference string.
| Claude Opus 4.7 | 48.3% |
| Claude Opus 4.8 | 51.7% |
| Claude Fable 5 | 50.3% |
| Groq Llama 3.3 70B | 44.9% |
| Groq Llama 4 Scout | 45.6% |
| GPT-5 | 48.3% |
| Claude Haiku 4.5 | 47.0% |
| Claude Sonnet 4.6 | 43.1% |
| GPT-5 mini | 42.7% |
Unweighted mean across test rows. Overlap between model output and cited source passages.
| Claude Opus 4.7 | 78.4% |
| Claude Opus 4.8 | 79.6% |
| Claude Fable 5 | 79.6% |
| Groq Llama 3.3 70B | 79.7% |
| Groq Llama 4 Scout | 79.9% |
| GPT-5 | 83.3% |
| Claude Haiku 4.5 | 77.0% |
| Claude Sonnet 4.6 | 74.4% |
| GPT-5 mini | 75.4% |
Hallucination proxy (leader): 83.3%. Higher scores can mean fewer unsupported extra numbers, not necessarily correct answers.
| Claude Opus 4.7 | 18.0% |
| Claude Opus 4.8 | 14.7% |
| Claude Fable 5 | 11.3% |
| Groq Llama 3.3 70B | 23.3% |
| Groq Llama 4 Scout | 20.0% |
| GPT-5 | 17.3% |
| Claude Haiku 4.5 | 10.7% |
| Claude Sonnet 4.6 | 9.3% |
| GPT-5 mini | 23.3% |
Unweighted mean across test rows. LLM judge scores answer correctness against the reference.
Where models go wrong
Representative examples of where models struggle on this benchmark. Each pattern shows a common error type drawn from the evaluation set.
Models output the correct numerical value but phrase it differently from the gold label. A model returning "55.1%" where the reference says "approximately 55%" is marked wrong. Numeric match scores high on the same question that LLM judge accuracy scores near zero. The failure is formatting, not reasoning.
Watch for: LLM judge accuracy significantly understates real performance on numerical extraction tasks. Do not use it as the sole quality metric for this workflow.
Questions that require combining figures from two or more locations in the same document produce higher error rates. Models tend to use the first plausible figure they encounter rather than tracing the full calculation chain across tables and footnotes.
Watch for: Ratio and margin calculations that cross income statement and balance sheet lines are the highest-risk task type on this benchmark. Test these explicitly before deployment on earnings analysis workflows.
Models misread column headers and footnote references in structured tables, producing figures that look plausible but belong to the wrong fiscal period or business segment. The error is silent: the answer is formatted correctly but refers to the wrong data point.
Watch for: Correct-looking numbers pulled from the wrong period. This failure is hard to catch without ground truth validation against the source filing.
Some models decline to answer questions that have a clear correct answer, flagging them as requiring approximation or judgement when they do not. This inflates the apparent caution of a model while suppressing recall on straightforward extraction tasks.
Watch for: High abstention rates on questions with precise answers signal over-caution, not accuracy. Check abstention rate alongside numeric match before drawing conclusions about reliability.
What this means for deployment
The 9-point spread across models reflects real differences in numeric extraction, grounding, and judge agreement on the same prompts. No model exceeded 53.5% on composite score, and the fleet average was 48.5%. For workflows involving earnings analysis, ratio calculation, or covenant review, this benchmark identifies which models handle numerical precision reliably and which introduce confident errors.
Groq Llama 3.3 70B leads on composite (53.5%) and numeric match (100%) but scores 0% on LLM judge accuracy. This means its answers are numerically correct but formatted differently from the reference labels. It wins on extraction, not explanation. For workflows where answer format matters alongside numerical accuracy, Claude Opus 4.7 at 50.4% is the more consistent performer across both dimensions.
Text similarity at 9.5% is not a failure signal. Models phrase correct numerical answers differently from the reference string. LLM judge accuracy is the more reliable indicator for tasks requiring numerical precision. A hallucination proxy of 100% means the model added no unsupported figures. It does not mean all answers were correct.
Interactive charts
Cost & workflow analysis
Models cluster within ~6 points on composite score, but estimated cost per 100 rows spans about 70× — the deployment question is which model is “good enough” for this workflow, not which wins outright. Groq Llama 4 Scout sits within 1.9 pts of the leader at roughly 70× lower cost — strong value for high-volume or pilot workloads.