What this dataset measures
Hybrid table-and-text reasoning on real financial reports. Questions require combining data from structured tables with explanatory prose.
Numeric match is high across the fleet; differentiation shows up in text F1 and LLM judge. Table read errors and scale/units drive failures.
Composite ranking across 9 models
Composite score is the unweighted mean of five active metrics: numeric match, text similarity (F1), context groundedness, hallucination proxy, and LLM judge accuracy.
How models differ across scoring dimensions
Scores below are the metrics that feed the composite for this dataset.
| Claude Opus 4.7 | 91.3% |
| Claude Opus 4.8 | 88.7% |
| Claude Fable 5 | 90.0% |
| Groq Llama 3.3 70B | 83.3% |
| GPT-5 mini | 78.0% |
| Groq Llama 4 Scout | 90.0% |
| GPT-5 | 81.3% |
| Claude Sonnet 4.6 | 87.3% |
| Claude Haiku 4.5 | 88.0% |
Unweighted mean across test rows. Measures whether extracted numbers match the reference answer.
| Claude Opus 4.7 | 40.5% |
| Claude Opus 4.8 | 39.2% |
| Claude Fable 5 | 34.0% |
| Groq Llama 3.3 70B | 33.2% |
| GPT-5 mini | 34.4% |
| Groq Llama 4 Scout | 27.1% |
| GPT-5 | 23.9% |
| Claude Sonnet 4.6 | 21.2% |
| Claude Haiku 4.5 | 19.2% |
Unweighted mean across test rows. Token-level F1 between model output and reference string.
| Claude Opus 4.7 | 71.3% |
| Claude Opus 4.8 | 73.0% |
| Claude Fable 5 | 74.1% |
| Groq Llama 3.3 70B | 67.9% |
| GPT-5 mini | 74.6% |
| Groq Llama 4 Scout | 67.8% |
| GPT-5 | 79.0% |
| Claude Sonnet 4.6 | 70.1% |
| Claude Haiku 4.5 | 66.9% |
Unweighted mean across test rows. Overlap between model output and cited source passages.
| Claude Opus 4.7 | 81.3% |
| Claude Opus 4.8 | 81.9% |
| Claude Fable 5 | 86.1% |
| Groq Llama 3.3 70B | 83.9% |
| GPT-5 mini | 83.4% |
| Groq Llama 4 Scout | 84.2% |
| GPT-5 | 85.4% |
| Claude Sonnet 4.6 | 82.7% |
| Claude Haiku 4.5 | 83.3% |
Hallucination proxy (leader): 86.1%. Higher scores can mean fewer unsupported extra numbers, not necessarily correct answers.
| Claude Opus 4.7 | 33.3% |
| Claude Opus 4.8 | 30.7% |
| Claude Fable 5 | 27.3% |
| Groq Llama 3.3 70B | 32.7% |
| GPT-5 mini | 30.0% |
| Groq Llama 4 Scout | 19.3% |
| GPT-5 | 12.0% |
| Claude Sonnet 4.6 | 10.0% |
| Claude Haiku 4.5 | 9.3% |
Unweighted mean across test rows. LLM judge scores answer correctness against the reference.
Where models go wrong
Representative examples of where models struggle on this benchmark. Each pattern shows a common error type drawn from the evaluation set.
Questions requiring a model to derive a value from context, such as identifying which of two years has the higher figure, produce errors when the model reads the wrong cell or misidentifies the comparison direction. The expected answer is simple (a year or a label), but arriving at it requires correctly reading a table structure first.
Watch for: Comparison questions over multi-column tables are a common failure point. A model that extracts numbers correctly may still fail when asked to rank or compare them. Test comparison tasks separately from extraction tasks.
Questions requiring multiple values across multiple years produce scale mismatches when the model reads figures in different units or applies inconsistent rounding. The expected answer "394, 0, 0" requires reading three cells correctly and returning them in order. Any unit mismatch or positional error on one cell fails the whole answer.
Watch for: Multi-value answers requiring consistent units across rows are disproportionately error-prone. For financial workflows involving year-over-year comparisons, validate that the model returns values in the same unit and format across all requested periods.
Percentage change questions require the model to locate the correct row, identify the correct year columns, and compute the change. Errors arise when the model reads from the wrong row, uses the wrong base year, or conflates absolute values with percentage changes. The expected answer is precise; the failure produces a plausible but wrong figure.
Watch for: Computed percentage change is one of the hardest table tasks on this benchmark. If your workflow involves period-over-period change calculations from structured tables, this is the failure mode most likely to surface in production.
Questions asking for values across multiple time periods require the model to return a structured list in the correct order. Models frequently return only one value, reverse the order, or merge the values into a single figure. The expected answer "6,490, 7,859" is failed by any answer that returns one number or swaps the sequence.
Watch for: Multi-span answers have poor tolerance for partial credit. A model that gets one of two values right scores similarly to one that gets both wrong. If your workflow requires multiple values from a single question, evaluate completeness and ordering separately from accuracy.
What this means for deployment
Claude Opus 4.7 leads at 63.6% composite across 150 TAT-QA tasks, with a fleet average of 59.0%. The key finding on this dataset is the divergence between numeric match (83-91% across models) and text similarity (19-40%). Models are getting the numbers right but structuring their answers differently from the reference format. For deployment, this means numerical accuracy is stronger than the composite score suggests.
Claude Opus 4.7 leads on composite (63.6%) with strong numeric match (91.3%) and context groundedness (71.3%), but LLM judge accuracy of 33.3% is low relative to its numeric match. This is the TAT-QA pattern: models extract correct figures but phrase answers in ways that diverge from the reference string. The numeric match score is the more reliable signal for table-and-text extraction workflows.
The 40.5% text similarity score does not mean models are wrong half the time. It means models return correct figures in different formats, orders, or with different surrounding text than the reference answer. Numeric match at 83-91% is the accurate picture of extraction performance. Hallucination proxy at 81.3% is moderate, meaning some models occasionally add unsupported figures alongside correct ones.
Interactive charts
Cost & workflow analysis
Claude Opus 4.7 leads on composite (63.6%); several alternatives trade a few points of accuracy for much lower run cost. Groq Llama 3.3 70B sits within 3.4 pts of the leader at roughly 31× lower cost — strong value for high-volume or pilot workloads.