Financial QA 10K

Questions asking for percentage changes or growth figures embedded in 10-K narrative prose produce errors when models read surrounding context instead of isolating the specific figure. The model often returns a directionally correct answer without anchoring to the precise number stated in the text.

Watch for: Workflows requiring exact percentage figures from MD&A sections are higher risk than those requiring directional or yes/no answers. Validate extraction accuracy specifically on numeric-in-prose questions before deploying on analyst screening tasks.

The expected answer for capability or descriptive questions is a specific phrase from the filing. Models that answer correctly but rephrase ("improves efficiency" vs "enhances productivity") score zero under LLM judge evaluation. The answer is substantively right; the scoring marks it wrong.

Watch for: For open-ended qualitative questions from 10-K filings, LLM judge accuracy understates real performance. Use context groundedness and human spot-check as the primary quality signals on this question type.

Questions requiring enumeration of industries, product categories, or named entities produce partial matches when the model captures some items but not all, or lists them in a different order. Text similarity scores drop sharply even when the answer is substantially correct.

Watch for: List-type answers have high variance in text similarity scoring regardless of factual accuracy. If your workflow involves extracting named lists from 10-K disclosures, validate with a stricter exact-match or set-overlap metric rather than text similarity.