The problem
A fluent answer can hide missing evidence. For enterprise AI, a useful interface should let someone inspect which passages support a response and notice when an important constraint is absent.
My work
I built a Python evaluation harness to compare document parsing, retrieval, context management, and tool-using agents on financial filings. The main evaluation uses 79 answerable FinanceBench questions over 31 filings from 21 companies, plus 15 authored unanswerable questions.
The harness records answers, source citations, retrieval metrics, input tokens, and costs. Requests are cached, and a budget guard stops a run when it reaches its configured limit. All main comparisons use the same generator, GPT-4.1 mini, with text-embedding-3-small embeddings.
System decisions
Table-aware parsing mattered more than adding retrieval components. Moving from naive chunks to Docling sections and compact tables increased gold-page recall from 44% to 79%. Hybrid fusion and the tested rerankers did not improve retrieval on this corpus, so the managed configuration leaves reranking off.
The managed pipeline filters by company and fiscal year, retrieves dense candidates, removes duplicates, and applies a context budget. The agent uses that retrieval with a calculator and a structured final-answer tool. This separates the problem of finding a figure from the problem of calculating with it.
The small explorer below is a separate teaching example. Its fictional battery documents and authored answers illustrate evidence coverage; its strategy labels are not measurements from the FinanceBench experiment.
Retrieved evidence (2)
Operations guide
Charging window
Charging is permitted between 02:00 and 05:00. Scheduled charging must also comply with the reserve and temperature limits in the safety addendum.
Safety addendum
Operating limits
Keep the battery reserve above 20%. Charging is allowed only when the cell temperature is between 10°C and 35°C. If a limit would be breached, postpone charging.
Supported answer
Charging is permitted between 02:00 and 05:00, with the reserve kept above 20% and the cell temperature between 10°C and 35°C. Postpone charging if a safety limit would be breached.
Sources: Operations guide, Safety addendum
Combining the time window with the safety addendum supports an answer with all stated conditions.
Read the full corpus & how this example works
These three fictional documents were written for this portfolio. Each strategy selects a curated set of passages and an authored answer. This demonstrates evidence coverage; it does not run a search model or measure the performance of a retrieval method.
Operations guide — Charging window
Charging is permitted between 02:00 and 05:00. Scheduled charging must also comply with the reserve and temperature limits in the safety addendum.
Safety addendum — Operating limits
Keep the battery reserve above 20%. Charging is allowed only when the cell temperature is between 10°C and 35°C. If a limit would be breached, postpone charging.
Tariff note — Price signals
An off-peak price signal is an economic preference, not permission to operate. A low price does not override the charging window, reserve requirement, or temperature limits.
This is a software illustration, not operational battery guidance.
Evaluation
On the 79 answerable questions, managed RAG scored 59% answer accuracy and the compact tool-using agent scored 76%. The paired bootstrap difference was +16.5 percentage points, with a 95% interval of +3.8 to +29.1. These are results from this small recorded study, not production accuracy guarantees.
| Approach | Answer accuracy | Gold page retrieved | Mean input tokens |
|---|---|---|---|
| Naive dense RAG | 54% | 44% | 4.4k |
| Structured RAG | 63% | 79% | 4.9k |
| Hybrid RAG | 65% | 71% | 4.9k |
| Managed context | 59% | 82% | 3.5k |
| Compact agent with tools | 76% | 75% | 5.0k |
Download the recorded results and intervals
Accuracy and retrieval metrics use the 79 answerable questions. Mean input-token counts include the 15 unanswerable checks as well.
Adding other-year filings exposed citation errors that answer accuracy alone could hide. Under corpus drift, wrong-filing citations rose from 9% to 46% for naive RAG, while the metadata-filtered pipeline changed from 1% to 6%.
The source dataset is FinanceBench by Patronus AI. The long-context oracle in the opening figure receives the correct filing and uses only 19 questions; it is not directly comparable to the full evaluation and is excluded from the table above.
Limitations
The study uses one generator and an LLM judge; its bootstrap intervals are wide. The managed configuration was chosen using the same evaluation questions, which introduces selection bias. Query rewrites preserved entities and periods, so that drift test was mild. Compact memory reduced tokens but failed one multi-turn follow-up, showing that compression can discard useful context.
The portfolio presents saved results and precomputed examples. It runs no live model or external API. The explorer’s fictional battery documents must not be used as operational guidance.