Read what came back
Before changing anything, take fifty failures and look at the retrieved passages for each one. Not the answers, the passages. In our experience of doing this on client systems, a large share of the failures resolve into one of a handful of causes that have nothing to do with the model's reasoning.
- The answer was not in the corpus at all, and the system produced a fluent guess because nothing told it to refuse.
- The answer was in the corpus but split across a chunk boundary, so every retrieved fragment held half of it.
- The answer was retrieved along with four near duplicates of an older version, and the older version won.
- The query and the document use different vocabulary for the same thing, and nothing in the pipeline bridged them.
- The right document was excluded by a filter, or included when the asker had no right to see it, which is worse.
Chunking is a decision about your documents
A fixed token window is a default, not a design. It is chosen because it is easy to implement, and it works acceptably on prose and badly on almost everything else: tables, forms, policy documents with clause structure, transcripts, code.
The test for a chunking strategy is simple and nobody runs it: take a labelled question, retrieve, and ask whether any single retrieved chunk actually contains enough to answer. If the answer is routinely no, the chunking is wrong and no amount of re-ranking rescues it. Structure aware splitting, overlap where the structure is genuinely continuous, and parent document expansion after retrieval all exist for this, and which one is right depends on documents that only you have seen.
Metadata does the coarse work
Vector similarity is good at finding things that are alike and indifferent to whether they are applicable. Applicability is a metadata question: which jurisdiction, which effective date, which product line, which customer, which permission scope.
Two consequences follow. First, filter before you search rather than sorting it out afterwards, because a filter narrows the candidate pool honestly and a re-rank only reorders whatever the search happened to surface. Second, version awareness is not optional for any corpus where the rules changed. A question about a decision made last year should retrieve the policy that applied last year. A system that silently answers with the current version is wrong in a way that is very hard to notice and very expensive when someone finally does.
Permissions belong in the index
The most serious retrieval bug is not a wrong answer, it is a correct answer drawn from a document the person asking was not allowed to open. It happens whenever the index is built by a service account with broad access and the access rules live somewhere else.
The permission scope has to travel with the document into the index and be applied as a filter at query time, per asker. This is unglamorous, it slows the ingestion pipeline down, and it is the difference between a system that survives a security review and one that gets switched off during it.
Measure retrieval separately
An end to end evaluation set measures the answer. It cannot tell you whether a wrong answer came from bad retrieval or bad reasoning, and those have opposite fixes. A separate labelled set of questions paired with the passages that should be retrieved gives you recall at k, and recall at k is the number that tells you where to spend the week.
It also protects against the quiet failure where the system answers correctly on retrieved rubbish, because the model happened to know the answer already. That looks like success and is a system with no grounding, which will fail the moment it is asked about something specific to you.
When it genuinely is the model
Sometimes it is. The signal is that retrieval measures well, the correct passages are present in the context, and the output still fails on reasoning or format. That is a real finding and it justifies changing the approach, which might mean a different provider, a different decomposition of the task, or a step that was asking one call to do three things.
The point is not that models never matter. It is that the retrieval work is cheaper, faster and more durable, and it is the work that is still paying off after the next model release makes everyone's benchmark chart obsolete.