Better RAG starts with retrieval you can inspect

A grounded assistant cannot use a paragraph it never receives. When an answer is wrong, teams often edit the generation prompt first. It is usually worth opening the retrieval results before doing that. The relevant passage may be absent, outdated, or buried beneath near-duplicate documents.

Consider a maintenance manual containing an error code, E-417. A user types the code exactly. Another user describes the same problem as “the pump stops after warming up.” These are different search problems, even though the useful destination may be the same page.

Give exact terms and meaning a place

Keyword search can preserve exact matches for identifiers and specialist terminology. Embedding search compares learned vector representations and can help connect different wording. A hybrid approach combines candidate sets or rankings from both. Its value must be measured on your queries rather than assumed from the architecture.

Avoid casually adding raw scores from unrelated retrieval methods. Their scales may differ. Use a documented fusion method or tune a combination against relevance judgments. Keep the original candidate lists available while debugging so you can see which branch found a useful result and which introduced noise.

Use reranking as a second decision

Sentence Transformers documents a retrieve-and-rerank pattern in which an efficient first stage finds candidates and a cross-encoder scores query-document pairs. The second stage can examine relationships more closely, at additional processing cost. It can reorder candidates; it cannot recover a document that never entered its candidate set.

For the maintenance example, inspect whether the E-417 page appeared in the initial results. If it did not, increasing reranker quality is unlikely to repair that request. If it appeared but ranked below irrelevant pages, reranking is a plausible experiment. This distinction saves time and makes failures easier to explain.

Preserve enough context to interpret a passage

Splitting every document into identical character counts can detach a warning from the instruction it qualifies. Keep headings, units, revision dates, and useful parent context with each chunk. Test tables and numbered procedures explicitly. A sentence containing “do not” is especially dangerous to separate from the action it governs.

Store a stable document identifier and location with every result. The final answer should link to evidence the reader can inspect. If the document changes, maintain a way to determine which revision supported an earlier answer, subject to your retention policy.

Apply permissions before evidence reaches the model

Restrict the search to records the current user can access. Filtering unauthorized passages after generation is too late: information may already have influenced the response. Apply equivalent access rules across keyword search, vector search, caches, and reranking inputs.

Test with two accounts that have different access to similar documents. Ask both the same questions and inspect retrieved identifiers. This is an application authorization check as much as a search-quality check; a fluent answer cannot compensate for the wrong evidence boundary.

Build a retrieval report

For a diagnostic query set, mark relevant passages and inspect recall at the candidate stage, ordering after reranking, and support for the final answer. Include exact codes, paraphrases, missing answers, and conflicting revisions. Keep retrieval timing separate from generation timing. Change one stage at a time so an improvement has an identifiable cause.

Further reading

Sentence Transformers: Retrieve and rerank documentation