Design RAG that retrieves the right company information and cites it

Imagine an equipment supplier whose staff ask whether a replacement item can be sent before the returned unit arrives. The answer may depend on the product family, customer agreement and current service policy. A paragraph containing the word “replacement” is not enough. This article follows that question through the retrieval process.

Collect the questions employees actually ask, including the short versions they type when busy. “Can we send it first?” requires context that “What is the advance replacement policy for this product family?” contains explicitly. The interface can ask for the product and agreement rather than forcing the model to guess them.

For each question, identify the evidence needed to answer it. Record the relevant document, section, effective date and exceptions. Include questions for which there is no approved answer. That gives the system an observable outcome: find sufficient evidence, ask a clarifying question or explain that the available material does not settle the issue.

Keep retrieval evaluation separate from answer evaluation. If the correct exception never reaches the model, changing the writing prompt is unlikely to solve the problem. Conversely, correct retrieval does not prove that the final explanation preserves every important qualification.

The document pipeline should preserve structure, not merely turn files into strings. Capture headings, table headers, lists and page locations. A table cell containing “eligible” means little without its product row and the condition in the column header. Scanned files need an extraction check before their contents become searchable.

Microsoft's document chunking guide describes fixed and structure-aware approaches. It also explains that a document may benefit from smaller chunks even when it fits within a model's input limit. There is no universal chunk size that settles every company-document problem.

For the replacement-policy example, keep the eligibility rule and its exception close enough to retrieve together. Store a section identifier so a small matching passage can lead to the surrounding explanation. Test overlap carefully: repeated fragments can crowd out distinct evidence if they occupy several top results.

Attach metadata that your business can maintain: document ID, version, language, product scope, effective period, owner and access groups. Define what happens when a document is superseded or deleted. If the original disappears but its chunks remain searchable, the assistant can continue giving answers from withdrawn material.

Exact identifiers and natural-language descriptions create different retrieval needs. A request mentioning a precise product code should not be satisfied by a semantically similar product. A question about “sending the new unit before the old one comes back” may not contain the policy's term “advance replacement.”

Hybrid search in Azure AI Search combines keyword and vector retrieval and merges their results. That provides a useful pattern to test when both exact terms and meaning matter. It is a candidate design, not proof that every query needs the same search configuration.

A practical experiment compares keyword-only, vector-only and hybrid candidates against the same evidence set. Inspect whether the correct passage appears near the top, whether irrelevant fragments dominate and whether a reranking step improves selection enough to justify its delay. Do not treat a search score as a probability that the final answer is true.

Retrieval must use the authenticated employee's permitted scope. Do not accept a user-supplied department name as proof of membership, and do not fetch all documents before asking the model to hide restricted ones.

Microsoft's security filter pattern explains how identifiers can filter search results, while making clear that those strings are not authentication themselves. The application still needs to establish identity and apply the correct filter consistently. This distinction matters when a search index is shared across teams or customers.

Check more than the main search request. Cached answers, suggested questions, citation previews and direct document links can expose information through other routes. Test access changes too: an employee who leaves a project should not retain its answers simply because a previous response was cached.

A useful citation opens the relevant source and indicates the section, version or page supporting the claim. Linking to a document homepage can look authoritative while leaving the reader to search the whole file. Preserve a stable reference to the passage used, with enough context to understand it.

In the replacement example, the answer should state the condition that permits advance shipment and the exception that prevents it. If two current sources conflict, show the conflict and route it to the policy owner. The model should not choose whichever paragraph sounds more confident.

Evaluate citations claim by claim on a sample of answers. Does the cited passage actually support the sentence? Does it apply to this product and date? Did the answer add an unsupported operational promise? These questions reveal problems that a simple count of citation links misses.

Begin with one maintained knowledge area and a named owner. Track unanswered questions, failed searches and stale-document incidents separately. An unanswered question may mean the policy is missing, not that the model needs a larger context window. Feed that finding back to the people responsible for the source material.

Add new document collections after their owners can supply versions and access rules. Record extraction, retrieval and answer configurations so a later change can be compared against the original question set. The goal is a dependable path from a business question to inspectable evidence.

For the wider information architecture, see which system should own each business record. For team adoption, consider a shared AI workspace that gives employees clear tasks and maintained knowledge.

Talvyn Labs can build a company knowledge assistant around your real documents and access boundaries. Bring the questions staff repeatedly escalate and the approved sources behind them. We can test retrieval, define evidence requirements and develop a product that helps the team act on supported answers.