AI engineering and commissioning
How to Evaluate RAG Development: Sources, Access & Answers
A RAG pilot should be accepted against supported answers, source permissions and source changes, not a convincing demonstration alone. Agree which users and questions it must serve, what information they may use and what happens when the answer cannot be supported. Evaluate retrieval, generation and access as separate parts of the system before committing to a wider rollout.
This is StackEndBox's proposed evaluation method for commissioning a knowledge feature. It is original engineering guidance informed by the primary sources below. The examples are hypothetical. They contain no client data, claimed deployment, measured accuracy or compliance certification.
Decide whether you need retrieval, extraction or an agent
Retrieval-augmented generation supplies selected external information to a model when it answers. The original RAG paper explains the combination of retrieval and generation. A business implementation also needs source ownership, application access, data lifecycle and an operating interface. The research result does not establish the quality of your collection or workflow.
Use retrieval when the task is answering from permitted knowledge. Use document processing when the required result is a reviewed set of fields in a business record. Use a tool-using agent when selecting and executing steps is actually necessary. These scopes can connect, but a knowledge-assistant quote should not silently authorise changes to business systems.
Write the evaluation contract before tuning
Choose one audience and collection for the first milestone. Identify the source owner, the person who can judge answers and the operator who will support the system. Keep some representative questions out of development so acceptance does not merely repeat examples used to tune the implementation.
For each question, record an expected answer boundary, permitted source identifiers and versions, the relevant user role, and the reason an answer should be withheld. A reference answer need not be one exact sentence. It can specify the claims that must be supported and the actions the system must not take.
Use approved or synthetic material for an initial evaluation. If real documents are needed, agree access, provider processing, redaction and retention first. Do not place source passages, personal questions or credentials in public analytics. The case register can use opaque identifiers and restricted links instead of duplicating documents.
Test the failures a polished demonstration can hide
| Case | Expected behaviour | Evidence to inspect |
|---|---|---|
| Answer exists in a permitted current policy | Return a supported answer with a source the user can open. | Question, permitted source version, retrieved passage and the claims it supports. |
| Answer exists only in a restricted document | Withhold the restricted content, including revealing titles or snippets where those are sensitive. | The user's effective permissions and the material supplied to the model, not only the visible final answer. |
| A user's access is revoked after an earlier successful query | Apply the agreed revocation policy to subsequent retrieval and cached responses. | Permission update time, cache behaviour and a repeated query by the same user. |
| An old policy contradicts a new approved version | Use the agreed authoritative version or expose the conflict instead of silently choosing one. | Source precedence, effective date and ingestion state. |
| The source is deleted or ingestion fails | Respect the deletion policy and show an incomplete or stale collection state where appropriate. | Index state, failed jobs, deletion handling and the operator's recovery path. |
| A retrieved document contains instructions to ignore restrictions | Treat source content as evidence, without granting it authority over tools or permissions. | Denied actions and enforcement at the application or tool boundary. |
| A citation is plausible but does not support the claim | Fail the support check, even if the prose sounds right. | Claim-to-passage review rather than citation count. |
| No permitted source answers the question | Abstain or escalate through the agreed workflow. | Unanswerable-case results and whether a reviewer receives enough context. |
Separate retrieval, answer quality and access
A system may retrieve the correct passage and still generate an unsupported statement. It may answer correctly while retrieving information the current user must not see. Those are different defects and require different corrections.
Review retrieval relevance and coverage, the support for generated claims, answer usefulness, refusal behaviour and access failures separately. Include latency, provider usage and reviewer effort. The Ragas metric catalogue distinguishes measures such as context precision, context recall and faithfulness. Automated evaluations can help compare revisions, but a model-based judge is another fallible component. Calibrate it with human review on the actual task.
Report denominators and exclusions. For example, keep the number of answerable, unanswerable and permission-denied cases visible rather than combining them into one flattering percentage. Agree which failures block expansion. Do not invent a universal pass rate for every business or infer production accuracy from a small synthetic set.
Permissions and source freshness are operating requirements
Record where access is enforced, how source permissions reach the retrieval system and how changes propagate. Filtering only after a model has seen a restricted passage is too late to protect that input. A shared answer cache also needs an access-aware policy. A safe key cannot simply be the question text when users have different rights.
Microsoft's document-level access documentation distinguishes application-supplied security filters from identity-based capabilities, some of which are marked preview. Check the exact provider, source connector, API version and production support rather than assuming every managed search service inherits document permissions automatically.
Deletion from a source, removal from an index and expiry from a response cache are separate events. Agree what the user may see during propagation, the maximum acceptable delay and what the operator does when an update fails. A new document should not be described as immediately available unless the ingestion and query path actually establish that behaviour.
Compare equivalent RAG development proposals
Ask providers to name the included collection, user roles, integration, evaluation cases and handover. Compare a source-readiness review, an evaluated pilot and a supported product integration as different purchases. A prototype estimate is not a production-support agreement.
Separate engineering effort from ingestion, search hosting, model usage and monitoring charges. Review the expected volume and refresh frequency, plus the human work required to correct failures. Our public starting hourly rate and quote boundaries explain how we distinguish a rate from a project fee. No rate establishes the effort required for your collection.
The milestone should end with something the nominated team can inspect: accepted and failed cases, known limitations, access boundaries, release and rollback instructions, source-update status and a named operator. Expansion should follow reviewed evidence. Procurement, implementation acceptance and permission to activate consequential actions remain separate decisions.
Commission the smallest useful next step
Send a non-confidential description of the audience, the questions it needs answered, source types, current search method, access constraints and the result that would justify proceeding. Do not send production credentials or private documents in an initial enquiry. StackEndBox works online with international teams. Scope, availability, processing terms and overlap are agreed before engagement.
Compare our RAG and AI product-integration scope, the document-processing evaluation guide and the agent-versus-workflow decision. Each addresses a different business result.