Knowledge · Analysis
Control RAG cost by improving evidence selection first
More retrieved text is not automatically more useful context.
Budget information before tokens
A larger context allowance can hide weak retrieval by sending many marginal passages to the model. Start by identifying which evidence the question actually needs. Keep retrieval count, passage length, reranking, and answer generation as separate cost drivers. The contextual-retrieval research provides experimental background; it does not establish a universal best amount of context for every company’s documents or tasks.
Measure cost per resolved question
For an illustrative policy assistant, compare a short evidence set with a broader set on the same questions. Record retrieval success, unsupported claims, latency, usage, and reviewer effort. A cheaper response that requires a long manual correction may be more expensive overall. Likewise a long answer with many citations may still fail to settle the employee’s actual question. Define resolution before choosing a cost target.
Workflow at a glance
Watch duplicate evidence
Overlapping chunks, mirrored folders, and translated copies can crowd out distinct information. Inspect a sample of retrieved sets and track whether extra passages add evidence or only repeat it. Apply version and scope filters where reliable metadata exists. Avoid compressing away exceptions just to reduce tokens. A smaller prompt is useful only when it retains the information necessary for a correct, authorized answer.
Set an operating envelope
Document expected question volume, refresh frequency, maximum useful evidence, and fallback behavior for large requests. Review actual costs after the pilot rather than promising a fixed saving from a laboratory result. Separate embedding and ingestion costs from recurring query costs. This makes it possible to explain why a frequently changing corpus behaves differently from a stable handbook even when employees ask the same number of questions.
A weekly evidence sample
Choose a small sample of expensive queries and inspect every retrieved passage. Remove redundant retrieval only after confirming that exceptions and contradictory evidence remain visible. Keep a before-and-after record of resolution quality. The cost review should protect useful evidence, not merely reward shorter answers.
Sources and further reading
- Introducing Contextual RetrievalIntroducing Contextual RetrievalAnthropic · Research source ·
- Configure and customize queries and response generationConfigure and customize queries and response generationAWS · Documentation ·
Editorial approach
AI-assisted original writing, checked against linked public sources. Practical recommendations and illustrative scenarios are editorial analysis, not reported client results. Source dates and product details may differ from this collection’s preparation date.