Evaluation · Guide
Build an evaluation set from the work, not the demo
A small, well-labeled test set can reveal whether retrieval and generation fail for different reasons.
Collect representative questions
Ask staff for questions they recently had difficulty answering, then obtain permission to turn them into redacted or synthetic tests. Include routine questions, exceptions, missing answers, and ambiguous requests. Bedrock evaluation documentation describes expected answers and retrieved text; your team still decides what correct means. Do not use only questions copied from section headings, because employees rarely ask in the exact language of a policy author.
Label evidence and acceptable behavior
For an illustrative operations assistant, each test record should contain the question, authorized user role, expected source, and acceptable response behavior. Some questions should lead to clarification rather than a single answer. Mark the critical facts that must be present and the claims that must not appear. This makes human review consistent and prevents a fluent but unsupported response from passing simply because it sounds helpful.
Workflow at a glance
Keep a held-out slice
Use one portion of the set for diagnosis and keep another portion untouched until the proposed change is ready. Otherwise prompt adjustments can memorize the visible examples. Record corpus, parser, model, and configuration versions with each run. Compare retrieval success, supported answer quality, and total review effort separately. A single aggregate score can hide a serious regression in a small but consequential category.
Grow from failures
After the pilot, add verified incidents and new workflow variants to the set, taking care not to store unnecessary personal information. Retire obsolete questions with an explanation instead of silently deleting inconvenient cases. Treat evaluation data as a maintained business asset. It is not a guarantee against rare errors, but it gives every future change a more disciplined comparison than an informal conversation with the latest model.
A release review meeting
At the release review, show several failed cases before the aggregate score. Ask the operational owner whether the remaining failures are acceptable for the proposed scope. Record accepted limitations and blocked use cases. This makes evaluation a decision tool rather than a decorative report attached after deployment.
Sources and further reading
- Evaluate the performance of Amazon Bedrock resourcesEvaluate the performance of Amazon Bedrock resourcesAWS · Documentation ·
- Enabling Large Language Models to Generate Text with CitationsEnabling Large Language Models to Generate Text with CitationsALCE authors / arXiv · Research source ·
Editorial approach
AI-assisted original writing, checked against linked public sources. Practical recommendations and illustrative scenarios are editorial analysis, not reported client results. Source dates and product details may differ from this collection’s preparation date.