Evaluation · Guide
Teach the assistant how to say it does not know usefully
An explicit uncertainty path is part of service design, not an embarrassing failure screen.
Define the unanswered states
Not found, contradictory, out of date, and not authorized are different outcomes. Give each a response and routing rule. A user should not be told that a document does not exist when it is merely outside their access. Keep messages helpful without revealing restricted information. The aim is to show what the employee can do next while respecting the limits of both the evidence and the user’s role.
Provide a concrete next step
For an illustrative onboarding assistant, a missing procedure might create a content-owner task, while an ambiguous client name should trigger a clarification question. Keep a link to the approved manual process when available. Avoid a generic apology that forces the user to start again. Record enough context for the human handoff so staff do not have to repeat the question, but minimize unnecessary customer data in the ticket.
Workflow at a glance
Test refusal and over-refusal
A system that refuses everything is easy to make cautious but not useful. Build paired tests where evidence is sufficient and where a small missing fact makes a direct answer inappropriate. Have business reviewers judge both. Count unnecessary escalation alongside unsupported answers. The right balance depends on the decision and on how quickly staff can resolve a handoff; it should not be chosen solely to maximize a model score.
Use uncertainty as a backlog
Group unresolved questions by cause and improve the highest-value gap first. Some need better source documents, some need retrieval work, and others need a clearer user interface. Do not automatically fine-tune a model because it lacks company knowledge. The linked evaluation and risk-management material gives context; the practical recommendation is a maintained uncertainty workflow with an owner, a resolution target, and a feedback loop.
An escalation capacity check
Estimate how many unresolved questions the named team can actually process. If the expected volume exceeds capacity, narrow the assistant’s scope or improve intake before launch. A technically cautious system can still harm service quality if every uncertainty becomes a ticket nobody has time to resolve.
Sources and further reading
Editorial approach
AI-assisted original writing, checked against linked public sources. Practical recommendations and illustrative scenarios are editorial analysis, not reported client results. Source dates and product details may differ from this collection’s preparation date.