Economics · Analysis
A compelling AI demo is not yet a business case
Move from a polished example to evidence about everyday performance.
Recognize selection
A demonstration usually shows a prepared input, a cooperative user, and a visible success. That can explain a concept, but it does not establish frequency of benefit or cost of exceptions. METR’s productivity research is a useful reminder that perceived assistance and measured task performance can diverge. Our recommendation is to keep the demo as a teaching aid and build a separate measurement plan.
Bring ordinary and awkward cases
Use routine examples, incomplete requests, unfamiliar terms, and cases that should be rejected. In an illustrative document workflow, a clean PDF may be easy while a revised attachment with conflicting dates creates costly checking. Include those cases deliberately instead of removing them as outliers. Ask staff to complete the whole task, because a convincing draft can still increase the time required to reach an approved final result.
Workflow at a glance
Make costs visible
Separate software charges from setup, cleanup, review, maintenance, and incident effort. Do not assign all improvements to AI if the team also redesigned the form or clarified responsibilities. Compare against that simpler process improvement. An honest business case can conclude that most value comes from organizing the work, with AI helping only a narrow part of the remaining task.
Decide what can be repeated
Before rollout, repeat the evaluation with a different reviewer or a later time period. Keep a record of failures and identify whether they are acceptable, repairable, or disqualifying. Avoid forecasting annual savings from the best observed case. The decision should rest on a range of outcomes and a support model the business can sustain, not on the emotional response to a single impressive output.
Keep a demonstration evidence log
Label each demonstration case as prepared, ordinary, or difficult. Record whether it was retried and whether an unseen person corrected the output. This does not invalidate a demo; it explains what the audience is seeing. Use the log to choose additional evaluation cases. It is especially valuable when a sales presentation and an internal test are run by different people who might otherwise mistake selected outputs for typical performance.
Sources and further reading
- We are Changing our Developer Productivity Experiment DesignWe are Changing our Developer Productivity Experiment DesignMETR · Research source ·
- AI RMF Playbook: MeasureAI RMF Playbook: MeasureNIST · Documentation ·
Editorial approach
AI-assisted original writing, checked against linked public sources. Practical recommendations and illustrative scenarios are editorial analysis, not reported client results. Source dates and product details may differ from this collection’s preparation date.