Economics · Research
Design a pilot comparison that can challenge your hopes
Avoid attributing every improvement to the tool you wanted to succeed.
Use comparable work
Compare tasks with similar complexity and completion standards. A team should not give the AI-assisted group the cleanest requests and then infer a universal benefit. METR’s 2026 experiment-design update discusses selection problems in productivity measurement. We use that as a warning about inference, not as a current speedup estimate for every tool. A small business can still improve its comparison by documenting how cases were selected.
Observe the whole task
Record start, review, correction, approval, and handoff rather than only generation time. In an illustrative report pilot, the output is complete only when a reviewer can use it confidently. Include time spent locating sources or resolving contradictions. If the team also changes templates or responsibilities, note that separately. The useful question is which combination improved the process, not whether AI deserves all the credit.
Workflow at a glance
Report distributions and exceptions
An average can hide a workflow that is fast on easy cases and disastrous on difficult ones. Show a range, typical cases, and the most important failures. Use small samples as directional evidence, not definitive proof of rare-error safety. Ask whether one serious failure would change the economic case. This discussion may justify a narrower deployment even when the average result looks attractive.
Allow a disappointing conclusion
Decide before testing how results will affect the decision. If no measured benefit appears, revise the hypothesis or stop rather than changing the success metric afterward. Keep staff feedback alongside observed timings, and explain disagreements. A result that challenges expectations can point to a better workflow or training need. It should not be suppressed simply because the demonstration and original budget assumed a positive outcome.
A comparison log
For each task, log selection reason, complexity, method used, reviewer, total time, and meaningful errors. Keep rejected or abandoned tasks in the record with an explanation. Otherwise, a comparison can become biased by retaining only completed successes. Review the log before calculating averages, and state any missing information so the business understands the strength and limits of the conclusion.
Sources and further reading
- We are Changing our Developer Productivity Experiment DesignWe are Changing our Developer Productivity Experiment DesignMETR · Research source ·
- AI RMF Playbook: MeasureAI RMF Playbook: MeasureNIST · Documentation ·
Editorial approach
AI-assisted original writing, checked against linked public sources. Practical recommendations and illustrative scenarios are editorial analysis, not reported client results. Source dates and product details may differ from this collection’s preparation date.