Operations · Guide
A review summary is not a customer survey
Separate review-source coverage, topic classification and representativeness before treating an AI summary as a customer survey.
A review summary is not a customer survey
Original AAHouse explanatory video. AI-assisted conceptual visuals and neural narration; original music.
English transcript · Simplified Chinese Mandarin transcript · Taiwan Mandarin transcript · Macao Cantonese transcript
The owner asks: Is everyone complaining?
Imagine a U.S. bicycle repair shop asking AI to summarize public reviews. The summary says customers care most about waiting time, so the owner considers adding staff. Pause before treating this as a survey result. People who post reviews, reviews displayed by a platform and reviews actually supplied to the model can be three different scopes. This is a fictional decision scenario, not a customer case.
- Who posted? What did the platform show?
- What actually reached the model?
Unchanged original source explanatory diagram.
The analyst answers: Define who first
Reframe the question: Which comments in this collected set mention waiting? Record the platform, date range, filters and review count. AAPOR’s survey best practices explain why sample design matters to the population that findings can describe.[1] Public reviews can reveal useful leads, but volume alone does not make them representative of every customer.
- Platform | date range | filters
- Review count is not the customer base
Unchanged original source explanatory diagram.
The editor asks: What did the model see?
Keep the review text and source obtained with appropriate permission, assigning each comment an internal reference. Require supporting references for every theme in the summary, then sample positive, negative and ambiguous wording. Do not count worth the wait as a complaint. Do not treat repeated descriptions of one person’s experience as independent opinions from several customers.
- Worth the wait is not a complaint
- One repeated account is not many people
Unchanged original source explanatory diagram.
The manager asks: How should we state the count?
In a synthetic example, six of twenty reviews mention waiting. Report exactly that: six of the twenty reviews in this set mention waiting. Do not rewrite it as thirty percent of customers are dissatisfied. A mention is not dissatisfaction, and a review is not a customer. Retain the denominator, time window and classification definition so a colleague can check the claim.
- 6 of 20 reviews mention waiting
- Mentioning waiting is not dissatisfaction
Unchanged original source explanatory diagram.
A buyer adds: Authenticity is not representativeness
The FTC’s Consumer Reviews and Testimonials Rule took effect on October 21, 2024, addressing deceptive practices including fake reviews.[2] That does not mean every review on a platform has been verified. Authenticity and representativeness are different questions. Nor should a business accuse a reviewer of fabrication merely because an AI model finds the wording suspicious.
- Authenticity: is the account credible?
- Representativeness: whom can it describe?
Unchanged original source explanatory diagram.
Make one decision at the next meeting
Use the themes as a starting point for investigation. Check appointment arrival and completion times, then seek feedback from different types of customers. If waiting really clusters in a particular period, try a limited staffing adjustment and observe the result. AI can help identify questions worth asking. A lasting cost commitment should rest on more complete business evidence.
- Review clues → check actual waiting records
- Limited change → observe before committing
Unchanged original source explanatory diagram.