Design Guide: Evaluate Your UX
A simple usability study with real customers surfaces issues self-testing alone won't catch.
Who to test with
- Recruit 8–12 participants: fewer testers won't surface enough patterns to observe, while more can yield diminishing returns.
- Don't test with anyone who has been involved in the project or seen the design before.
When to test
- Test with customers on a stable build with major known bugs fixed: you want to observe usability issues rather than known technical failures.
How to structure the session
- Give participants one or two simple, realistic tasks framed as a scenario, not a set of instructions. Example: "You're planning a vacation for yourself and whoever you normally travel with. Your goal is to use Alexa to find and book a hotel."
- Let participants choose their own variables (such as destination, dates, and party size) to capture realistic decision-making.
- Remind participants they can use voice or touch.
- If your experience includes or ends in an expected error state (for example, payment not enabled in your test build), brief participants beforehand: "The experience will end with an error message. That's our signal you've completed the task."
Sample questions
Ask a few context-setting questions before the task to understand participants' baseline habits:
- How often do you [primary task your add-on enables]? For what purpose?
- How do you currently do this? What's your preferred method, and why?
- What factors matter most to you when [doing this task]?
- On a scale of 1–5, how satisfied are you with your current method?
After the task, ask:
- How do you think that went?
- How easy or difficult was it to learn how to use this experience? (1 = very difficult, 5 = very easy). Why that score?
- How easy or difficult was it to complete [the primary task]? (1–5). Why that score?
- How satisfied or unsatisfied were you with this experience overall? (1–5). Why that score?
- How confident would you be [completing the core action] this way? (1–5). Why that score?
- How likely are you to use Alexa for this in the future? (1–5). Why that score?
- How does this compare to your current preferred method? (1 = much worse, 5 = much better). Why that score?
- What would make this experience better?
Scores should be 4s and 5s, with a total average satisfaction score of at least a 4.3.
Related topics
| Go back to: Accessibility | Next: Testing Your Add-on CX |
Was this page helpful?
Last updated: Jul 21, 2026
