Published: December 1, 2023
Key takeaways
Among the common reasons skills might fail any of the tests performed to certify your skill, you can prevent many of them early in your design process.
Need quick advice?
Read the Checklist for common design-related certification failures
In this article:
When you submit your skill to the Alexa skill store, Amazon performs the following tests to certify it. These test include:
- Functional tests verify that the skill works as intended
- Policy tests verify the skill follows Alexa policy guidelines and is rated appropriately
- Security: Verifies the skill’s hosting method meets Alexa security requirements
- Experience. Skills need to have a robust voice user interface that customers can use to interact with it effectively
Your skill design will need to accommodate especially (2) the need to meet policy requirements and (4) creating a robust conversation design. Some of the most common certification failures can be prevented when you design your skill aware of these requirements first.
There are a few design-related reasons that skills often fail certification, and they are easy to prevent in the early stages while you’re still creating a skill. This article will cover these common failures and design solutions for them.
Learn more about the Trustbusters that can cause certification failures.
Learn more about Natural Speech to prepare your design for UX certification (and your customers!)
First, determine whether specific policies apply to your skill, such as if it deals with financial information, is created for children, handles information about health, and more.
Checklist for preventing common design-related certification failures:
▢ Check that your interaction model contains a robust set of customer utterances for each intent
▢ Check that your interaction model does not contain intents with the same utterances
▢ Include a “welcome message“ when customers invoke your skill without a request for the first time
▢ Include a “welcome message” for customers who have used your skill repeatedly
▢ Your skill should offer help when the customer reaches an error
▢ Your skill should respond when the customer asks directly for “help”
▢ Include proactive messaging about account linking in your skill
▢ Send an account linking card to the customer’s Alexa app when appropriate
▢ Test your account linking experience with a large sample size of customer testers
▢ Indicate whether the skill session stays open or closes after each response
▢ Check your skill’s dialog for spelling, punctuation, and grammar mistakes
▢ Test and listen to all your skill’s dialog for TTS errors on an Alexa device
Problem: Not enough utterances for each of your intents
Result: The customer can’t speak naturally to your skill
Solution: When you create your Interaction Model (IM) for your skill, you’ll need to include a robust list of things the customer can say to reach each intent and to fill each slot. (Learn more about Intents, Utterances, and Slots.) When customers say something you didn’t anticipate and add to your IM, they will encounter an error instead of the response you intended. When you submit your skill for certification, also remember to thoroughly test the sample utterances you include.
Bad:
Customer: Alexa, ask Sports Wire for today’s football highlights.
<“Today’s football highlights“ is not in the interaction model>
Matthew: Today on Sports Wire, you can ask me about baseball, soccer, and football preseason. What would you like?
Customer: Preseason
<“Preseason“ used by itself is not in the interaction model>
Matthew: Sorry. I’m not sure about that. Today you can ask me about baseball, soccer, and football preseason.
Customer: Football!
<“Football“ used by itself is not in the interaction model>
Matthew: I can tell you about football preseason, or play the radio highlights. Which would you like?
Customer: Highlights please
<Pleasantries like “please” aren’t in the interaction model>
Matthew: Sorry. I’m not sure about that. You can say “football preseason” or “radio highlights.” What do you want?
Customer: I want the radio highlights!
<Natrual syntax like “I want…” isn’t in the interaction model>
Matthew: Sorry. I’m not sure about that. You can say “football preseason” or “radio highlights.” What do you want?
Customer: Radio. Highlights.
Matthew: You got it. Here are today’s radio highlights.
Better:
Customer: Alexa, ask Sports Wire for today’s football highlights.
<Football is in the interaction model, but has two possible matches>
Matthew: Hey, it’s Sports Wire. Did you want to hear about football preseason, or the radio highlights?
Customer: Give me the radio highlights.
<Natural syntax like “give me…” are included in the interaction model>
Matthew: You got it. Here are today’s radio highlights.
Best:
Customer: Alexa, ask Sports Wire for today’s football highlights.
<“today’s football highlights“ maps directly to one function in the interaction model>
Matthew: You got it. Here are today’s radio highlights on Sports Wire
Problem: Missing or unhelpful welcome message
Result: The customer can’t invoke the skill without a request; or when they invoke a skill without a request, they receive an unhelpful response.
Solution: Your skill will need to include a “welcome message,” the speech customers will hear when they invoke your skill without making any additional request. This message should tell customers a little bit about your skill. Because come customers will only every launch your skill this way, you will need to include less detail about your skill over time, lest it become repetitive and annoying (which affects Retention).
For example, the first time a customer visits your skill, it might sound like …
Customer: Alexa, start plant pal
Brian: It’s your Plant Pal, here to help you be a pal to your plants. You can ask me questions about houseplants, or link an account to start tracking your own plants. How can I help your garden grow today?
After a customer is familiar with your skill, it might welcome them back like this …
Customer: Alexa, ask Plant Pal to check on my plants.
Brian: Welcome back, Plant Pal. Your plants have missed you! (And so have I). It’s time to check on your Monstera, and 13 more plants. Ready to get started?
Learn how to use a Large Language Model to add variation to your skill’s responses. <live article link here>
Problem: Missing or poor help experience
Result: The customer can’t get more information about how to use the skill; or the help the skill gives doesn’t help the customer solve their problem.
Solution: A customer should always be able to ask your skill for “help” directly, or be proactively offered it if they reach two consecutive errors. A good help message tells customers briefly about what the skill can do, and end with a question that prompts the customer to respond further. The message should also be contextually relevant. If, for example, your skill searches for and books fitness classes, your skill shouldn’t say the same thing if the customer asks for help while searching versus asks for help while paying. For example …
Customer: Alexa, ask Sports Wire for football
Matthew: On Sports Wire, today in football, I can give you the preseason schedule, or play the radio highlights. What’ll it be?
Customer: I don’t know.
Matthew: With Sports Wire Basic, you can ask me about schedules and upcoming games, or play radio highlights. Or you can upgrade or log in to play live games. What would you like?
Customer: Seahawks
Matthew: Want to play the seahawks radio highlights?
Customer: Help me play the game
Matthew: To save and play favorite teams and shows, you’ll want to link an account. I sent some information to your Alexa app to help you. For now, I can play live radio. How about KLMN 99.9, Seattle sports talk?
This hypothetical skill both offered help when the customer directly said “help,” as well as proactively offering it when they indicated they might need help (“I don’t know”).
Learn more about preventing and handling errors.
Problem: Account linking failures
Result: Customers don’t know how to or fail to account link. There is no voice response to a customer asking about account linking. Account linking cards fail.
Solution: Carefully design an account linking conversation and proactive messaging about account linking in your skill so customers understand they need to use their Alexa app. Have a helpful answer ready for when customer asks “How do I link my account?” or “Is my account linked?” and similar. Account linking is important for highly personalized experiences, creating a companion to an existing app, and for making purchases of goods, and it’s important for retention and monetization to make the process as easy as possible for the customer. Don’t leave account linking out of your testing with real customers before you launch. For example:
Customer: Alexa, start plant pal
Brian: It’s your Plant Pal, here to help you be a pal to your plants. You can ask me questions about houseplants, or link an account to start tracking your own plants. How can I help your garden grow today?
Customer: I want to track my plants.
Brian: To start tracking your plants, you’ll need to link a Plant Pal account. I sent some details to your Alexa app to help you. In the meantime, I can answer your questions about houseplants, pests, care, warnings, and more. How can I knock your stalks off today?
Learn more about designing a skill for Account Linking
Problem: Poor session management
Result: The session closes immediately after the skill’s speech, or the microphone opens for the customer’s response without asking a question.
Solution: Each time the skill responds to the customer, you must decide whether to end the session, or keep it open with a follow-up prompt before opening the mic. For example …
Don't:
Customer: Alexa, ask Plant Pal if any of my plants need water
Brian: Welcome back, Plant Pal. Your Fig Tree and 16 more of your plants might need water. You can say “start watering” or do something else.
Do:
Customer: Alexa, ask Plant Pal if any of my plants need water
Brian: Welcome back, Plant Pal. Your Fig Tree and 16 more of your plants might need water. Ready to start?
Learn more about closing the skill session.
Problem: TTS errors
Result: Alexa doesn’t surface a response; or Alexa mispronounces words, acronyms, or misuses two words with the same spelling; or there are errors in your SSML.
Solution: If you’re using SSML, ensure there are no errors that cause undesirable sound (or lack of sound at all). Check for punctuation errors if you notice awkward pauses or clipped speech. If Alexa is mis-pronouncing words, check for spelling errors, then try using the “say-as” tags for correcting issues with abbreviations, characters, acronyms, numbers, and more.