Application Design

• fully conforms to the specification (voice user interface) you created during the design phase
  
• meets the business need you first set out to fulfill

This is a tall order. Testing is neither easy nor fun, but it is essential. Testing separates decent applications from great applications. If you don't test, your callers (customers) will test, and they will react much more differently than you when they find a problem. What they experience will reflect on the company. Rolling out an untested application will have the opposite effect of the original intentions that caused you to want the application in the first place. If you intended to save cost, you will increase the cost by having to deal with customers who are upset and who need to go through several iterations to accomplish the task they set out to do. If you intended to raise the quality of customer service, you will decrease it by skewing the caller's perception about how much your company cares about them. "How can they say 'your call is important to us' if they keep me in this infinite loop and then hang up on me!!" - Annoyed Customer.

We are going to focus on 3 kinds of testing. Each one builds on the strength of the previous one. If you don't have time to do every type of testing, start at the beginning, then proceed down the list.

Functional Testing
This is a simple battery of procedures that asserts whether the application conforms to the voice user interface design.

• Traversal test: The application transitions to all the proper dialog states from the current state according to the responses outlined in the spec
  
• Prompt Completeness test: The application plays all the proper prompts on all specified conditions (no input, no match, confirmation, response out of range)
  
• Transaction test: The application successfully displays the desired behavior for all possible data combinations (e.g.: order processed, order cancelled, no order found, etc.)

Usability Testing
Usability is an entire discipline with seasoned professionals devoted exclusively to it. Draw on their help if you can. They tend to test for the following conditions:

• Speech Recognition test: Is the system accurately recognizing the spectrum of utterances (what callers say) that my customers will speak?
  
• Task Completion test: What percentage of callers accomplishes the task they set out to do? How does this compare to how it was accomplished before? What are the reasons for abandonment?
  
• User Expectations test: Melissa Dougherty from VoicePartners, a renowned expert, suggests (QuickTime) five important principles:
  
  • Look at the call from the caller's perspective
  • Help callers anticipate what's next
    
  • Support the user
    
  • Use characteristics of natural conversations
  • Recognize that their experience reflects on you

Usability testing involves releasing your application into a controlled group, and gathering feedback from their interactions. This may encompass call recording, surveys, and analysis of logged data.

Load Testing
These tests attempt to unearth potential problems that will occur when a lot of callers simultaneously attempt to use the system. If the system replaces live traffic, you will have a good handle on the required throughput (callers per hour).

You can find these Erlang calculators useful.

Things to test:

• System response: As traffic increases, do responses slow down? At what point do response times become unacceptable? (a commonly accepted threshold is 2.5 seconds)
  
• Transaction throughput: How long are transactions (dynamic access points) taking to execute on average? What is the worst case?
  
• Lines into agents: Do I have enough agents to handle calls once they get out of the automated system? At what point does the experience degrade because of wait times?