What does it take to effectively test and verify speech-rec-based applications? This will be our focus, over the next several issues. With this knowledge you should be able to at least determine if your equipment vendor is doing the right verification tests before your system goes live; or supervise an independent testing organization in performing such tests.
Most would-be purchasers of speech-rec-based IVR need educating about factors that can affect real-world performance. A trade show bench test of an SR-IVR system can be made to look pretty good, in sterile test conditions. But does such a test give you any indication of how these systems will react to real calls from real people under real calling conditions? Absolutely not - as our CT Labs testers know from first-hand experience.
What do we mean by SR-IVR performance? Common sense, mostly. You want a product that answers calls on the first ring, plays speech prompts without audible breakups or quality degradation, responds to all spoken or touch-tone commands within a few seconds, and accurately recognizes spoken input a high percentage of the time - doing so for a wide variety of callers and under varying call conditions. And you want similar performance up to maximum-rated call loads - or at least not severe performance degradation.
Caller attributes and call conditions that conspire to unravel your SR-IVR system's performance include diverse caller demographics and accents, caller devices that don't always produce clean speech (i.e. cell phones in marginal reception areas, cheap speaker phones, or VoIP calls with low-bandwidth vocoders or high levels of data channel impairment). Not bad enough yet, you say? How about calling in from a cell phone in a marginal reception area WITH a high level of automotive wind noise mixed in? (Speech recognizers really like that one.) Add multi-line call loads and spoken commands that "barge-in" during prompt-playback, and you're starting to understand what a real-world SR-IVR system has to deal with.
As if all the above is not enough, SR-IVR platform resources can also adversely affect performance. If a system vendor is trying to keep their costs down, a slower processor with less than the ideal amount of host memory can impact performance, especially when the multi-line call loads start ramping up. Having enough host memory is critical since the vast majority of speech recognition engines now use the host processor. Low available memory can cause virtual disk-based memory to kick in which can slow a system to a crawl.
Moreover, no SR-IVR is an island. Most IVR systems maintain active connections to one or more corporate databases (e.g., customer info, shipping info, or other information that's being fed back to callers). Many of these databases are even connected remotely via the Internet or across contentious WAN or private network segments. So delays due to slow database server response are also possible. A properly-designed SR-IVR system will warn callers if a slow database might cause response delays to spoken commands. Without such a warning message, callers may not have a favorable opinion of your system and may not want to use it again.
The point of knowing all the factors that can affect performance of your SR-IVR system is this: we should now be able to develop tests that will VERIFY such systems under real-world conditions.
Next month: We will discuss SR performance metrics - like recognition accuracy rate- that should be directly or indirectly derived from an effective SR-based system performance test.
Chris Bajorek is co-founder of CT Labs, an independent full-service converged communications and IP Telephony product testing and certification lab. He can be reached at cbajorek@ct-labs.com.
