Nu Echo Blog

Here are the slides from my SpeechTEK 2009 presentation on grammar coverage analysis. Since part of the presentation was in the form of interactive demos, I will soon post screencasts to complement those slides. Stay tuned!

Most people know that IVR applications should be extensively tested before they’re deployed. They should be tested to make sure that they perform as specified and that they will continue to do so under load. Yet, most applications are not tested nearly as much as they should, sometimes with dire consequences when a major problem is detected after the application goes live. One reason for this, naturally, is cost. Another reason is time, as slips in the project schedule often mean less time available for testing.

That’s where automated testing comes in. Automated tests should be a critical element of any IVR testing strategy since they can greatly reduce the time required for application testing, lower development costs, accelerate software release cycles, and increase application quality and stability.

Unfortunately, commercially available testing offerings are usually costly, complex to set up and operate, or both, which is why they are so rarely used. To address this problem, we have developed the NuBot Automated IVR Application Testing Platform, which we demonstrated this week at SpeechTEK 2009 in New York City. The NuBot Platform is an affordable automated IVR application testing solution that is both powerful and easy to use.

During the NuBot Beta 2 Program announced earlier this week, the NuBot Platform will be available free of charge as a SaaS solution through the NuBot Hosting Service. If you’re interested to learn more about the NuBot Beta Program, please contact us at nubot-beta@nuecho.com.

Hurry because the Beta Program only lasts until the end of November, 2009.

As I mentioned in a previous post, on August 27 at SpeechTEK in New York City, I will be giving a SpeechTEK University course entitled Advanced Speech Application Tuning Topics. I thought it might be worthwhile for me to give a bit more detail about some of the specific topics I’ll be talking about.

So here are a few highlights:

• The “out-of-grammar” challenge - No matter what we do, users say things we didn’t anticipate. And, unfortunately, that happens quite a lot. It’s the harsh reality with which most speech applications have to deal and how we manage this challenge has a huge impact on success rate and user experience. I’ll present some of the most effective techniques we have been using to make sure that the application performs as optimally as possible in real conditions (i.e., dealing with real users).
• Are confidence scores good enough? - Confidence scores are essential in order to decide when to accept, reject, or confirm a speech recognition result. Unfortunately, confidence scores produced by recognition engines are often quite suboptimal, leading to unnecessary confirmations and dialog failures. We’ll show that it’s possible to get much better confidence scores.
• Identify problems with discriminative grammar weights - It’s well known that grammar weights can be automatically trained to learn the relative frequency of grammar alternatives. It’s not as well known that training discriminative weights can be an effective way to identify problems in a grammar. We’ll talk about this.
• Know where to focus - With limited amounts of time allocated to tuning, it’s important to be able to focus where tuning will have the biggest payback. We’ll talk about different techniques that help us find where the biggest problems are - and therefore, where improvements will have the largest impact.
• Confidence thresholds - Not long ago, someone on the Yahoo Voice User Interface Designers group complained about some application being too ‘confirmation happy’. But what’s the best way to determine confidence thresholds in a given dialog? As a matter of fact, what are good dialog-level performance metrics? We’ll show how dialog simulations can help us find thresholds that optimize your favorite performance metrics. We’ll also show how we can improve performance by using thresholds that depend on the recognition result.
• Rule-based expansion of phonetic pronunciations - Optimizing phonetic pronunciations is one of the most effective ways of improving speech recognition accuracy. Finding words that have recognition problems and fixing their phonetic pronunciations can bring large improvements. But how do you tune pronunciations for a 20,000-word vocabulary, especially when most of that vocabulary won’t even find its way into the tuning corpus? We’ll show how rule-based pronunciation expansion can bring surprising improvements.

These are just some of the topics I’ll be talking about. In the meantime, I’d be interested to hear about your ideas or experiences on these, or any other topic related to speech application tuning.

New York is the place to be on August 24-26! It’s the host of SpeechTEK 2009, the world’s biggest speech technology conference and exhibition!

Again this year, Nu Echo will have a very strong presence there, with:

• a SpeethTEK University course by Yves Normandin on advanced speech application tuning topics at (session STKU-6 at 9am on Thursday);
• a presentation by yours truly on speech grammar coverage analysis (session C302 at 11:45 am on Tuesday);
• a booth where we will demonstrate NuGram IDE, our flagship Eclipse-based development environment for speech recognition grammars, and NuBot, a full-featured automated application testing tool for both inbound and outbound IVRs; and
• a number of very exciting announcements; and
• of course, several of us from the Nu Echo team, who are looking forward to meeting you all!

Don’t wait any longer! Register online and use the registration code VIPNUE to get a 25% discount on the conference pass or a free exhibit hall pass.

Come see us at booth 513!

On August 27, I will be giving a SpeechTEK University course entitled Advanced Speech Application Tuning Topics.

This course will provide a synthesis of the speech application tuning methodology and techniques that we have been using - and continuously enhancing - over the past several years at Nu Echo. In essence, I will be describing the foundations (technical and methodological) of our tuning practice, which has proven so effective at delivering applications with very high success rates.

Even to those of you with significant tuning experience, I believe we will be able to provide a novel and, quite possibly, surprising perspective to this very challenging problem.

Here is the abstract, as it appears in the SpeechTEK program:

This course will teach participants a rigorous, data-driven speech application tuning methodology that will enable them to build robust speech applications that effectively deal with how real users actually behave, not how we would like them to behave. Topics include utterance and dialogue-level performance metrics, managing out-of-grammar utterances, techniques to effectively identify and address performance problems, dealing with multitoken utterances, tuning phonetic dictionaries, computing enhanced confidence scores, setting confidence thresholds, and running dialogue simulations. The presentation will be illustrated by numerous examples and interactive demonstrations using field data from real-life applications.

I am looking forward to seeing you there. And if you can’t make it to the course, please come see us at booth 513. I would be happy to give you a demonstration of some of the tuning tools we are using daily in our speech practice.