Tag Archives: grammar coverage

NuGram Speech Technologies application tuning

What’s the point of having 98% in-grammar accuracy if 40% of user utterances are out-of-grammar?

February 15, 2011 – 4:06 pm

How many times have you heard people say that they “achieve 95% speech recognition accuracy” (or more)? That sounds really impressive, doesn’t it?

It shouldn’t. What they don’t tell you is that they actually measure “in-grammar accuracy”, which means that accuracy is measured only on utterances that are perfectly covered by the grammar. For instance, for a date grammar, an utterance such as “well, uh, january fourth” would be considered out-of-grammar (and therefore ignored from the accuracy calculation) if “well, uh” is not covered by the grammar.

Unfortunately, in the real world there’s no way to force users to stick to in-grammar utterances. In fact, users usually have no way of even knowing what the grammar covers other than through hints provided by the prompts. Even well-behaved users can hesitate, correct themselves, or use an unexpected formulation (which sounds perfectly natural to them), all of which are likely to be out-of-grammar. They can even say things that they believe will help the machine understand them (for instance using “victor” instead of “v” when spelling).

As a result, it’s not unusual to have between 30% and 50% of user utterances that are considered out-of-grammar, many of which are perfectly legitimate responses to the application prompt. So what’s the point of reporting in-grammar accuracy if this ignores a large chunk of legitimate user utterances? You tell me.

Just to illustrate, you want to know one of the most effective ways of improving in-grammar accuracy? Just reduce grammar coverage. Sure, your out-of-grammar rate will increase but, hey, you’ll improve in-grammar accuracy! Isn’t that great? This tells you how useless in-grammar accuracy is at telling you whether you improved the grammar.

This is why we always report accuracy by considering every legitimate user utterance (i.e., the ones that contains a valid response to the prompt, regardless of wording or extraneous speech). This way, we make sure that we don’t conveniently ignore the utterances that happen to be the more challenging and we get results that accurately represent the real recognition performance (not some imaginary performance calculated on an idealized set of clean utterances).

But the best reason for doing it our way is that it enables us to truly measure improvements when we tune grammars. The reason is simple. Changing the coverage of a grammar always involves a trade-off. We can improve accuracy by covering more user utterances, but this can reduce overall accuracy if the new grammar paths introduce new speech recognition errors. The only way we can measure improvement is if we measure accuracy on a fixed set of valid utterances that doesn’t depend on the actual grammar coverage.

Tags , , , , | Comments (2)
Development NuGram Speech Technologies

A proven yet simple grammar conversion process

November 4, 2010 – 11:35 am

Grammar Conversion Process

As old speech recognition engines are being replaced by newer ones, we see more and more organizations having to convert their old grammars to standard formats. Given the right process and set of tools, converting grammars from one engine to another should be a straightforward task with mostly no risk of breaking the associated IVR application.

The issues

There are several issues associated with the conversion of grammars:

  • Syntax. First, there is the syntax of the grammar itself. If we are converting a grammar bewteen two engines that support GrXML or ABNF, then there’s not much else to say. But if we are converting from Nuance GSL to GrXML or ABNF, that’s a different story. GSL has very different operator precedences than ABNF, for instance. We have to be careful.
  • Semantics. The second issue is the language used inside the semantic tags. Again, if both engines support SISR, we have nothing to do. But if we convert from GSL to ABNF+SISR, we may have a harder time. For example, SISR does not support the concept of a top-level slot that can be assigned from anywhere in the grammar (using the <slot value> syntax).
  • Pronunciation lexicons. Almost all speech engines use a different format for lexicons. Not to mention that even different versions of the same engine sometimes support different phonetic alphabets.

A proven process

If you follow a rigorous process, the first two issues above can be easily mitigated. Here is one that has proven very effective:

  1. A coverage test set is produced from the original grammar. The test set should ideally ensure that all semantic tags are executed at least once (this is not always sufficient if the semantic tags contain conditional code, but that’s a good starting point.)
  2. The grammar is converted to the new format.
  3. The converted grammar is tested against the coverage test set of the original grammar (and problems are fixed, if any, until all tests pass).

Some tools

Some ASR engines already provide tools to convert grammars from old proprietary formats to the new standard ones. For instance, Nuance ships a tool to automatically convert GSL grammars to GrXML + SISR. It does not support all features of GSL as some of them have no equivalence in GrXML and SISR. And one of the problems with this converter is that the semantic tags produced are not easily maintainable.

NuGram IDE also provides some tools to help with the above process. In particular, it offers:

  • Great support for creating and running coverage tests.
  • A sophisticated sentence generation tool. The tags coverage strategy, for instance, is very effective when converting grammars as it helps generating sentences that will cover all semantic tags.
  • Support for all major semantic tags formats (GSL, Nuance OSR extensions, IBM and Microsoft, etc.).

Of course, to use NuGram effectively, your grammars will need to be converted to ABNF first. No problem! NuGram provides GSL and GrXML to ABNF converters to help you, as well as converters from ABNF to GSL or GrXML. That means all you have to worry about is really the conversion of the semantic tags. In this case, the whole process now becomes:

  1. Grammars are first imported in ABNF.
  2. A coverage test set is produced from the original grammar.
  3. Semantic tags are converted.
  4. The converted grammars are checked for errors by running the coverage tests of the original grammars. In case of errors, they are fixed and all tests are re-run.
  5. Convert the grammars to the desired target format.

What about pronunciation lexicons?

Unfortunately, converting phonetic dictionaries is still a manual and error-prone process, for which there are no good solutions as of this writing. And this task is more part of the tuning process that follows the grammar conversion process anyway. In most cases, a grammar’s pronunciation lexicon is used to fix incorrect or missing pronunciations in the ASR engine’s own dictionary for very specific words. The phonetic dictionary of the target ASR engine may not have the same limitations or deficiencies. At best, the original grammar’s pronunciation lexicon can act as an inspiration for the creation of the new pronunciation lexicon.

Tags , , | Comment (1)
Speech Technologies tuning

Reducing false accepts with decoys

January 12, 2010 – 8:57 am

As discussed in a previous post, one of the unfortunate consequences of out-of-grammar utterances is that they can cause many false accepts that may seriously degrade application performance and user experience. In order to illustrate this, let’s use the simple example of a small menu where callers must choose among three options: “validate”, “repeat”, and “cancel”.

We use a test set of 5042 field utterances distributed as follows:

Menu choice Number of utterances Proportion of test corpus
cancel 367 7.28%
repeat 896 17.77%
validate 3478 68.98%
OOG 301 5.97%

As we can see, this is a fairly clean test set with only about 6% of out-of-grammar utterances. As usual, these include background speech, various noises, side conversations, some common OOG utterances (“yes”, “no”, “okay”, “options”, “oh”, etc.), as well as a wide variety of rambling responses of different kinds.

Naturally, since the grammar can only recognize one of the three keywords (and legitimate variants), most of these OOG utterances are misrecognized as one of the keywords. That wouldn’t be a problem if the corresponding confidence scores were low and we could safely reject them, but that’s not always the case. In fact, many of these have a confidence score over 0.9, resulting in damaging false accepts.

An effective way to reduce false accepts is to add decoys to the grammar. For instance, you would normally want to start by adding common OOG responses, on the ground that it’s easier to reject an OOG utterance if you can recognize it correctly. You could also add more “general” decoys, for instance a phoneme loop, to help reject hard to predict OOG utterances. There are more advanced techniques that can be used in order to come up with “optimal” decoys for a given grammar, but I won’t go into them now.

In all cases, it is of course absolutely necessary to evaluate, on a large enough test corpus, the impact of these decoys since they could easily end up reducing recognition accuracy, sometimes significantly. In particular, one should be careful not to add decoys that could be confused with legitimate sentences or keywords.

Let’s illustrate the impact of decoys using the set of field utterances described above. The graph below compares the performance of a grammar without decoys (red curve) to that of the same grammar to which appropriate decoys were added (blue curve).

As can be seen, even for for a fairly clean test corpus with a low OOG rate, the addition of decoys can significantly improve performance. For instance:

  • For a False Accept rate of 0.5%, the Correct Accept rate increases from 95% to over 97.5%, which is equivalent to reducing the error rate by more than 50%.
  • For a Correct Accept rate of 97.5%, the addition of decoys decreases the False Accept rate from 1.5% to around 0.3%. That’s one fifth the False Accept rate for the same Correct Accept rate.

Another interesting observation is the impact of decoys on confidence thresholds. Let’s say we want to have a False Accept Rate of 0.5%. Then, we would need to use a confidence threshold of 0.73 for the grammar without decoys, but only  0.25 for the grammar with decoys. That’s quite a difference! This clearly shows that using “default” threshold values may sometimes produce results that are quite inadequate.

All of this once again demonstrates how important it is to pay close attention to out-of-grammar utterances in a tuning process and how decoys can provide an effective tool for containing the negative impact of such utterances on application performance.

Tags , , , , | Comment (0)
NuGram Speech Technologies

Grammar problem #1 – repeated tokens

December 9, 2009 – 10:40 am

It is quite easy to write a speech recognition grammar. After all, it’s only a text file. And with the help of a good editor, we can expect the grammar to be free of syntax errors, i.e. to conform to the SRGS specification (ABNF or XML).

The real challenge is in making sure that the grammar does not contain any “error” from the point-of-view of the set of sentences it accepts, and the semantic values it associates to these sentences. We also have to make sure that it does not over-generate (accept sentences that are not part of the usual spoken language or cannot be uttered in the context of the question asked).

This post is the first in a series that will show the most common types of errors made when developing grammars and how they can be found and fixed using the advanced features of NuGram IDE. The examples I’ll use are all variants of grammars we’ve seen in the course of our grammar developments projects, either internally or for our customers.

Problem 1: Repeated Tokens

We’ll start this series with a very simple one. Suppose I’m editing a long list of ordered tokens. It is very tempting to copy one of the items and paste it as many times as needed and edit the copies. I do this all the time. It’s such a common pattern in text editing (and programming, unfortunately…) Of course, it is very easy to forget editing one of the copies.

For example, let’s say I want to write a number grammar. I’ll start writing something like:

public $r1To9 =
  one {out.number=1} |

and then copy/paste the first item 8 times, replace “one” by the digits “two” to “nine” and do the same for their corresponding semantic value. I’ll get something like:

public $r1To9 =
  one {out.number=1} |
  two {out.number=2} |
  three {out.number=3} |
  four {out.number=4} |
  five {out.number=5} |
  five {out.number=6} |
  seven {out.number=7} |
  eight {out.number=8} |
  nine {out.number=9}
;

Of course, you’ve already seen the error (probably much faster than I have). It’s easy here since you have the offending fragment right before our eyes. But when developing a grammar with many rules, it may not be that obvious. And even carefully reviewing the grammar may not suffice. (How many times do we miss typos in our own texts that another reviewer finds in a matter of seconds?)

So how do we find the problem? In this case, I simply need to build a coverage test set with the sentence generator using the Tags Coverage strategy. The following video shows how to do that:

Of course, in this example, I knew how to proceed to find the problem. In practice, and this will be a recurring idea in the series, a grammar needs to be tested in many different ways. There are many techniques and tools that need to be applied. The Tags Coverage (or the All Paths) generation strategy is often the first we use. It has the advantage of exercising all the semantic actions and finding lots of potential problems very early on in the debugging process.

In my next post in this series, I’ll write about ambiguities, how they affect speech recognition performance, and how to detect and deal with them.

Tags , , , | Comment (0)
Speech Technologies tuning

You can only tune what you can measure

November 23, 2009 – 1:44 pm

This is the first in a series of posts I’ll do on speech application tuning over the coming weeks. Hopefully, this will provoke interesting feedback and, who knows, even spark some lively discussions.

I’m starting with the very important topic of speech recognition metrics because that’s necessary in order set the stage for most of what I’ll talk about next. Although this may not be the most exciting topic, it is clearly a very important one.

Some terminology

Tuning a speech application involves attempts to optimize a certain number of key performance metrics. Improvements or deterioration of these metrics is what tells us whether or not we’re making progress. Although that should be intuitively obvious to most people, what’s perhaps less obvious is how to select metrics that correlate best with the application’s success rate and user experience in the field.

Let me start by defining some terminology:

  • In-grammar / out-of-grammar —This determines whether an utterance is covered or not by the grammar. Later on in this post, I’ll talk at length about the different ways the word “covered” may be interpreted.
  • Accepted / rejected —This determines whether the recognition result’s confidence score is greater (accepted) or smaller (rejected) than the given confidence threshold. Note that there may be more than one confidence threshold for a given recognition context. If we’re talking about the high threshold, then “accepted” usually means that no confirmation is required. If we’re talking about the low threshold, then “accepted” means a confirmation will be required and “rejected” means that the user needs to be re-prompted.
  • Correct / incorrect —This determines whether or not a recognition result is correct. Although the definition of “correct” may vary, it is often interpreted to mean that the top recognition result in the N-best list has the correct semantic result (i.e., it doesn’t matter that not all words were correctly recognized as long as the semantic result is correct). Note that we assume here that only in-grammar utterances can be classified as either correct or incorrect.

When we perform a recognition test for a grammar using utterances collected in the field, we compute a set of 6 counters for each confidence threshold value in a range from 0.0 to 1.0. These counters are:

  • AC — Number of in-grammar utterances that are accepted and correct (often called CA-in in the industry)
  • AI — Number of in-grammar utterances that are accepted and incorrect (often called FA-in)
  • RC — Number of in-grammar utterances that are rejected and correct
  • RI — Number of in-grammar utterances that are rejected and incorrect
  • Aoog — Number of out-of-grammar utterances that are accepted (often called FA-out)
  • Roog — Number of out-of-grammar utterances that are rejected (often called CR-out)

Note that the value FR-in, often seen in the industry, is equal to RC+RI. We like to keep these two values separate since they allow us to distinguish recognition errors from rejection errors. We add two important variables, that are computed from the above counters:

  • ing — Number of in-grammar utterance (= AC+AI+RC+RI)
  • oog — Number of out-of-grammar utterances (= Aoog+Roog)

With these, we define the two key metrics that we’ll use constantly:

  • Correct accept rate (CA-rate) —This is the percentage of in-grammar utterances that are accepted with a correct result. It is computed as CA-rate = AC/ing.
  • False accept rate (FA-rate) — We use two versions of this metric:
    1. The percentage of all utterances that are incorrectly accepted. It is computed as: FA-rate = (AI+Aoog)/(ing+oog) = (AI+Aoog)/all
    2. The percentage of accepted utterances that are incorrect. It is computed as FA-rate = (AI+Aoog)/(AC+AI) = (AI+Aoog)/A

Here’s an example that will hopefully help clarify all this. The following graph plots the CA-rate as a function of the FA-rate for a phone number recognition experiment. I’ll use this type of graph constantly, so you might want to familiarize yourself with it. Note that, in order to avoid any confusion, the axes are labeled with the metric’s definition, not its name.

In the graph, the hidden variable is the confidence threshold. As the confidence threshold decreases from 1.0 to 0.0, both the CA-rate and the FA-rate increase. If we are using two thresholds then we would want to set the high threshold so that the FA-rate is very low (less than 1%, say), while the low threshold would be set in order to have an appropriate balance between confirming too many incorrect results and rejecting too many correct results.

Using a graphical representation of results has several advantages. One advantage is that it provides a visually clear view of how effectively we avoid false accepts. A curve that grows slowly from left to right is a clear indication that we’re not effectively rejecting out-of-grammar utterances. We’d like the curve to initially have a very steep slope and then to taper off when the CA-rate gets close to the maximum value.

Another very important advantage is that it makes it easy to compare results from different experiments. A curve that’s above another immediately tells us that it’s a better result (for a given FA-rate, we have a better CA-rate). That, however, assumes that the results truly are really comparable, which brings us back to an issue that we had earlier postponed: The definition of “in-grammar”.

The importance of correctly defining “in-grammar”

I had mentioned earlier that an in-grammar utterance is an utterance that is “covered” by the grammar. But what does “covered” mean? For much of the industry, this simply means that the sentence transcription can be parsed by the grammar.

This definition turns out to be quite problematic. Fundamentally, the main problem is that the in-grammar utterances are those we consider valid and therefore those that we should recognize as best as possible while out-of-grammar utterances are those the application should be rejecting. However, the definition of “valid” should be based on what application users perceive, not on what we have decided that the grammar should cover. If, for speech recognition accuracy considerations, we decide not to cover certain forms of user responses that are not used very often, this doesn’t make them any less valid. It just makes them less frequent.

Let me illustrate this with a simple example. The graph below shows the results from three date recognition experiments. The blue curve shows the result obtained using a grammar that supports two main forms: <month><day>[<year>] (“January fifth”) and <day><month>[<year>] (“Fifth of January”). Let’s say we want to see what happens if we decide not to support the second, rarer form. The result is the red curve which, as we can see, seems to indicate better performance.

But that’s an illusion. The problem is that the red curve considers all dates of the form “Fifth of January” as out-of-grammar. We’re of course doing a better job of recognizing the now reduced set of in-grammar utterances, but the problem is that a larger proportion of user utterances are considered invalid. In other words, we’re now correctly handling a smaller proportion of valid user utterance. From the user’s perspective, that’s certainly not an improvement. In fact, if we consider both forms as in-grammar (i.e., valid), then we get the green curve, which clearly tells us that we indeed have considerably deteriorated results.

In order to get meaningful results, we need to determine which utterances are in-grammar and which are out-of-grammar based on what should be considered a valid response, regardless of what we decide to include in our grammar. This has several important benefits:

  • It makes it possible to get meaningful comparisons between results obtained with grammars having different coverages since the sets of in-grammar and out-of-grammar utterances is the same in all cases.
  • We get results that are much more representative of user experience. To make sure that this is the case, we normally decide that an utterance is “in-grammar” if a human listener would consider this to be valid and unambiguous response to the question (within an acceptable “domain” of responses).
  • Last, but not least, we get applications that deliver similar or better success rates with fewer confirmations, and therefore better user experience. The reason is that, very often, many sentences that can’t be parsed by the grammar nonetheless give a high-confidence, correct semantic result. If these were considered out-of-grammar, then they would become false accepts. It takes very few of these to have a significant impact on the FA-rate, with the unfortunate consequence that we would end up using confidence thresholds much higher than necessary, resulting in many unnecessary confirmations.

So this pretty much concludes what I wanted to talk about today. In the next post, I’ll talk about tuning challenges related to out-of-grammar utterances.

Tags , , , | Comment (0)

Next Page »