Grammar conversion : lessons learned

Lately, I have been involved in a number of grammar conversion projects. This has been a great opportunity to put our process and  tools to the test once again. And since every project has its peculiarities, we learn constantly.

The process we outlined about a year ago omitted  a number of small details. That was OK for small scale conversion projects. But when you have to deal with much larger projects (with thousands of grammars to convert), these details add up significantly. Let me share some of the issues we face daily.

It’s not just semantic tags

When you have tools to automatically convert semantics tags from one format to another, grammar conversion can seem to be a no-brainer. But reality is not that simple. Grammars are not written for an abstract specification, they are written for a very specific recognition engine. They often contain:

  • Words (tokens) that map to very specific pronunciations or that try to model some disfluencies (like hesitations, for instance), but for which the SRGS $GARBAGE rule is more appropriate.
  • Multiword duplicates, with one sequence of space-separated words, and a similar sequence of underscore-separated words to allow cross-word phonetization (like “thirty one” and “thirty_one”).
  • Words that map to very specific, tuned pronunciations. Such words often have an unusual orthography to make sure they are not confused with real words.

All this means that there are a number of transformations either to the original grammar or to the converted grammars that must be applied. This can be by means of regular expression search&replace, or manually inspecting grammars.

Generation of coverage sets

When dealing with hundreds (if not thousands) of grammars, it is not feasible to create initial coverage test sets manually. This is way too time consuming. That means you have to find a way to generate those initial coverage test sets automatically in batch. But how do you do that?

Fortunately, NuGram IDE already provides sophisticated tools to analyze grammars and generate sentences from them. We just built on this foundation a tool to automatically generate coverage tests sets for a set of ABNF grammars. The tool also reports problems found in the grammars, like the use of digits in voice grammars, or words in DTMF grammars.

The coverage set generation tool uses a combination of  configuration and sophisticated analyses to determine how to generate sentences and how many sentences to generate. For example, it’s not possible to generate all sentences from a grammar that covers an infinite number of sentences. When that’s the case (or when the number of sentences covered by the grammar is above a certain threshold), the tool reverts to other generation strategies.

Recognition tests as part of the QA process

Finally, even a syntactically valid grammar may fail to load in the ASR for a variety of reasons, the most common one being a limitation or constraint from the ASR  itself. For this reason, we got to the conclusion that doing recognition tests (ideally benchmarking of the converted grammars) is a very useful addition to the QA process. Of course, simply compiling grammars may catch a number of problems. But doing a “before and after” comparison can detect conversion problems that were not caught by the coverage tests when they are not exhaustive.

Another benefit of doing recognition tests is the ability to check the performance of the converted grammars to identify those needing additional work. Some converted grammars may have words that prove difficult to recognize with the new engine because they are not properly phonetized, thus calling for application-specific (or even grammar-specific) phonetic dictionaries.

What about DTMF?

In the specific case of converting GSL grammars to GrXML or ABNF,  a complication arises with the presence, in the same grammar, of both DTMF sequences and words. I will discuss this issue in a separate post.

NuBot 3.0 on the starting blocks: what’s new!

Because our latest version of our NuBot Platform product is on the verge of being released, I’d like to present a few of the noteworthy new features out in this release.

Enhanced Composite States

Composite states now feature full-fledged encapsulation. In our previous installment, because of the limitations of the transition model, composite states had to be aware of their parent to function properly, greatly limiting the ability to reuse callflow components in some specific cases. Composite states now give more freedom in this regard, thanks to our new transition model, which gives you the ability to break down a transition into two parts, where one part would be defined in the parent, and the other part in the child.

UI Cosmetic Improvement

As part of our effort to offer the best user experience, we have improved a few UI items which should make your life easier. For instance, any section content available from the Results view now features a copy to clipboard button. Also, the Callflow editor now provides the ability to add comments to the layout or move transition labels while offering a more pleasant color scheme. We have also removed some under-used or plainly obtrusive features which did not add any value to the experience.

Callflow Anchor Points and new Transition Model

Our callflow transition model has been refined and now features the ability to break down a given transition into two distinct parts. This characteristic gives you the ability to not only lay out your callflow more easily by routing transitions around states but also enhance your Composite States encapsulation.

Augmented Test Descriptor Editor

We have moved some of the configuration parameters off the Preferences to the Test Descriptor Editor itself, where it makes more sense. All test configuration parameters are now neatly organized into one single location. Such parameters include Call Profile and Call Sampling.

REST Communication Protocol

In order to facilitate both customer firewall and forward proxy traversal, we got rid of the RMI protocol in favor of a friendlier HTTP one. While from a user perspective, this change does not bring much to the table, thanks to a nicely encapsulated communication layer, this protocol overhaul opens the door for a plethora of new integration schemes. For one, you can now use our HTTP API to not only tap into our test call data set but also interact with the platform itself, launching a given test at a specific time from the command line, or PUT/GET test resources on/off the server. The API can also be used to monitor your minutes usage or simply to poll for server status.

# get all scheduled test
curl -X GET -H "X-nuecho-access-key: foobar" \
-H "X-nuecho-secret-key: $4$02Y4dYz+$d61BHuJq/GqylW0p6jVzs1/Arxs$" \
'https://nubot.nuecho.com/api/v0.1/foobar/operations/scheduled'

# delete resource bucket
curl -X DELETE -H "X-nuecho-access-key: foobar" \
-H "X-nuecho-secret-key: $4$02Y4dYz+$d61BHuJq/GqylW0p6jVzs1/Arxs$" \
'https://nubot.nuecho.com/api/v0.1/foobar/resources/3868dd95-a81e-4480-aca6-fa05012075ff'

# get stats from audio service
curl -X GET -H "X-nuecho-access-key: foobar" \
-H "X-nuecho-secret-key: $4$02Y4dYz+$d61BHuJq/GqylW0p6jVzs1/Arxs$" \
'https://nubot.nuecho.com/api/v0.1/foobar/audio/stats'

# launch a test session
curl -X PUT -H "X-nuecho-access-key: foobar" \
-H "X-nuecho-secret-key: $4$02Y4dYz+$d61BHuJq/GqylW0p6jVzs1/Arxs$" \
'https://nubot.nuecho.com/api/v0.1/foobar/operations/launched/3868dd95-a81e-4480-aca6-fa05012075ff?recording=false'

Performance Optimizations

The platform has been re-architectured to handle larger call volumes. Internal services can now be horizontally scaled for high throughput and greater performance. More than ever, you can use our NuBot platform to not only perform regression or functional testing but also to launch much larger load or stress tests.

Conclusion

I hope you are as thrilled as we are about this upcoming release and that you will see benefits from this sneak preview! I intend to present some of those new features in separated posts in the coming weeks.

Grammar problem #2 – ambiguous grammars

While working on a grammar conversion project from Nuance GSL to SRGS ABNF, I stumbled upon a few grammars all having the same design problem: using optional parts to make a few words repeat a varying number of times. This is a pattern we’ve observed regularly on various projects.

Here is an example of such a grammar for recognizing sequences of 4 to 8 digits (I omitted the semantic tags for clarity):

#ABNF 1.0 ISO-8859-1;

mode voice;
language en-US;
root $digits4To8;

public $digits4To8 =
  $digit $digit $digit $digit [$digit] [$digit] [$digit] [$digit]
;
...

The original GSL grammar looked like this:

Digit4To8 (
  Digit Digit Digit Digit ?Digit ?Digit ?Digit ?Digit
)

The GSL syntax does not support the <N-M> syntax like in ABNF to repeat an expansion from N to M times. That’s a reason why the grammar was written this way in the first place. In ABNF grammar , it would have instead been written as:

#ABNF 1.0 ISO-8859-1;

mode voice;
language en-US;
root $digits4To8;

public $digits4To8 = $digit <4-8>
;
...

In GSL, it would have been better to write the grammar as:

Digit4To8 (
  Digit Digit Digit Digit ?Digit1To4
)

Digit1To4 ( Digit ?Digit1To3 )
Digit1To3 ( Digit ?Digit1To2 )
Digit1To2 ( Digit ?Digit )

Both grammars are equivalent, right? So what’s the problem?

Ambiguities

Well, both grammars recognize the same language (the same set of sentences), but the first grammar has a very different behavior. It is highly ambiguous. That means some sentences can be parsed in two or more different ways. See what you get when you interpret one such sentence in NuGram IDE:

The interpreter tells us (at the top-left of the window) that there are 6 different parses for the sentence. (I’ve seen grammars generating more than 100 parses for a given sentence!).

The problem with ambiguous grammars is they can impact both recognition accurary and recognition performance. Suppose a grammar covers a sentence that is highly ambiguous and another sentence which is not, but is phonetically close to the former. Since speech recognition engines limit their recognition search space, it is possible that the latter be pruned from the search space at the beginning of the recognition window even if it’s the one that would come up with the best score at the end of the recognition.

The other problem is recognition performance. All semantic tags are typically executed at the end of the recognition process, once the user has finished talking. If there are lots of identical hypotheses with the same score, the recognition engine will have to execute all tags (interpreted ECMAScript code), most of them being redundant and useless, thus causing longer delays in the speech application.

Determining that a grammar is ambiguous (or not) is a very hard problem (it’s an undecidable problem). That means, whatever tool you use that’s supposed to decide for ambiguities will inevitably make mistakes. But that doesn’t mean there are no tools available to help detecting ambiguities. For instance, NuGram IDE will tell you if there are two or more different parses for a given sentence. And the sentence generator tool can also be configured to detect sentences that are ambiguous at the semantic level (sentences producing two or more different semantic values).

NuGram IDE new licensing scheme

Last week, we released a new version of NuGram IDE. In addition to supporting UTF-16 and UTF-8 with byte-order mark (BOM), the free Basic Edition also comes with a new licensing scheme.

Was does this mean for you? Well, simply that you will have to request a new license file every 90 days. The installation process is fairly simple:

  1. You install NuGram IDE as before, by adding http://nugram.nuecho.com/update-site to your Eclipse update sites.
  2. You request a new license file from our web site. You will then receive an email containing a link to the license file.
  3. You follow the link and save the downloaded document to a file in the $HOME/nuecho directory (this can be changed in the Eclipse preferences).

It’s as simple as that. And we have an automatic process that will remind you by email to renew your license (steps 2 and 3) just a few days before its expiration.

The rationale

You may wonder why we decided to change the licensing scheme. Downloading and installing the Basic Edition was much more straightforward before. And as a user of free software myself, I don’t like complicated registration processes. I’m usually turned off when I need to enter lots of personal information, I often simply go away.

Then why? NuGram IDE is a relatively specialized tool, but we get downloads at a surprising rate. However, we don’t really know how many people or organizations use it on a regular basic. Do they simply take a look at it and uninstall it? Do they use it only once a year, for a very punctual task? The problem lies in the way the software is obtained (by means of an Eclipse update site). By asking our users to request a new license at a regular interval, we hope to better know our user base. We do think that the free edition of NuGram IDE is a software with real value and it’s not too much asking in return.

Of course, if you don’t like the idea of updating your license every 90 days, you can still buy the Professional Edition and not be annoyed anymore…

NuGram IDE 2.3.0 is out!

The Nu Echo team is pleased to announce the availability of NuGram IDE 2.3.0. This new release features full support for UTF-16 and UTF-8 with byte-order mark (BOM) and fixes a number of problems with non-European languages.

The free Basic Edition is available at the usual location.