Nu Echo is on the lookout for dynamic and talented people that are passionate about their work and are motivated by our obsession for delivering products of uncompromising quality and performance and by the excellence of our professional services, for which we are now widely recognized in the industry. We are currently looking to fill three full-time positions:
Interested? Send us your resume at hr@nuecho.com.
Lately, I have been involved in a number of grammar conversion projects. This has been a great opportunity to put our process and tools to the test once again. And since every project has its peculiarities, we learn constantly.
The process we outlined about a year ago omitted a number of small details. That was OK for small scale conversion projects. But when you have to deal with much larger projects (with thousands of grammars to convert), these details add up significantly. Let me share some of the issues we face daily.
It’s not just semantic tags
When you have tools to automatically convert semantics tags from one format to another, grammar conversion can seem to be a no-brainer. But reality is not that simple. Grammars are not written for an abstract specification, they are written for a very specific recognition engine. They often contain:
All this means that there are a number of transformations either to the original grammar or to the converted grammars that must be applied. This can be by means of regular expression search&replace, or manually inspecting grammars.
Generation of coverage sets
When dealing with hundreds (if not thousands) of grammars, it is not feasible to create initial coverage test sets manually. This is way too time consuming. That means you have to find a way to generate those initial coverage test sets automatically in batch. But how do you do that?
Fortunately, NuGram IDE already provides sophisticated tools to analyze grammars and generate sentences from them. We just built on this foundation a tool to automatically generate coverage tests sets for a set of ABNF grammars. The tool also reports problems found in the grammars, like the use of digits in voice grammars, or words in DTMF grammars.
The coverage set generation tool uses a combination of configuration and sophisticated analyses to determine how to generate sentences and how many sentences to generate. For example, it’s not possible to generate all sentences from a grammar that covers an infinite number of sentences. When that’s the case (or when the number of sentences covered by the grammar is above a certain threshold), the tool reverts to other generation strategies.
Recognition tests as part of the QA process
Finally, even a syntactically valid grammar may fail to load in the ASR for a variety of reasons, the most common one being a limitation or constraint from the ASR itself. For this reason, we got to the conclusion that doing recognition tests (ideally benchmarking of the converted grammars) is a very useful addition to the QA process. Of course, simply compiling grammars may catch a number of problems. But doing a “before and after” comparison can detect conversion problems that were not caught by the coverage tests when they are not exhaustive.
Another benefit of doing recognition tests is the ability to check the performance of the converted grammars to identify those needing additional work. Some converted grammars may have words that prove difficult to recognize with the new engine because they are not properly phonetized, thus calling for application-specific (or even grammar-specific) phonetic dictionaries.
What about DTMF?
In the specific case of converting GSL grammars to GrXML or ABNF, a complication arises with the presence, in the same grammar, of both DTMF sequences and words. I will discuss this issue in a separate post.
While working on a grammar conversion project from Nuance GSL to SRGS ABNF, I stumbled upon a few grammars all having the same design problem: using optional parts to make a few words repeat a varying number of times. This is a pattern we’ve observed regularly on various projects.
Here is an example of such a grammar for recognizing sequences of 4 to 8 digits (I omitted the semantic tags for clarity):
#ABNF 1.0 ISO-8859-1; mode voice; language en-US; root $digits4To8; public $digits4To8 = $digit $digit $digit $digit [$digit] [$digit] [$digit] [$digit] ; ...
The original GSL grammar looked like this:
Digit4To8 ( Digit Digit Digit Digit ?Digit ?Digit ?Digit ?Digit )
The GSL syntax does not support the <N-M> syntax like in ABNF to repeat an expansion from N to M times. That’s a reason why the grammar was written this way in the first place. In ABNF grammar , it would have instead been written as:
#ABNF 1.0 ISO-8859-1; mode voice; language en-US; root $digits4To8; public $digits4To8 = $digit <4-8> ; ...
In GSL, it would have been better to write the grammar as:
Digit4To8 ( Digit Digit Digit Digit ?Digit1To4 ) Digit1To4 ( Digit ?Digit1To3 ) Digit1To3 ( Digit ?Digit1To2 ) Digit1To2 ( Digit ?Digit )
Both grammars are equivalent, right? So what’s the problem?
Ambiguities
Well, both grammars recognize the same language (the same set of sentences), but the first grammar has a very different behavior. It is highly ambiguous. That means some sentences can be parsed in two or more different ways. See what you get when you interpret one such sentence in NuGram IDE:
The interpreter tells us (at the top-left of the window) that there are 6 different parses for the sentence. (I’ve seen grammars generating more than 100 parses for a given sentence!).
The problem with ambiguous grammars is they can impact both recognition accurary and recognition performance. Suppose a grammar covers a sentence that is highly ambiguous and another sentence which is not, but is phonetically close to the former. Since speech recognition engines limit their recognition search space, it is possible that the latter be pruned from the search space at the beginning of the recognition window even if it’s the one that would come up with the best score at the end of the recognition.
The other problem is recognition performance. All semantic tags are typically executed at the end of the recognition process, once the user has finished talking. If there are lots of identical hypotheses with the same score, the recognition engine will have to execute all tags (interpreted ECMAScript code), most of them being redundant and useless, thus causing longer delays in the speech application.
Determining that a grammar is ambiguous (or not) is a very hard problem (it’s an undecidable problem). That means, whatever tool you use that’s supposed to decide for ambiguities will inevitably make mistakes. But that doesn’t mean there are no tools available to help detecting ambiguities. For instance, NuGram IDE will tell you if there are two or more different parses for a given sentence. And the sentence generator tool can also be configured to detect sentences that are ambiguous at the semantic level (sentences producing two or more different semantic values).
(This post has nothing to do with speech technologies or IVR applications. It’s merely a discussion on an implementation detail I described at the Erlang Montreal meetup and it’s rather technical.)
In my previous post about my talk at the Erlang Montreal meetup, slide 15 briefly outlines how session timeouts are implemented in NuGram Hosted Server. The code is duplicated here:
receive
…
after Timeout ->
db:expire_session(self())
end
This code uses the Erlang receive..after construct to handle timeouts. The construct tries to extract a message from the process mailbox, and waits at most Timeout milliseconds if there are no matching messages (variables start with an uppercase letter in Erlang).
This is great when sessions are represented using plain Erlang processes (I described this technique here). But there is a much better way to achieve the same effect when implementing servers using OTP’s gen_server behaviour. (One of our hard learned lessons is to take time to properly learn OTP, Erlang’s Open Telecommunications Platform, before building a production-grade system. It’s definitely worth the investment. It’s what puts Erlang in a totally different category than most programming languages and systems.)
When implementing a server using gen_server, one has to implement a few callback functions (namely handle_call for synchronous calls, handle_cast for asynchronous ones, and handle_info for other messages). In order to specify request timeouts, values returned by those three functions must provide the optional timeout:
handle_call(Request, From, State) ->
Reply = ...
{ok, Reply, NewState, Timeout};
...
If the server does not receive any message during the next Timeout milliseconds, the timeout message is sent to the process and must be handled by the handle_info function. To stop the process, something like the following can be done:
handle_info(timeout, State) ->
%% Do some clean up
{stop, normal, State};
...
This simply tells the server is to be shut down normally and that its last state is State (a great thing to know when things go wrong).
Last week at the first Erlang Montreal meetup, I gave a talk on what we’ve learned at Nu Echo developing the NuGram Hosted Server in Erlang. I just put the slides from the presentation on SlideShare. Here they are:
Great speech applications don't happen by accident!