Category Archives: Speech Technologies

Development Speech Technologies

CouchDB for call analysis data – a case study

May 18, 2011 – 7:38 am

At Nu Echo, we’ve been developing and refining our own VoiceXML application framework for years now. As part of our nth rewrite (and I’ll talk more about that rewrite and why we did it in another post), we decided to experiment with CouchDB. (For those new to CouchDB, it’s a schema-less document-oriented database. A so-called NoSQL database.)

The first area where we saw a fit for CouchDB was the storage of call analysis data. This data consists of various attributes associated with a call, information about each interaction (like recognition results) and each transaction (groups of interactions). It can also be augmented with the recordings saved by the ASR engine. Call analysis data is used by our call viewer tool to listen to calls, search for calls exhibiting some specific caller behaviors, produce reports, etc.

In the previous incarnation of our framework, call analysis data was stored on disk in a plain text file, and optionally in a SQL database. Due to the richness of our model, the SQL schema consisted of about 15 tables. And the representation of the same data in the text file was quite complex (tab-separated values, with some fields encoded in JSON format). At the end of each call, data collected during the call was stored on disk and optionally stored in the SQL database. We also had a script that could read all the files on disk and push the data in the SQL database at a later time.

Adding support for CouchDB

The very first step toward our support of CouchDB consisted in rewriting the serialization code to produce JSON-encoded call analysis data instead of our complicated text format.  Now, data for  a call is written as a single JSON object, one per line, prefixed by the call Id. This greatly simplified the code to read data back into memory.

The next step was to write a script to push the data to CouchDB. The script simply reads the call data, one call per line, and PUTs them to CouchDB in batches of 100 calls using the bulk API in order to increase performance.

Finally, we had to rewrite the part of our call viewer tool connecting to a database to retrieve calls data matching some patterns. It relies on some simple CouchDB views, but not that much in order to be as independent as possible of the database layer (it is possible to retrieve calls from text files as well from the call viewer).

Benefits

We obtained several benefits by moving to CouchDB:

  1. Performance – Loading call analysis data in the CouchDB database is way faster than putting the same data in a MySQL database. Our preliminary results show a speed up factor of about 100 (this does not take the loading of audio recordings into account, though). Ok, we are comparing apples and oranges. CouchDB does not update the view indexes until they are requested, while MySQL updates its indexes as rows are inserted. And only a single document is inserted in CouchDB, compared to lots of rows in more than 15 tables in SQL. On the other hand, if insertions are done at application runtime (after the completion of the call), you better do it fast, especially if the IVR handles many hundred (if not thousand) ports.
  2. Evolution – Making modifications to a complex schema is painful, especially when you have applications deployed in the field. As documents do not have to follow a rigid schema, it is much easier to adapt our code to multiple versions.
  3. Attachments - Even if audio recordings can be stored in a traditional SQL database as blobs, a custom application is still required to access them. With CouchDB, recordings are stored as attachments to the JSON document for the corresponding call. Moreover, these recordings are easily accessible by other tools since CouchDB is itself a webserver and all documents and attachments have a URL.

Conclusion

Of course, there is no panacea and CouchDB is no exception. There are still some aspects of our system for which CouchDB does not provide a better solution than an SQL database. One of them is the support for custom queries. In the call viewer tool, it was possible to write custom SQL queries to find calls matching very specific criteria. Of course, CouchDB supports temporary views to do something equivalent. The main problem is the time taken to build the view. When hundreds of thousands or even millions of calls are processed, creating a temporary view can take a long time (several minutes).  Not so good for an interactive tool.

But overall, we have been very pleased by the performance of CouchDB and the flexibility it gives us.

Tags , | Comments (5)
Development Speech Technologies Training tuning

Loquendo and Nu Echo announce partnership to expand PS offering

March 21, 2011 – 7:17 am

Loquendo and Nu Echo are proud to announce today the launch of a new partnership to provide a complete offering for the North American market, from speech technologies to professional services. This new partnership enables the companies to offer North American customers speech application development, dialog design, porting and tuning services to complement Loquendo’s existing portfolio of highest quality multilingual technologies, for a simpler yet more effective and enhanced integration of speech.

Read the full press release.

Tags , , | Comment (0)
NuGram Speech Technologies application tuning

What’s the point of having 98% in-grammar accuracy if 40% of user utterances are out-of-grammar?

February 15, 2011 – 4:06 pm

How many times have you heard people say that they “achieve 95% speech recognition accuracy” (or more)? That sounds really impressive, doesn’t it?

It shouldn’t. What they don’t tell you is that they actually measure “in-grammar accuracy”, which means that accuracy is measured only on utterances that are perfectly covered by the grammar. For instance, for a date grammar, an utterance such as “well, uh, january fourth” would be considered out-of-grammar (and therefore ignored from the accuracy calculation) if “well, uh” is not covered by the grammar.

Unfortunately, in the real world there’s no way to force users to stick to in-grammar utterances. In fact, users usually have no way of even knowing what the grammar covers other than through hints provided by the prompts. Even well-behaved users can hesitate, correct themselves, or use an unexpected formulation (which sounds perfectly natural to them), all of which are likely to be out-of-grammar. They can even say things that they believe will help the machine understand them (for instance using “victor” instead of “v” when spelling).

As a result, it’s not unusual to have between 30% and 50% of user utterances that are considered out-of-grammar, many of which are perfectly legitimate responses to the application prompt. So what’s the point of reporting in-grammar accuracy if this ignores a large chunk of legitimate user utterances? You tell me.

Just to illustrate, you want to know one of the most effective ways of improving in-grammar accuracy? Just reduce grammar coverage. Sure, your out-of-grammar rate will increase but, hey, you’ll improve in-grammar accuracy! Isn’t that great? This tells you how useless in-grammar accuracy is at telling you whether you improved the grammar.

This is why we always report accuracy by considering every legitimate user utterance (i.e., the ones that contains a valid response to the prompt, regardless of wording or extraneous speech). This way, we make sure that we don’t conveniently ignore the utterances that happen to be the more challenging and we get results that accurately represent the real recognition performance (not some imaginary performance calculated on an idealized set of clean utterances).

But the best reason for doing it our way is that it enables us to truly measure improvements when we tune grammars. The reason is simple. Changing the coverage of a grammar always involves a trade-off. We can improve accuracy by covering more user utterances, but this can reduce overall accuracy if the new grammar paths introduce new speech recognition errors. The only way we can measure improvement is if we measure accuracy on a fixed set of valid utterances that doesn’t depend on the actual grammar coverage.

Tags , , , , | Comments (2)
Development NuGram Speech Technologies

Testing dynamic grammars

February 3, 2011 – 1:29 pm

In my post on NuGram and CouchDB, I neglected to mention how the dynamic grammar was authored and, most importantly, tested. Having a repeatable process for testing grammars is very important when developing a speech application, as most grammars change and get more complex over time.

Of course, the grammar was authored with NuGram IDE. NuGram IDE has some great features to test grammars, and especially dynamic grammars. Dynamic grammars (like the streets grammar) have always been more difficult to debug than static grammars. They can be very easy to write for small applications or prototypes (or blog posts…), but in real applications their coverage tests are often (and should!) run in batch as part of an automated build process. But this is often too cumbersome in practice. For instance, a dynamic grammar implemented as a JSP page requires a web application server to run and if the JSP page makes queries to a database, the DB must be running somewhere too. This greatly complicates the setup to make batch coverage tests. Moreover, writing and testing the dynamic grammar requires some programming skills that speech scientists don’t always have (at least not in large organizations).

With NuGram’s template language, a dynamic grammar can be tested in NuGram IDE Basic Edition in two different ways:

  • Using predefined data encoded as a JSON object (a JSON context), or
  • Using some custom Java code (a Java context).

Both ways require the creation of an instantiation context. It’s simply a mapping between variable names and values. An instantiation context must provide a value for each and every variable used in the grammar template. The values are used to populate the template and produce the resulting (ABNF or XML) grammar. The way the instantiation context is created depends on the type of context. For a JSON context, the instantiation context is the JSON document itself. For the Java context, some Java code populates a map from strings to objects.

The following video shows how to create a JSON context for the street grammar:

This one shows the steps required to create and use a Java context:

Note: there was a subtle (uncovered) bug in the previous version of NuGram IDE. If you want to create Java contexts like in the video above, please make sure to download the latest version.

The whole project used in the videos is available on github. The Java context initializers use the following open-source libraries:

In the next post, I will show how to use the Java context initializer to deploy the streets grammar on the Java-based version of NuGram Server.

And you, how do you test your dynamic grammars?

Tags , , , , , | Comment (1)
Development NuGram Speech Technologies

Don’t let NuGram choose session IDs for you

January 11, 2011 – 8:51 am
NuGram Session Viewer

NuGram Session Viewer

Just after hitting the “Publish” button last week for my previous post on NuGram and CouchDB, I realized the code I wrote had a serious flaw.

Remember it? Here’s an excerpt:

#...
post '/ask_street.json' do
  session = GrammarServer.new.create_session "username", "password"

  tropo_event = Tropo::Generator.parse request.env["rack.input"].read
  zipcode = tropo_event.result.actions.zip_code.value

  grammar = session.instantiate "streets.abnf",
                                :zipcode => database.get(zipcode)

  tropo = Tropo::Generator.new do
    on :event => 'continue', :next => '/say_street.json'
    on :event => 'hangup', :next => '/hangup.json'
    ask({ :name => 'street',
          :bargein => 'true' }) do
      say :value => "What is your street address, beginning with your street number?"
      choices :value => grammar.get_url("grxml")
    end
  end
  tropo.response
end
#...

There are basically two problems with this code:

  1. On line 3, the application creates a new session on NuGram Hosted Server and the unique session ID is returned to the application. But there is no code to free the session. We cannot free the session on line 20 because the speech recognition engine will have to fetch a resource (the speech grammar) associated with the session.
  2. The other handlers of the application cannot reuse the same session, making it impossible to reuse resources (like grammars) from one request to the other.

Problem #1 is not a very serious one. The session will be automatically reclaimed by the server after 15 minutes of inactivity on the session.

But problem #2 is more serious, as it can lead to wasted resources on server side. If an application wants to ask the same question from two different handlers, using the same dynamic grammar, it can’t using the approach above. Two or more sessions will have to be created, and the grammar will have to be instantiated at least twice.

Providing a session ID explicitly

In fact, NuGram’s RESTful API was not the culprit. The Ruby API was simply missing a little something (I have since fixed it on github and the Ruby gem will soon be updated to reflect the change): a way to specify the session ID explicitly instead of letting NuGram forge a new one. This way, each handler can use a shared session ID when accessing NuGram.

(The Sinatra/Rack web framework provides a session management API to let the application store data that survives a single request. This would solve our problem too. But, as will be explained below, the explicit session ID is a superior solution.)

The updated Ruby API can now specify a session ID explicitly:

  session = GrammarServer.new.session "username", "password", sessionid

The question is now: what do we use as the session ID?

Every communication platform I know (VoiceXML platforms, cloud telephony platforms, Asterisk, etc.) associates a unique identifier with each call. This is usually referred to as the call ID or the session ID. This is certainly one of the best IDs for your NuGram session. Why? Because this greatly simplifies debugging your application. If a problem occurs, it becomes very easy to correlate the communication platform’s logs and NuGram’s logs. But it can also be the application’s session ID (the jsessionid in the case of a Java web application). The idea is to use an ID that uniquely identifies your session across all the servers: communication platform, application server, grammar server, etc.

For instance, Tropo provides a session ID with each response. Here is how to use it in our previous example:

#...
post '/ask_street.json' do
  tropo_event = Tropo::Generator.parse request.env["rack.input"].read

  zipcode = tropo_event.result.actions.zip_code.value

  sessionId = tropo_event.result.sessionId
  session = GrammarServer.new.session "username", "password", sessionId
  grammar = session.instantiate "streets.abnf",
                                :zipcode => database.get(zipcode)
  #...

Tags , | Comment (1)

« Previous Page — Next Page »