Sports Scores Speech Recognition System Major League Baseball Score System

Development Team Members Dan Corkum (Director) Jason NguyenTrieu Dan Ragland (Producer) Quang Vu Andrew Wagner Sponsor: Jim Larson, Intel Corporation

Goals & Objectives Develop a compelling Speech Recognition Application for Retrieval of Sports Information. Incorporate Ease of Use Techniques including: Tapered Prompts, Global Commands, Barge-In, Repair Dialogs, and others. Develop an Architecture that is both Robust and Modular. Design for Reuse.

Core Modules “Web Viking” – Parse Internet Web Pages to retrieve sports information. Data Warehousing & Querying – Database for storage of searchable information. Client and Server Communication – Enables communication between Server and remote Clients. VUI (Voice User Interface) Voice Prompts and Response System – The core engine that controls the entire VUI. Dialog Database – Contains the content for the text- to-speech prompts and response criteria.

Web Viking The purpose of the Web Viking is to retrieve data from web sites, parse and format it into a format so that the database interface can understand it. There are three data collection scripts: Schedule, Scores, and Standing/Ranking The data comes from 2 sources: – Major League Baseball – ESPN Two chances to get the right data: – First, we get data from MLB web site and parse it. If it fails for any reason, we'll try to get data from the ESPN web site.

Web Viking How is the data retrieved? – We used the library functions available in the CPAN (Comprehensive Perl Archive Network.) – The module: package up the URL request – The module: handle the data coming back. How the data is parsed: 1. Match and strip off unnecessary data. 2. Regular expression 3. Split 4. Format data and check result.

Data Warehousing & Querying The Database was implemented using MS Access. It functions as a storage site keeping track of team names, scores associated with each team, league/division ranking information, and the schedules for each game. The Database Handler was written in Java. Its primary purpose is to query the database and fetch the results to the sport score server.

Client & Server Communication danC stuff

Client & Server Communication danC #2

VUI (Voice User Interface) Voice Prompts and Response System User Interface and Underlying Logic

Design Considerations Two Options For Design: 1. Dialog logic coded directly into code. 2. Dialog logic entered into a data structure and presented by separate internal logic. VUI

Advantages & Disadvantages of Hard-Coded Dialogs Fast initial implementation Ultimate flexibility of features Duplicated code Difficult to provide consistent global functionality Hard-coded grammars VUI

Advantages & Disadvantages of Dialog Database Good design: Data separated from presentation Consolidation of code Easy to create and maintain dialogs Features aided by use of recursion Computer-generated grammars Much work required before any results seen Difficult to customize specific components VUI

Decision: Dialog Database Sports Score dialogs all follow the same basic pattern Implementation could be modularized by separating the dialogs from their presentation logic The gains made by the ease of entry and flexibility for the end-user outweighed the losses in implementation time Some features require recursion VUI

VUI Infrastructure Design Features – Tapered, User-Level Sensitive Prompts – Tapered, User-Level Sensitive Help – Barge-In capability – User shortcut capability (users can answer future prompts from any prompt) – Navigational user commands (“back”,”quit”,etc) – Enumerated user commands to allow the user to say a number as an alternative to the command VUI

Dialog Components Prompt – Point of user interaction – Has associated Prompt Text levels, including text to be read, the user level for which it is to be read, and the number of visits before the next user-level is used – Has associated Commands, or phrases the user is allowed to say and the actions to take – Has a parameter name to be used in a query VUI

Dialog Components (cont) Commands – The text the user says to access the command – The text that will be returned when this command is accessed (used in a query) – A flag to indicate whether or not the command is to be enumerated – The action the system is to take when the command is accessed VUI

Dialog Components (cont) Scripts – Series of prompts to be called in succession Script Steps – Individual prompts belonging to a script – Each contains its own grammar (reflecting shortcuts available later in the script) – Each contains a flag indicating whether or not a query will be performed following the step VUI

How A Prompt Works Shortcuts take place when a user answers multiple prompts in a row, so the first thing the prompt does is checks for overflow from the last prompt. If there is overflow, jump ahead to the command processing. Otherwise, cycle through the following: – Find the appropriate prompt text to be read to the user based on user level and number of visits. VUI

How A Prompt Works (cont) – If the user requires help, find the appropriate help text to be read – Begin the reading of the help and prompt text to the user – At the same time, begin listening for a user response (if the user responds while it is reading, interrupt the reading) – When the computer finishes reading, begin timing. After five seconds with no user speaking, time-out. VUI

How A Prompt Works (cont) – Attempt to match what the user said to a command that is available at this prompt. Match using the longest available command, so if the user said “New York Yankees”, match “New York Yankees,” not “New York” Any portion of what the user said that was not matched (if anything was matched at all) gets sent to proceeding prompts for processing. Example: The user said “score New York” in the first prompt. If the prompt matches “score”, “New York” will get passed to any following prompts. VUI

How A Prompt Works (cont) When a command is matched, the command’s return value is attached to the parameter name of the prompt The action that is then performed is dictated by the command. Some possibilities are: –Calling another script/prompt and returning all values –Calling another script/prompt and returning only those values –Repeating the prompt and reading help to the user –Changing the user level –Running a query and repeating or calling another prompt/script VUI

How A Script Works A script is presented simply by presenting the first script step in the script A script step presents its associated prompt, using its own grammar (reflecting the ability of the user to shortcut to the next script step) After the script step is executed, a query may be performed and the next script step (if any) may be performed VUI

Other Dialog Routines Components are also involved in routines to build grammars acceptable to Microsoft’s SAPI interface – The dialog structure is descended recursively, with all dependent grammars being included in each prompt’s grammar – Global commands are also created and added to grammars VUI

Queries All query parameters are accumulated in an XML document When a query occurs, the document is sent to the server The server returns an XML document containing results The results are read to the user based on administrator-defined result strings VUI

Why XML? XML is fast becoming the industry standard for data transfer over the Internet XML’s hierarchical structure lends itself to this application Several XML parsers already exist for various platforms (we used IBM’s XML4J) The HTML-like nature of XML makes results easy to read, even for a human.

How Query Results Are Read The administrator defines parameter-value pairs as criteria for which response is read Each response consists of segments of literal text along with parameter values (which can be drawn either from the client or server)

Query Results Example Criteria – Function = “Score” – Team = “Yankees” To be read – “The Yankees score was” – – “To” –

The Results The front-end is very customizable Dialogs can be built simply and quickly The system administrator needs no knowledge of programming concepts The overall behavior of the system could be changed without changing each prompt The computer speech engine is accessed in only one area of code, so it could be swapped with minimal effort VUI

Dialog Structure The Dialog System consists of: – Prompts – Responses – Help System All Dialogs are tapered (Prompts, Responses, & Help) Repair Dialogs – Example: Two teams from same city (New York  Mets and Yankees)

Dialog Structure Overview

Summary We not only developed a powerful Speech Recognition Application for Retrieval of Sports Information, we also developed a reusable framework which can be easily modified for use in other applications. We incorporated Ease of Use Techniques including: Tapered Prompts, Global Commands, Barge-In, Repair Dialogs, and others.