Talking to Robots Using MS Speech SDK 5.1 in C# Sebastian van Delden USC Upstate

Microsoft’s Speech SDK 5.1 Libraries developed by MS to facilitate the development of speech recognition and text- to-speech generation applications. For Windows XP, MS Speech SDK 5.1 can be downloaded as a self extracting executable. Download and install it.

Getting Started… For a new C# project, you’ll need to import “SpeechLib” as a Reference and then instantiate the recognition objects, event handlers, etc… Just download the simple starter project that is provided for you on the course website:  SimpleSpeechProject.zip Unzip this file and open the solution in Visual C#

The Sample Program: Top text box will display the literal speech that was recognized Bottom text box displays the semantics/meaning of the speech recognized.

The code that creates the speech recognition and grammar objects. User Created Grammar File. More on this in a minute

Event Handlers that trigger when speech is recognized… More on these in a minute…

Defining a Grammar The Speech Recognition engine will only try to match utterances which have been defined in a grammar.  A “grammar” defines the rules of a language. MS Speech SDK 5.1 uses XML to define these grammars.  MS has there own special grammar specification. The outer XML tag must always be “GRAMMAR” and can set a language ID (for example, 409 means stand English.

Microsoft XML Grammar Style … Grammar rules to be inserted here… …

Defining a New Rule Each rule should have a user-defined name and also say if it should be actively matched on the “TOPLEVEL”  If it is not TOPLEVEL active, that means that it is a sub-rule that is using by another rule

Rule Tags Optional Match:  … List (match only 1 of these)  … Paragraph (all literal items must be in P tags)  … Referencing another rule  To recognize new literal text not defined in the grammar, you can use DICTATION inside of tags. 

Example Grammar/Rule Continued… References a sub- rule that is defined on a later slide

Properties As the grammar rules recognize the actual text being spoken, the semantics/meaning of the text can be captured using tag attributes. PROPNAME=“name”  Defines a property and gives it a name VALSTR=“value”  Defines the value that should be returned for this property

Example Grammar/Rule Continued…

Getting Results… This event handler (shown earlier in these slides) is trigger when the grammar rules recognize something. The recognized text and properties are all stored in the “ISpeechRecoResult” Result object. void RecoContext_Recognition(int StreamNumber, object StreamPosition, SpeechRecognitionType RecognitionType, ISpeechRecoResult Result){ …

ISpeechRecoResult Result Contains an object called PhraseInfo which contains:  A GetText( ) method Returns literal text  Properties attribute Which contains all the property/values pairs recognized. For example, to get the name and value of the first property, you do this:  Result.PhraseInfo.Properties.Item(0).Name  Result.PhraseInfo.Properties.Item(0).Value  Many other methods/attributes you can play with..

Example Output of Sample Speech Project that you can download:

Connecting to the V+ Robot Controller (CS7B) Once you have recognized and interpreted the meaning of a command, you can tell the robot what to do. Download the robot communication socket classes (2) from the course website which can be used to establish a TCP/IP connection with the V+ robot controller. Include the two classes in your speech project.

Establishing a connection Create a new Controller object.  This will also require a program to be ran from the V+ side to establish the connection.  This is also in the zip file on the website “control.v2” Run the C# code first. Then run the control.v2 program on the V+ controller. After the link has been established, the C# code can use its robot controller object to pass strings to the robot which are executed.  The string must be a valid V+ line of code

Example Controller robot; // establish a connection to the robot robot = new Controller(); // Waits until you run the V+ control program // Once connection established, issue V+ commands: robot.execute(“move #a”); robot.execute(“ready”);