1. Build 2015
4/16/2017
© 2015 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

2. Cortana Extensibility and Speech Platform In-Depth
Build 2014
4/16/2017
3-716
Cortana Extensibility and Speech Platform In-Depth
Adina Trufinescu - Principal Program Manager
Khuram Shahid - Senior Developer
© 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

3. Cortana Extensibility and Speech Platform In-Depth
Build 2014
4/16/2017
3-716
Cortana Extensibility and Speech Platform In-Depth
Adina Trufinescu - Principal Program Manager
Khuram Shahid - Senior Developer
© 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

4. Hey Cortana, set a reminder to ….
Build 2014
4/16/2017
Hey Cortana, set a reminder to ….
Hey Cortana, send to Jonas Brandell ….
Hey Cortana, call Jonas Brandell
Hey Cortana, text Jonas Brandell …
© 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

5. Hey Cortana, what’s up with Jonas Brandell on Facebook
Build 2014
4/16/2017
Hey Cortana, what’s up with Jonas Brandell on Facebook
Hey Cortana, create a station for Hozier on Pandora
Hey Cortana, send a message to Jonas Brandell on Viber
Hey Cortana, Tweet flying into San Francisco tonight
© 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

6. Developer Opportunity
Build 2014
4/16/2017
Developer Opportunity
Differentiate your app with voice commands
Delight your users with natural interactions
Increase user engagement
Increase productivity
Increase discovery
© 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

7. Agenda Voice Commands in the Foreground Speech Recognition in your app
Build 2014
4/16/2017
Agenda
Voice Commands in the Foreground
Speech Recognition in your app
Voice Commands in the Background
© 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

8. Voice Commands in the Foreground
Build 2014
4/16/2017
Voice Commands in the Foreground
© 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

9. Voice Commands for Universal Windows Apps
Build 2014
4/16/2017
Voice Commands for Universal Windows Apps
Foreground
Voice Commands
Windows Universal Apps can register voice commands for users to say to Cortana
Cortana launches the app to complete the task
Available for Windows Universal Apps built with XAML and JavaScript
© 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

10. Demo: Voice Commands for Adventure Works

11. Foreground Voice Commands Architecture
Build 2015
4/16/2017 3:47 PM
Foreground Voice Commands Architecture
Adventure Works, show my trip to London X
Application
Call 7
Text 2
Remind
Voice Command Definition
Voice Command 1 3 6
Microsoft Speech
Recognition Service
Windows Speech
Platform 5 4
© 2015 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

12. Foreground Voice Commands - How To
Create the Voice Command Definition (VCD)
Register the VCD XML on App Startup
Handle Voice Command Activation

13. Build 2014
4/16/2017
Voice Command Intent
Determine the task to enable and what should the user say
Start a chat with {contact}
Send a message to {contact}
Determine what Cortana should say as the app is launched
Starting a chat with {contact}
Provide an example of what the user can say for Cortana to show on “What can I do?”
Start a chat with Jonas
© 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

14. Voice Command & App Name
Build 2014
4/16/2017
Voice Command & App Name
Users need to say the app name as part of the voice command
Viber, start a chat with Jonas
Facebook, what’s up with Jonas
Adventure Works, show my trip to London
By default uses the app’s display name
Provide a alternate such as short form when the app name is long or hard to pronounce
© 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

15. Voice Commands Guidance
Build 2014
4/16/2017
Voice Commands Guidance
Explicitly specify the app name where feels most natural
Start a chat with Jonas on/with/using Viber
What’s up with Jonas on Facebook
Show my Adventure Works trip to London
Increase the recognition accuracy by providing a phonetically-rich phrase
Don’t: Get Jonas on Viber
Do: Start a chat with Jonas on Viber
Avoid using a Cortana reserved command, e.g. Call, Text, Start, Open, Find. If you have to, use the app name as a prefix
Viber, text Jonas I’m on my way
© 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

16. Voice Command Registration
Build 2015
4/16/2017 3:47 PM
Voice Command Registration
Register the Voice Command Definition on app startup
Use VoiceCommandDefinitionManager API in Windows.ApplicationModel.VoiceCommands namespace
Update the VCD with application data
© 2015 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

17. Handle the Voice Command Activation
Override App.OnActivated method
Check for ActivationKind.VoiceCommand
Use VoiceCommandActivatedEventArgs.Result to determine what the user said
Navigate to page and continue the interaction

18. Handle the Voice Command Activation
protected override void OnActivated(IActivatedEventArgs e) {
Frame rootFrame = Window.Current.Content as Frame;
if (rootFrame == null) { //create a frame }
if (e.Kind == ActivationKind.VoiceCommand)
{ var commandArgs = e as VoiceCommandActivatedEventArgs;
SpeechRecognitionResult speechRecognitionResult =
commandArgs.Result;
  string voiceCommandName = speechRecognitionResult.RulePath[0];
  if (voiceCommandName == “showTripToDestination")
rootFrame.Navigate(typeof(TripPage), speechRecognitionResult); }
  Window.Current.Activate();

19. Demo: Code walkthrough for “Show my trip to London for AdventureWorks”

20. Voice Commands for Web-Hosted Apps
Voice Commands for Web-Hosted Applications
Websites can be submitted to Windows Store
Cortana can launch Web-Hosted Apps with Voice Commands
Same Voice Command Definition schema and activation model as for JavaScript apps

21. Voice Activation for Web-Hosted Apps
Build 2015
4/16/2017 3:47 PM
Voice Activation for Web-Hosted Apps
if (typeof Windows != 'undefined') {
var activation = Windows.ApplicationModel.Activation;
Windows.UI.WebUI.WebUIApplication.addEventListener("activated", function (args)
if (args.kind === activation.ActivationKind.voiceCommand)
var speechRecognitionResult = args.result;
var properties = speechRecognitionResult.SemanticInterpretation.Properties;
//... }
});
Voice.js X
Application
default.html
<meta name="msapplication-cortanavcd" content="
<script src="voice.js"></script>
Call 1
<Application
Id="App"
StartPage="
<uap:VisualElements
...
</uap:VisualElements>
<uap:ApplicationContentUriRules>
<uap:Rule Type="include" WindowsRuntimeAccess="all“ Match="
</uap:ApplicationContentUriRules>
</Application>
Package.appxmanifest
Text 2
Remind
Contoso.com
/default.html
Package.appxmanifest
Voice Command 7
Vcd.xml 3 6
Microsoft Speech
Recognition Service
Windows Speech
Platform 4 5
© 2015 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

22. Voice Commands for Web-Hosted Apps
Create the Voice Command Definition and host on web service
Register the VCD on the default HTML page
Handle Voice Command Activation in .js

23. Demo: Voice Activation for Web-Hosted App

24. Speech Recognition in your app
Build 2014
4/16/2017
Speech Recognition in your app
© 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

25. In-App Speech Recognition
Build 2014
4/16/2017
In-App Speech Recognition
Check if the interaction was input through voice or text
For voice, use Speech to continue the user interaction in the app
Use Windows.Media.SpeechSynthesis APIs to talk to the user
Use Windows.Media.SpeechRecognition for speech recognition
© 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

26. On-Demand Speech Recognition
Build 2014
4/16/2017
On-Demand Speech Recognition
Use SpeechRecognizer.RecognizeWithUIAsync to present the user with System listening GUI that can be customized
Use SpeechRecognizer.RecognizeAsync for on demand recognition and provide a listening UI experience
System predefined grammars: Dictation, Web Search, Form Filling
App grammars: a list of strings or SRGS file
© 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

27. Continuous Speech Recognition
Build 2014
4/16/2017
Continuous Speech Recognition
SpeechRecognizer.ContinuousRecognitionSessionenables listening continuously
Can be used with Dictation grammar for scenarios where users speak for long periods
Can be used with app-provided grammars for always-listening in-app commands
© 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

28. Demo: Adventure Works walkthrough for Speech Recognition

29. Voice Commands in the Background
Build 2014
4/16/2017
Voice Commands in the Background
© 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

30. Demo: Voice Commands in the Background

31. Voice Commands in the Background
Build 2014
4/16/2017
Voice Commands in the Background
Background Voice Activation
Universal App can register voice commands to run in an AppService
AppService takes action on the voice command
May use Cortana’s canvas and voice to present feedback to the user
Available for Store Apps built with XAML and JavaScript
© 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

32. Background Voice Commands Architecture
Build 2015
4/16/2017 3:47 PM
Background Voice Commands Architecture
Adventure Works, what trips do I have coming up?
Your App
Go to Adventure Works
Here are your upcoming trips A
Yosemite
August 15th
San Francisco
May 29th
VCD
Web Service 1 7
Voice Command
App Service
Render XAML 2 8
Isolated Storage 9 3 6
Microsoft Speech
Recognition Service
Windows Speech
Platform 4 5
© 2015 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

33. Background Voice Commands - How To
Create the Voice Command Definition (VCD)
Register the VCD XML on App Startup
Handle the Voice Command Activation in an App Service

34. Voice Command Definition
<VoiceCommands xmlns="
<CommandSet xml:lang="en-us" Name=“en-us-CommandSet">
<AppName> Adventure Works </AppName>
<Example> Show my trip to San Francisco </Example>
<Command Name=“showTripToDestination">
<Example>Show my trip to San Francisco</Example>
<ListenFor >show [my] trip to {destination}</ListenFor>
<Feedback>Looking for trip to {destination}</Feedback>
<VoiceCommandService Target=“AdventureWorksService"/>
</Command>
Register the Voice Command Definition on app startup

35. Create an AppService Inherit from IBackgroundTask and implement Run()
Register the AppService in the package.appxmanifest
public async void Run(IBackgroundTaskInstance taskInstance)
<Extensions>
<Extension
Category="windows.appService"
EntryPoint="AdventureWorks.AdventureWorksService">
<AppService Name=“AdventureWorksService"/>
</Extension>
</Extensions>
See Session 765 For Details on AppServices

36. WinRT Namespaces and APIs
using Windows.ApplicationModel.AppService;
using Windows.ApplicationModel.Background;
using Windows.ApplicationModel.VoiceCommands;
VoiceCommandDefinitionManager
VoiceCommandDefinition
VoiceCommandServiceConnection
VoiceCommand
VoiceCommandUserMessage
VoiceCommandContentTile
VoiceCommandResponse

37. VoiceCommandServiceConnection
Request actions from Cortana:
RequestConfirmationAsync(…)
RequestDisambiguationAsync(…)
RequestAppLaunchAsync(…)
Send updates to Cortana:
ReportProgressAsync(…)
ReportFailureAsync(…)
ReportSuccessAsync(…)
Handle Events:
VoiceCommandCompleted

38. Cortana Canvas Interactions from Background
Text-To-Speech (TTS)
Which one do you wanna cancel?
App Screen
Present UI on Cortana’s canvas
Speak a message using Cortana’s voice
App Icon A
App GUI Text
Which trip would you like to cancel?
May 14th 2015
Las Vegas
Content Tile
May 28th 2015
San Diego
Content Area
August 24th 2015
Phoenix
Link to your app
Go to Adventure Works

39. VoiceCommandResponse
Which trip would you like to cancel?
May 14th 2015
Las Vegas
May 28th 2015
San Diego
Go to Adventure Works
August 24th 2015
Phoenix
Which one do you wanna cancel? A

40. VoiceCommandResponse
Which trip would you like to cancel?
May 14th 2015
Las Vegas
May 28th 2015
San Diego
Go to Adventure Works
August 24th 2015
Phoenix
Which one do you wanna cancel? A
VoiceCommandResponse {
VoiceCommandUserMessage Message;
VoiceCommandUserMessage RepeatMessage;
List<VoiceCommandContentTile> VoiceCommandContentTiles;
string AppLaunchArgument; }

41. VoiceCommandContentTile{
string Title;
string TextLine1;
string TextLine2;
string TextLine3;
Object AppContext;
string AppLaunchArgument;
Windows.Storage.StorageFile Image;
VoiceCommandContentTileType ContentTileType; } A
Which one do you want to add to your trip?
Title Name
Text Line One
Text Line Two
Text Line Three

42. VoiceCommandContentTile – Text
Icon, Title
Icon, Title, Line1, Line2, Line3 A A
Which one do you want to add to your trip?
Which one do you want to add to your trip?
Title Name One
Title Name Two
Title Name Three
Title Name Four
Title Name One
Long text lorem ipsum dolor sit amet, Consec tetuer amias Sed do eiusmod
Title Name Two
Long text lorem ipsum dolor sit amet, Consec tetuer amias Sed do eiusmod
Go to Adventure Works
Title Name Three
Long text lorem ipsum dolor sit amet, Consec tetuer amias Sed do eiusmod
Title Name Four
Listening…
Listening…
Listening…

43. VoiceCommandContentTile– Icons
68x68 Icon, Title, Line1, Line2
68x92xIcon, Title, Line1,
Line2, Line3
240x140 Icon, Title, Line1, Line2, Line3
Which one do you want to add to your trip?
Listening…
Title Name One
Long text lorem ipsum
Title Name Two
Title Name Three
Title Name Four
Title Name Five
Go to Adventure Works A A A
Which one do you want to add to your trip?
Which one do you want to add to your trip?
Title Name One
Long text two lines lorem ipsum consec tetuer amias
Title Name Two
Long text two lines lorem ipsum consec tetuer amias
Title Name One
One line text
Title Name Three
Long text two lines lorem ipsum consec tetuer amias
Long text two lines lorem ipsum consec tetuer amias odoran
Title Name Four
Long text two lines lorem ipsum consec tetuer amias
Title Name Two
One line text
Listening…
Listening…

44. Demo: Code walkthrough for “When’s my trip to Las Vegas with AdventureWorks”

45. Disambiguation Confirmation Progress Completion A A A A Build 2015
4/16/2017 3:47 PM
Which one do you want to cancel?
May 14th 2015
Vegas Tech Conference
August 18th 2015
Party in Phoenix
Listening…
Disambiguation A
Cancel this trip?
Listening…
May 14th 2015
Vegas Tech Conference
Yes No
Confirmation A
Cancelling this trip…
Cancel
May 14th 2015
Vegas Tech Conference
Go to Adventure Works
Progress A
Cancelled this trip.
May 14th 2015
Vegas Tech Conference
Go to Adventure Works
Completion A
Which ‘Vegas’ trip do you wanna cancel? Vegas Tech Conference or Party in Vegas.
Do you wanna cancel ‘Vegas Tech Conference’?
Cancelling this trip…
I’ve cancelled it. [success-earcon]
Cancel my Adventure Works trip
© 2015 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

46. Which trip do you wanna cancel?
Build 2015
4/16/2017 3:47 PM
Disambiguation
Which trip do you wanna cancel?
The user may select an option via speech or UI input.
Cortana automatically listens for utterances that uniquely identify a content tile.
RequestDisambiguationAsync() returns only after a selection has been made.
Which one do you want to cancel?
May 14th
Vegas Tech Conference
August 18th
Party in Phoenix
Listening… A
Vegas Tech Conference
The first one
The one on August 18th
© 2015 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

47. Do you wanna cancel ‘Vegas Tech Conference’?
Build 2015
4/16/2017 3:47 PM
Confirmation
Do you wanna cancel ‘Vegas Tech Conference’?
Always confirm before taking destructive or irreversible actions
Cancel this trip?
Listening…
May 14th 2015
Vegas Tech Conference
Yes No A
Yes No
© 2015 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

48. Progress Text To Speech is optional
Build 2015
4/16/2017 3:47 PM
Progress
Cancelling this trip…
Text To Speech is optional
You may send multiple progress updates. Updates are coalesced where possible.
Send progress during long-running operations. If there are no updates for 5 seconds, Cortana may cancel the flow. A
Cancelling this trip…
Cancel
May 15th 2015
Vegas Tech Conference
Go to Adventure Works
© 2015 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

49. Looking for your trip to San Francisco…
Build 2015
4/16/2017 3:47 PM
Cortana Supplied Progress Screens
Looking for your trip to San Francisco…
Shown if launching the App service, or the initial response takes > 500ms
Feedback string from VCD, if any, is spoken out loud.
Shown after RequestConfirmation or RequestDisambiguation if the App does not respond quickly (500ms)
Content derived from the previous Confirmation/Disambiguation Response
I’ll get Adventure Works on that.
Adventure Works
Looking for trip to San Francisco
Cancel
Search “Cancel my Adventure Works trip to Vegas” A
Working on that…
Cancel
Vegas Tech Conference A
May 14th 2015
Go to Adventure Works
Listening…
© 2015 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

50. Build 2014
4/16/2017
Interaction Summary:
Canvas Interactions from Background
Ask the user to confirm before taking action on the voice command
Ask the user to disambiguate if there are more than one action or target of the action
Show progress as the voice command is handled
© 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

51. Interaction Flow Your AppService Cortana RequestDisambiguationAsync
Build 2015
4/16/2017 3:47 PM
Interaction Flow
Cortana
Your AppService
Adventure Works, cancel my trip!
I’ll get Adventure Works on that!
Invoke IBackgroundTask::Run()
Inspect Voice Command
RequestDisambiguationAsync
Which one?
Las Vegas
Cancun
//build/
Las Vegas
VoiceCommandDisambiguationResult
check SelectedItem
RequestConfirmationAsync
Are you sure?
yes no
yes
VoiceCommandConfirmationResult
Process request
(Talk to web service,
save data
etc.)
© 2015 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

52. Your App Cortana Event: VoiceCommandCompleted
Build 2015
4/16/2017 3:47 PM
Cortana
Your App
Adventure Works, cancel my trip!
I’ll get Adventure Works on that!
Inspect Voice Command
Which one?
Las Vegas
Cancun
//build/
Are you sure?
yes no
yes
check SelectedItem
Invoke IBackgroundTask::Run()
RequestDisambiguationAsync
RequestConfirmationAsync
VoiceCommandConfirmationResult
VoiceCommandDisambiguationResult
Process request
(Talk to web service,
save data
etc.)
Canceling…
"Done!
Still canceling…
ReportSuccessAsync
ReportProgressAsync
Event: VoiceCommandCompleted
Event: VoiceCommandCompleted
Teardown
foreground
App Launch
UI Interaction
© 2015 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

53. Demo: Authoring & Debugging an App that supports background Voice Commands

54. Conversational Writing Guidance
Build 2014
4/16/2017
Conversational Writing Guidance
Canvas Interactions from Background
Be efficient. Less is more!
Be relevant. Provide information that’s relevant to the given task, content, and context.
Be clear. Avoid ambiguity. Use everyday language.
Be trustworthy. Present accurate information. Be transparent. Respect privacy.
© 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

55. Build 2014
4/16/2017
Localization
Canvas Interactions from Background
For each supported language, localize the CommandSet in the Voice Command Definition
When launched, inspect the value of Language property of the VoiceCommandServiceConnection
Localize the dialog on the app provided screens to match Cortana’s language
© 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

56. Voice Commands for Universal Apps - Recap
Create a Voice Command Definition
Register the VCD when the app starts
Handle the Voice Activation in the app or AppService

57. Voice Command Handling Guidance
Build 2014
4/16/2017
Voice Command Handling Guidance
Foreground
Background
for tasks that require complex interactions
for tasks that demand the user’s attention for a long time
for simple tasks that don’t require additional input beyond Confirmation and Disambiguation
for tasks that complete within seconds
to initiate longer tasks that execute remotely
may use RequestAppLaunchAsync method to launch the app and complete remainder of the flow
© 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

58. Call to Action References:
Launch and deep link into your app from Cortana
Extend Cortana with your app’s capabilities through voice commands in the background
Add speech recognition as an input modality to your app
References:
See AppServices Session 765
See Web-Hosted Apps Session 665
See Adventure Works Sample Code
See Design Guidelines for extending Cortana