1. Extending Office 365 Using Cortana, Siri and Google Now Analytics
Paul J. Swider
CTO and Founder Real Activity
Puerto Rico

2. Agenda Xamarin Digital Experience REST API and Universal Apps
Build 2014
11/27/2017
Agenda
Digital Experience
REST API and Universal Apps
Speech Platform
Cortana and Apps
Deep Linking
Completing tasks
Finding information quickly
Xamarin
Android Studio
IOS
Q&A
© 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

3. INFORMATION MOVES SLOWLY COMMAND AND CONTROL
TRADITIONAL HIERARCHIES
RESPONSIVE NETWORKS
INFORMATION MOVES SLOWLY COMMAND AND CONTROL
INFORMATION TRAVELS FAST LEARN AND ADAPT

4. Speech and developer vision
Digital Experience
DATA
HTML

5. Platform Overview (Office 365 and Azure)
Apps
1st & 3rd party apps
Android App
iOS App
Windows App
Cordova MDD App
Xamarin App
..Net Web Site
Java Web Site
Java Script Site
Office Extensions
Controls
HTML-Based and Native Controls
HTML Controls for key high value data( + Native controls for differentiation): People Picker; File Picker; Item Previews…
SDKs
Platform-specific SDKs, partially generated from service metadata and open-sourced on GitHub
Android SDK
iOS SDK
Windows SDK
Cordova SDK
Xamarin SDK
.Net Server SDK
Java Server SDK
Office.js
Protocol
Single domain, namespace, and protocol
REST
Auth
3rd-party app registration and runtime authentication through OAuth 2.0
Azure Active Directory
Workload
Individual service workloads
Files
Mail
Calendar
People
Groups
Spread Sheets
Documents
Presentations
Tasks
Social Comms
Dynamics CRM …
Salesforce.com 16

6. Universal Windows Platform
Build 2014
11/27/2017
Universal Windows Platform
Desktop
Mobile
Xbox
IoT
Holographic
Surface Hub
Universal Windows Platform
Core APIs
Unified Developer Platform
Single Application Package
Adaptive Code/UX
Scales to Phone, Tablet, PC, XBOX and More
© 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

7. Adaptive UX vs Digital Experience
Based on a set of adaptive controls and enable an experience tailored to the device
Base building blocks:
Ensure large and small screen appropriateness
Deliver polished and attractive user experience
Support mouse, keyboard, touch, etc.

8. Adaptive Experience (UX and Voice)
11/27/2017
Adaptive Experience (UX and Voice)
Same code, Same controls, Optimized lyout
Tablet (landscape) / Desktop
Phone (portrait)
© 2012 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.
The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

9. REST and Office 365 APIs Simple REST/OData Service APIs
Build 2015
11/27/2017 3:33 PM
REST and Office 365 APIs
Simple REST/OData Service APIs
Part of Azure AD Common Consent Framework
Expose Files, Sites, Mail, Calendar, Users, Groups, Conversations and Notes
Videos, Yammer and OneNote in preview
EVENTS
MANAGEMENT
Mail
FILES
© 2015 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

10. Azure AD and Office 365 APIs
Single authentication flow
Azure AD Graph, Exchange, SharePoint
Mobile apps and web sites
Admin and end-user consent
Industry Standard Protocol
OAuth 2.0
No capturing user credentials
Fine-grained access scopes
Supports MFA and federated user sign-in
Long-term access through refresh tokens
630: Get Your Hands Dirt with the Office 365 APIs, Authentication and SDKs

11. https://graph.microsoft.com/
Office 365 Graph
Your App

12. Extended Profile SharePoint User Profile Exchange HD Picture
PEOPLE
GROUPS
SharePoint User Profile
Exchange HD Picture

13. Cortana and Speech

14. Developer Opportunity
Build 2014
11/27/2017
Developer Opportunity
Differentiate your app with voice commands
Delight your users with natural interactions
Increase user engagement
Increase productivity
Increase discovery
© 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

15. Voice Commands for Universal Windows Apps
Build 2014
11/27/2017
Voice Commands for Universal Windows Apps
Foreground
Voice Commands
Windows Universal Apps can register voice commands for users to say to Cortana
Cortana launches the app to complete the task
Available for Windows Universal Apps built with XAML and JavaScript
© 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

16. Foreground Voice Commands Architecture
Build 2015
11/27/2017 3:33 PM
Foreground Voice Commands Architecture
Adventure Works, show my trip to London X
Application
Call 7
Text 2
Remind
Voice Command Definition
Voice Command 1 3 6
Microsoft Speech
Recognition Service
Windows Speech
Platform 5 4
© 2015 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

17. Foreground Voice Commands - How To
Create the Voice Command Definition (VCD)
Register the VCD XML on App Startup
Handle Voice Command Activation

18. Build 2014
11/27/2017
Voice Command Intent
Determine the task to enable and what should the user say
Start a chat with {contact}
Send a message to {contact}
Determine what Cortana should say as the app is launched
Starting a chat with {contact}
Provide an example of what the user can say for Cortana to show on “What can I do?”
Start a chat with Jonas
© 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

19. Voice Command & App Name
Build 2014
11/27/2017
Voice Command & App Name
Users need to say the app name as part of the voice command
Viber, start a chat with Jonas
Facebook, what’s up with Jonas
Adventure Works, show my trip to London
By default uses the app’s display name
Provide a alternate such as short form when the app name is long or hard to pronounce
© 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

20. Voice Commands Guidance
Build 2014
11/27/2017
Voice Commands Guidance
Explicitly specify the app name where feels most natural
Start a chat with Jonas on/with/using Viber
What’s up with Jonas on Facebook
Show my Adventure Works trip to London
Increase the recognition accuracy by providing a phonetically-rich phrase
Don’t: Get Jonas on Viber
Do: Start a chat with Jonas on Viber
Avoid using a Cortana reserved command, e.g. Call, Text, Start, Open, Find. If you have to, use the app name as a prefix
Viber, text Jonas I’m on my way
© 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

21. Voice Command Registration
Build 2015
11/27/2017 3:33 PM
Voice Command Registration
Register the Voice Command Definition on app startup
Use VoiceCommandDefinitionManager API in Windows.ApplicationModel.VoiceCommands namespace
Update the VCD with application data
© 2015 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

22. Handle the Voice Command Activation
Override App.OnActivated method
Check for ActivationKind.VoiceCommand
Use VoiceCommandActivatedEventArgs.Result to determine what the user said
Navigate to page and continue the interaction

23. Handle the Voice Command Activation
protected override void OnActivated(IActivatedEventArgs e) {
Frame rootFrame = Window.Current.Content as Frame;
if (rootFrame == null) { //create a frame }
if (e.Kind == ActivationKind.VoiceCommand)
{ var commandArgs = e as VoiceCommandActivatedEventArgs;
SpeechRecognitionResult speechRecognitionResult =
commandArgs.Result;
  string voiceCommandName = speechRecognitionResult.RulePath[0];
  if (voiceCommandName == “showTripToDestination")
rootFrame.Navigate(typeof(TripPage), speechRecognitionResult); }
  Window.Current.Activate();

24. In-App Speech Recognition
Build 2014
11/27/2017
In-App Speech Recognition
Check if the interaction was input through voice or text
For voice, use Speech to continue the user interaction in the app
Use Windows.Media.SpeechSynthesis APIs to talk to the user
Use Windows.Media.SpeechRecognition for speech recognition
© 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

25. Background Voice Commands Architecture
Build 2015
11/27/2017 3:33 PM
Background Voice Commands Architecture
Adventure Works, what trips do I have coming up?
Your App
Go to Adventure Works
Here are your upcoming trips A
Yosemite
August 15th
San Francisco
May 29th
VCD
Web Service 1 7
Voice Command
App Service
Render XAML 2 8
Isolated Storage 9 3 6
Microsoft Speech
Recognition Service
Windows Speech
Platform 4 5
© 2015 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

26. Background Voice Commands - How To
Create the Voice Command Definition (VCD)
Register the VCD XML on App Startup
Handle the Voice Command Activation in an App Service

27. Voice Command Definition
<VoiceCommands xmlns="
<CommandSet xml:lang="en-us" Name=“en-us-CommandSet">
<AppName> Adventure Works </AppName>
<Example> Show my trip to San Francisco </Example>
<Command Name=“showTripToDestination">
<Example>Show my trip to San Francisco</Example>
<ListenFor >show [my] trip to {destination}</ListenFor>
<Feedback>Looking for trip to {destination}</Feedback>
<VoiceCommandService Target=“AdventureWorksService"/>
</Command>
Register the Voice Command Definition on app startup

28. Create an AppService Inherit from IBackgroundTask and implement Run()
Register the AppService in the package.appxmanifest
public async void Run(IBackgroundTaskInstance taskInstance)
<Extensions>
<Extension
Category="windows.appService"
EntryPoint="AdventureWorks.AdventureWorksService">
<AppService Name=“AdventureWorksService"/>
</Extension>
</Extensions>
See Session 765 For Details on AppServices

29. WinRT Namespaces and APIs
using Windows.ApplicationModel.AppService;
using Windows.ApplicationModel.Background;
using Windows.ApplicationModel.VoiceCommands;
VoiceCommandDefinitionManager
VoiceCommandDefinition
VoiceCommandServiceConnection
VoiceCommand
VoiceCommandUserMessage
VoiceCommandContentTile
VoiceCommandResponse

30. Interaction Flow Your AppService Cortana RequestDisambiguationAsync
Build 2015
11/27/2017 3:33 PM
Interaction Flow
Cortana
Your AppService
Adventure Works, cancel my trip!
I’ll get Adventure Works on that!
Invoke IBackgroundTask::Run()
Inspect Voice Command
RequestDisambiguationAsync
Which one?
Las Vegas
Cancun
//build/
Las Vegas
VoiceCommandDisambiguationResult
check SelectedItem
RequestConfirmationAsync
Are you sure?
yes no
yes
VoiceCommandConfirmationResult
Process request
(Talk to web service,
save data
etc.)
© 2015 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

31. Build 2014
11/27/2017
Localization
Canvas Interactions from Background
For each supported language, localize the CommandSet in the Voice Command Definition
When launched, inspect the value of Language property of the VoiceCommandServiceConnection
Localize the dialog on the app provided screens to match Cortana’s language
© 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

32. Speech and Cross Platform Support

33. Developer tools Rich experiences Breadth of devices Desktop apps – WPF
11/27/2017
Developer tools
Win32
C++
Desktop apps – WPF
Universal Windows Apps
.NET
Visual C++ for cross-plat
Rich experiences
Breadth of devices
Visual Studio tools for
Apache Cordova
HTML
Browser-based applications
© 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

34. .NET and Xamarin Shared client app C# logic
11/27/2017
.NET and Xamarin
Share app logic across platforms
Native apps for Windows, iOS and Android devices using .NET
Most productive and powerful platform for creating native applications
Maximize code reuse with Universal Projects & Portable Class Libraries
100% APIs exposed
Everything you can do in xCode or ADT you can do with Visual Studio z
C# + XIB
C# + AXML
C# + XAML
Native UI
Shared
client app C# logic
© 2012 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.
The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

35. Build 2015
11/27/2017 3:33 PM
Demo: Visual C++ for Cross Platform Mobile Development
Create shared C++ components across iOS, Android, & Windows and complement with a presentation tier in Xamarin or Cordova
© 2015 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

36. Visual Studio Emulator for Android
Fast, x86 image built from AOSP
Runs on Hyper-V
Comes with Visual Studio
Speaks Android Debugging
Native capabilities and sensors
Zoom, Rotation, Network, Location, Accelerometer, Battery, OpenGL
SD Card, Camera, Audio playback, Keyboard Input, Screenshots,
Version and Screen Size Configurations, Drag&Drop APKs

37. Android Studio

38. Demo: