VUI01: Using Natural User Interface (NUI) Technologies to Improve User Experience Tim Huckaby Founder / Chairman, InterKnowlogy Founder / CEO, Actus Software Microsoft RD & MVP

About Tim… InterKnowlogy ( Tim Huckaby, Founder/Chairman ● Custom Application Development / Consulting / Software & Systems Engineering Firm headquartered in Carlsbad, CA ● Design, Architect, Build and Deploy Enterprise Class Applications ● Industry Experts: Most of the employees are published, MVPs, RD, Speakers, etc. Microsoft.NET Application development since 2000 Microsoft.NET Rich Client Pioneers / Industry Leaders Microsoft Touch Application Development Leaders (Surface / Windows 7 / Silverlight 3) Information Worker Solutions (VSTO & SharePoint) Integration / Messaging, B2B / B2C, Wireless / Mobility Leading Edge: Surface, WPF, Silverlight, Win7 touch, SharePoint, VSTO Cutting Edge Solutions on emerging Microsoft technologies Largest Client: Microsoft Actus Software ( Tim Huckaby, Founder/CEO Software Product Company, Coming soon…

A little bit about me and why I am here…

Touch Capable Computing: Using Natural User Interface (NUI) Technologies Improve The User Experience Multi-Touch Software built in Microsoft.NET: WPF, Surface, Silverlight, & the phone… Hardware & Software Platforms for Touch Computing Multi-Touch enabled applications used to improve the communication, education and user experience across the software continuum. Presentation Agenda Demos, Demos, Demos

About the Computer I am Using Today… Acer Aspire 1420P Give away “free” to all attendees at the PDC One year old… Price in October-2009: $420 Price in November-2009: $350

Tim Huckaby Founder / Chairman, InterKnowlogy Founder / CEO, Actus Software Microsoft RD & MVP

Multi-Touch ScatterView Photos Application Tim Huckaby Founder, InterKnowlogy & Actus Software Microsoft RD & MVP

The InterKnowlogy “Firebrick” Silverlight Multi-Touch ScatterView Control What is it? ● It’s a Silverlight Control for Intuitive & Comprehensive Multi-Touch Object Manipulation for your Web Applications. Download Site: Example Problem(s) it solves: ● Silverlight’s Touch API Is rudimentary at best. Basically the API gives you “I know something has been touched; but, I don’t know what or where.” Is difficult; takes a ton of programming to implement at any significant level ● The Silverlight Multi-Touch ScatterView Control Makes building a Multi-Touch capable application “Brain-dead simple” in a matter of minutes Contains all the “heavy lifting”: thousands of lines of calculus and trigonometry Contains a sample application and detailed “How-to” documentation Installation Notes: ● Free Developer and Low Cost ($99) production versions available Usage Notes: Multi-Touch is supported in SL3 and above and currently only on Windows 7

Touch Capable Computing Devices are Not New Past Future (Now) Present 1972: PLATO IV Touch Screen Terminal The Microsoft Surface Windows 7 Multi- touch

What is Relatively New is:  Inexpensive Touch Capable, Non Proprietary Hardware  A Multi-Touch Capable Windows OS  A Consistent Multi-Touch API for.NET across 3 Windows Hardware/Software Platforms  Developer Productivity in Touch Capable Apps  The Forecast of Touch Capability in Every New Computing Device in the Short Term

Tim Huckaby Founder / Chairman, InterKnowlogy Founder / CEO, Actus Software Microsoft RD & MVP Surface WPF 3D: The Scripps Research Institute VitruView

History of Surface The first prototype is presented to Bill Gates A dedicated group is formed, and more than 85 prototypes are developed Microsoft Surface is unveiled to the world The concept of an interactive table is born The look and feel of Surface is finalized

Single Touch vs. Multi-Touch You get single touch “for free” in Windows 7 ● Almost everywhere in Windows 7, a touch = a mouse click Multi-Touch is the capability to recognize 2 or more fingers on the screen at once. ● The iPhone made the “pinch gesture” a cultural norm in usability ● Vernacular on hardware capability: “4 touches”

Touch is here on the Windows Platform ● Hardware is ready ● Consumers want it; Has a “cool” factor ● Flexible and extensible programming models

TOUCH HARDWARE IMPLEMENTATIONS  Capacitive  Infrared  Resistive

Sample Hardware Capacitive HP Touchsmart tx2 N-trig (2-10) Infrared HP Touchsmart IQ816 NewWindow(2) Capacitive Dell LatitudeXT2 N-Trig (2-10)

How to touch Resistive Think Push Down and drag

How to touch Capacitive Think Electric Impulse

Common hardware implementations Capacitive ● iPhone / iTouch – Capacitive multi-touch cell phone / PDA ● HP TouchSmart TX2 – Dual touch capacitive laptop screen. ● Dell Latitude XT – Multi-Touch tablet using N-Trig capacitive technology ● MacBook Pro – Capacitive Multi-Touch pad with traditional click ● N-Trig – Capacitive monitor overlay vendor (Windows 7 Drivers)

How to touch Infrared Think laser pointer(s)

Common hardware implementations Infrared ● Surface – Infrared diffused illumination multi-touch “table” device ● Perceptive Pixel – Frustrated total internal reflection multi- touch panels ● HP TouchSmart – Infrared dual touch bezel all in one computer ● NextWindow – Infrared Multi-Touch bezel vendor (Windows 7 Drivers)

NUI GESTURES Tap Flick/Fling/Scroll Pinch/Spread Drag

Tim Huckaby Founder / Chairman, InterKnowlogy Founder / CEO, Actus Software Microsoft RD & MVP Surface Craps

SOFTWARE IMPLEMENTATIONS (ON THE MICROSOFT PLATFORM)  Windows Desktop: WPF  Surface: WPF  Browser: Silverlight  Phone: The Windows Mobile 7 phone

NUI Multi-Touch Scenarios

Touch Scenarios and Windows 7 Developer Platform  At the root is the touch developer platform that exposes touch APIs for any application UI Enhancements  Focusing on core scenarios, many parts of the core UI optimized for touch experiences Gestures  Multitouch gestures enable consistent panning and zooming in most applications Applications  Powerful Multitouch-focused applications to ship separately from Windows 7  Navigating and consuming the Web  Reading and sorting  Viewing photos  Playing casual games  Consuming music and video  Navigating files and arranging windows  Using Microsoft Office Applications

VitruView – Badge Authentication Microsoft Surface

VitruView – 3D manipulation Microsoft Surface

VitruView – Adding annotations with the Virtual Keyboard in 3D

Patient Management Microsoft Surface

Bedside Machine Management Microsoft Surface

Hospital Management Microsoft Surface

Molecule Viewer Microsoft Surface

Silverlight… “WPF Light” Interactive experiences for devices, PC, & Mac platforms (and others…) Strict subset of Windows Presentation Foundation Lightweight download (4.2mbs) Programming model uses XAML + Jscript + WPF Access to WPF tooling Touch Capability in Version 3.0 Lightweight WPF runtime OK LightBlue WPF-based Tools

Eclipsys Maestro: Hospital Management

Tim Huckaby Founder / Chairman, InterKnowlogy Founder / CEO, Actus Software Microsoft RD & MVP EMN8 & REZN8

from our Marketing VP Madeline Pantalone: “It's okay for you to use application for demonstration as long as you are sure to preface it with the phrase: "This is test data only. Products you see may have been created for testing purposes only."

NUI: Games as a Tool for Learning Learning made Fun / Entertaining Proven as a learning tool beyond just Children Computer based Games are a significant part of popular culture Educational Video Game Curriculums ● MIT, UC Irvine, Minnesota, Mich State, etc.

Tim Huckaby Founder / Chairman, InterKnowlogy Founder / CEO, Actus Software Microsoft RD & MVP NASA: Mars Bound

XBOX Kinect (Natal) Motion Based Interface $ Availability: November 4th 40 million Xbox 360 owners RGB camera, three sensors ● (one depth sensor, one motion sensor, and one multi-array microphone) tracks full-body movement, facial expression, and voice with precision Navigate the entire Kinect Hub interface on your Xbox 360 without touching a conventional 360 controller. While you're watching a movie, you can verbally instruct the Xbox to stop, fast- forward, rewind, exit out, etc. by calling out easy commands like, "Xbox. Fast- forward." or "Xbox. Pause.“ Kinect sensor bar scans and monitors 1 million points on the body

XBOX Kinect (Natal) Tim Huckaby Founder / Chairman, InterKnowlogy Founder / CEO, Actus Software Microsoft RD & MVP

The Bad News: Not many applications at launch Not really any applications at launch; just games “little kid games” No exposed API at launch, plus the legal restrictions of “hacking it” ● No building your own apps ● No integration with any of the Microsoft Stack ● No vertical solutions ● ….for the immediate future XBOX Kinect (Natal)

Neural Interfaces

Conscious and non-conscious Interfaces High fidelity EG acquisition device No conductive gel, etc. Non-invasive Use Case Examples: Conscious: mind manipulation of objects Non-Conscious: Ranking music First target market: gaming $300!!!

Tim Huckaby Founder / Chairman, InterKnowlogy Founder / CEO, Actus Software Microsoft RD & MVP

C++ Header files and Import Libraries …which means… Wrap the SDK in managed C++ thereby exposing a the API to a C# application That is very powerful… Emotiv SDK

Multi-Touch Capable ● Common gestures of touch Gesture Capable ● Motion Based Interface ● Tracking of full-body movement, facial expression, and voice with precision Neural Capable ● Conscious and non-conscience interfaces ● Non Invasive NUI The Natural User Interface

Summary  We are at the Cusp of the NUI computing revolution  There are still many “standards” yet to be adopted  Multi-Touch computing will become a mainstream aspect of NUI and consequently of software development  Paradigm Shift from GUI to NUI (Natural User Interface)  Touch capable computing changes the way you design systems and differentiates your applications  Shifting from desktop applications to multi-touch based applications will be a cultural change like the mouse was  There are other components of NUI other than just touch…  Neural & Gesture based interfaces

Your Feedback is Important Please fill out a session evaluation form drop it off at the conference registration desk. Thank you!

More info on InterKnowlogy: ● ● Silverlight.InterKnowlogy.com More info on Actus Software: ● Coming soon…. About Tim Huckaby… ● Founder/Chairman, InterKnowlogy ● Founder/CEO, Actus Software ● Microsoft® Regional Director – Southern California ● Microsoft®.NET Partner Advisory Council Founder / Member ● Microsoft® MVP -.NET ● INETA Speaker – International.NET Users Group Association ● Board Member: Association for Competitive Technology ● Board Member: Dev Connections ● Author / Speaker Tim Huckaby Contact me: Tim Huckaby Phone: Office: Mobile: Blog : FaceBook: LinkedIn: