Untold Stories of Touch, Gesture, & NUI Joe Fletcher Design Manager, Microsoft Surface

Touch is not good for everything Touch is a computing evolution It is not an end all solution Touch is a computing evolution It is an addition element for users to interact with their content in new and unique ways Touch is great for specific aspects New technology has enabled touch systems, which can enhance certain activities

Microsoft Surface Microsoft Surface is a revolutionary computer that responds to natural hand gestures and real-world objects, creating unique new ways for people to collaboratively interact with digital content.

Microsoft Surface

Vision Strategy Design

Surface Windows 7 Massive Multi-touch, Multi-user Unique Spot PC’s, Form Factors, Technology

Larger Surface Team

Touch is not good for everything Touch is an evolution Touch is great for specific aspects

Natural User Interface Touch The act of directly touching your digital content Gesture Hand motions to take an action Manipulation Alteration or freeform movement of digital content using touch Natural User Interface The combination of these and other elements the create a unique way to interact with your digital content

Surface Design Principles History and Ecology Surface Design Principles Touch Challenges

Surface Design Principles History and Ecology Surface Design Principles Touch Challenges

Why Evolve?

Command Line Interface (CLI)

Graphic User Interface (GUI)

Natural User Interface (NUI)

Natural User Interface (NUI)

Natural User Interface (NUI)

Microsoft Surface and NUI Microsoft Surface computing uses sensing and display technology to infuse everyday surfaces with digital content and is comprised of the following unique attributes: Direct interaction Multi-touch Multi-user Object recognition

NUI vs. Web 2.0

NUI & Touch should not be a novelty

Keyboard Mouse Stylus Single point Multi point Input Ecology Accuracy Good for Accuracy Con Limited Mouse Good for Moving across the screen efficiently Con Indirect Stylus Good for Freeform input Con Low accuracy Need large controls Single point Good for Direct object selection Con Not for quick input Low accuracy Need of large controls Mis-hits from fingernails of perceived hit area Multi point Good for Manipulating objects Social Con Not for input Low accuracy Need large controls Support of hands and screen

[Partial] Touch Ecology Multi point (screen) Reacts to direct input Multi Touch (screen) Reacts to multiple inputs Orients UI to the direction of the touch Spatial recognition of system NUI (Gesture & Physical/Digital) Reacts to multiple inputs Spatial Recognition - Orients UI to the direction of the touch Object recognition - Blurring physical and digital

Tap is not the new click ≠

Context, task, and use define the solution

Context, task, and use define the solution Touch is not a solution in search of a problem

Touch is not good for everything Touch is an evolution Touch is great for specific aspects

Surface Design Principles History and Ecology Surface Design Principles Touch Challenges

Create an inflection point

The start of almost every successful technological innovation is marked by an inflection point, which defines a moment of dramatic change.

Companies spend 90% of their energy trying to imitate

Microsoft Surface and principle driven design

Microsoft Surface and principle driven design Using history to develop the future

CLI GUI NUI Text Graphics Objects

Recall Recognition Intuition CLI GUI NUI Text Graphics Objects

CLI GUI NUI Directed Exploratory Contextual Recall Recognition Intuition CLI GUI NUI Text Graphics Objects

CLI GUI NUI High-Low Double Medium Fast Few Directed Exploratory Contextual Recall Recognition Intuition CLI GUI NUI Text Graphics Objects

CLI GUI NUI Disconnected Indirect Unmediated High-Low Double Medium Fast Few Directed Exploratory Contextual Recall Recognition Intuition CLI GUI NUI Text Graphics Objects

CLI GUI NUI Static Responsive Evocative Disconnected Indirect Unmediated High-Low Double Medium Fast Few Directed Exploratory Contextual Recall Recognition Intuition CLI GUI NUI Text Graphics Objects

NUI Principle of Performance Aesthetics Evocative Principle of Direct Manipulation Unmediated Principle of Scaffolding Fast Few Principle of Contextual Environments Contextual Principle of the Super Real Intuition NUI

Microsoft Surface and principle driven design

Microsoft Surface and principle driven design Three Super Principles

Social Multiple simultaneous users

Social Seamless Multiple simultaneous users Digital and Physical combined

Seamless

Social Seamless Spatial Multiple simultaneous users Digital and Physical combined Spatial Kinesiology

Overcome Homeostasis

Touch is not good for everything Touch is an evolution Touch is great for specific aspects

Surface Design Principles History and Ecology Surface Design Principles Touch Challenges

Let’s become practitioners

Unique Spot PC’s, Form Factors, Technology

Challenge Form Factor & Technology

Touch Hardware Ecology

Touch Hardware Ecology Capacitive Capacitive Infrared Vision system Stylus

Touch Hardware Ecology Capacitive N-Trig (2-10) Capacitive N-trig (2-10) Infrared HP (2) Vision system Surface (52) Stylus Wacom (1)

Challenge Patterns

Swipe Left: Go back in history Swipe Up: Go to the top of the page FireFox Swipe Left: Go back in history Swipe Up: Go to the top of the page

Swipe Left: Go Forward in History Swipe up: Go to the bottom of a page Apple iPhone Swipe Left: Go Forward in History Swipe up: Go to the bottom of a page

Challenge Orientation

Horizontal Vertical

Challenge Cardio Effect

Challenge Paradigms

To button or not to button

Challenge Getting people to touch

Exploration Instruct

Challenge Visualizing Touch

Is it working?

Future Where do we go?

Your effort to remain what you are is what limits you Ghost in the Shell

Debating the future of touch is useless

Creating the future is more interesting

Touch is not good for everything Touch is an evolution Touch is great for specific aspects

Take advantage of Microsoft Surface Available from the Surface Community site Download SDK Workstation Edition Learn how to build applications from scratch on Surface by viewing videos on Surface.com Develop for Surface Interested in purchasing a Surface? Email to sbizdesk@microsoft.com Purchase a Surface

Future Vision 2019

Joe Fletcher joeflet@microsoft.com Design Manager, Microsoft Surface Las Vegas, Thank you! Joe Fletcher joeflet@microsoft.com Design Manager, Microsoft Surface

Please Complete an Evaluation Form Your feedback is important! MIX 09 11/7/2018 12:00 PM Please Complete an Evaluation Form Your feedback is important! Evaluation forms can be found on each chair Temp Staff at the back of the room have additional evaluation form copies © 2009 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.