An Update on the Pebbles Project: PocketPCs, Smartphones and TabletPCs for Universal Remote Control and A New Input Method for Small Devices Brad A. Myers July 30, 2003 Brad A. Myers July 30, 2003 Human Computer Interaction Institute School of Computer Science Carnegie Mellon University

Brad Myers2 Pebbles Project l Use of multiple devices at the same time l Pocket PC and desktop PC l Mobile Phone and “Smart Home” l Multiple handhelds in a meeting l Pocket PC and appliances l Multiple users with their devices l Single user with multiple devices l Use of multiple devices at the same time l Pocket PC and desktop PC l Mobile Phone and “Smart Home” l Multiple handhelds in a meeting l Pocket PC and appliances l Multiple users with their devices l Single user with multiple devices

Brad Myers3 Handhelds are communicating l l BlueTooth l Cell-phone network (G3) l (Infrared) l (Wires or cradle) l l BlueTooth l Cell-phone network (G3) l (Infrared) l (Wires or cradle)

Brad Myers4 Premises of our Research “With the coming wireless technologies, connecting the PCs and PDAs together will no longer be an occasional event for synchronization. Instead, the devices will frequently be in close, interactive communication.” — Brad Myers, “Using Hand-Held Devices and PCs Together,” Comm. ACM, Vol. 44, No. 11. Nov., pp

Brad Myers5 Pebbles is: PEBBLESPEBBLES PEBBLESPEBBLES DAs for ntry of oth ytes and ocations from xternalources. DAs for ntry of oth ytes and ocations from xternalources.

Brad Myers6 Previously Reported Here l Microsoft has supported this research since ‘98 l Prior presentations here about Pebbles: l For business meetings: l For group work l How laser pointers can be integrated l To augment desktop applications: l For individuals l For classrooms l To make lectures more interactive l For military command posts: l To facilitate communication and collaboration l Microsoft has supported this research since ‘98 l Prior presentations here about Pebbles: l For business meetings: l For group work l How laser pointers can be integrated l To augment desktop applications: l For individuals l For classrooms l To make lectures more interactive l For military command posts: l To facilitate communication and collaboration

Brad Myers7 Example: Slide Show Commander l Released 1999 l Remote Control of PowerPoint l Two-way communication l Picture of slide, notes, titles, other applications, preview l Scribble, click on links l Released 1999 l Remote Control of PowerPoint l Two-way communication l Picture of slide, notes, titles, other applications, preview l Scribble, click on links

Brad Myers8 Slide Show Commander, #2 l List of applications l Preview of other slides l (Soon: continuous preview ahead) l List of applications l Preview of other slides l (Soon: continuous preview ahead)

Brad Myers9 Download SlideShow Commander l Formerly was for sale, but no longer  l Research version distribution limited to academics and Pebbles project supporters, including Microsoft employees: l Formerly was for sale, but no longer  l Research version distribution limited to academics and Pebbles project supporters, including Microsoft employees:

Brad Myers10 Example 2: Shortcutter l User-created panels of controls l Create custom interfaces and extensions to PC applications l Augment desktop applications l Useful for prototyping l Example: two handed interaction: scrolling with left hand l Direct manipulation for edit, then set properties l User-created panels of controls l Create custom interfaces and extensions to PC applications l Augment desktop applications l Useful for prototyping l Example: two handed interaction: scrolling with left hand l Direct manipulation for edit, then set properties

Brad Myers11 Shortcutter Widgets l Buttons l Virtual Joystick l Virtual Knob l Sliders l Mouse pad l Gesture panel l Buttons l Virtual Joystick l Virtual Knob l Sliders l Mouse pad l Gesture panel

Brad Myers12 Scenarios of Use l Lean-back mail reading l Controlling media players l … and many others l Lean-back mail reading l Controlling media players l … and many others

Personal Universal Controller l PhD research of Jeffrey Nichols

Brad Myers14 Today’s Talk l Personal Universal Controller l Last year, reported first steps l New this year: l Tablet PC, Smartphone l New devices: UPnP, WMP, Lutron, etc. l Ported to C# from Java l Portability across devices l Better quality UIs due to “high-level templates” l Personal Universal Controller l Last year, reported first steps l New this year: l Tablet PC, Smartphone l New devices: UPnP, WMP, Lutron, etc. l Ported to C# from Java l Portability across devices l Better quality UIs due to “high-level templates”

Brad Myers15 Problem l Appliances are too complex

Brad Myers16 Problem l Too many remotes

Brad Myers17 Problem l April 8, 1991

Brad Myers18 Approach l Use a personal handheld device as an interface to the appliances

Brad Myers19 Why Appliances Are Hard to Use l Too many complex devices, each with its own idiosyncratic interface l Stereo system l Telephones l ATM l Fax machine l Photocopier l Hotel alarm clock l Increasingly computerized l Too many complex devices, each with its own idiosyncratic interface l Stereo system l Telephones l ATM l Fax machine l Photocopier l Hotel alarm clock l Increasingly computerized

Brad Myers20 Existing “Universal” Controls l Pre-programmed at the factory with a subset, l Or, Laboriously hand-programmed by the user l Pre-programmed at the factory with a subset, l Or, Laboriously hand-programmed by the user

Brad Myers21 Our Approach l Two-way communication l Appliance describes its functions l Handheld PUC l Automatically creates interface l Controls the appliance l Displays feedback about appliance status l Two-way communication l Appliance describes its functions l Handheld PUC l Automatically creates interface l Controls the appliance l Displays feedback about appliance status Specifications Control Feedback

Brad Myers22 Automatic Generation of UIs l Benefits l All interfaces consistent on the controller l With conventions of the handheld l Even from multiple manufacturers l Addresses hotel alarm clock problem l Can take into account user preferences l Multiple modalities (GUI + Speech UI) l A Hard Problem l Previous automatic systems have not generated high quality interfaces l Benefits l All interfaces consistent on the controller l With conventions of the handheld l Even from multiple manufacturers l Addresses hotel alarm clock problem l Can take into account user preferences l Multiple modalities (GUI + Speech UI) l A Hard Problem l Previous automatic systems have not generated high quality interfaces

Brad Myers23 Initial User Study l Compared hand-designed interfaces to manufacturers’ l Stereo and telephone l Using PUC, users took 50% less time & made 50% fewer errors l All differences are significant (p < 0.05) l Compared hand-designed interfaces to manufacturers’ l Stereo and telephone l Using PUC, users took 50% less time & made 50% fewer errors l All differences are significant (p < 0.05)

Brad Myers24 Properties of PUC Language l XML l State variables & commands l Each can have multiple labels l Useful when not enough room l Typed variables l Base types: Boolean, string, enumerated, integers, fixed-point, floating-point, etc. l Optional labels for values l Hierarchical Structure l Groups l XML l State variables & commands l Each can have multiple labels l Useful when not enough room l Typed variables l Base types: Boolean, string, enumerated, integers, fixed-point, floating-point, etc. l Optional labels for values l Hierarchical Structure l Groups

Brad Myers25 Dependency Information l Crucial for high-quality interfaces l Expressed as clauses l Operations: l Equals, Less-Than, Greater-Than l Combined Logically l AND, OR l Used for: l Layout l Widget selection l Dynamic graying out l Crucial for high-quality interfaces l Expressed as clauses l Operations: l Equals, Less-Than, Greater-Than l Combined Logically l AND, OR l Used for: l Layout l Widget selection l Dynamic graying out

Brad Myers26 Specifications l Have working specifications for: l Windows Media Player l Simulated Elevator l Axis Pan-Tilt-Zoom camera through UPnP l GMC Denali Driver Information Console l GMC Denali Climate Control System l GMC Denali Navigation System l Lutron lighting system l Sony CamCorder l X-10 lights control l Audiophase stereo l Audio ReQuest hardware MP3 player l WinAmp Media Player l UPnP Light Control l Have working specifications for: l Windows Media Player l Simulated Elevator l Axis Pan-Tilt-Zoom camera through UPnP l GMC Denali Driver Information Console l GMC Denali Climate Control System l GMC Denali Navigation System l Lutron lighting system l Sony CamCorder l X-10 lights control l Audiophase stereo l Audio ReQuest hardware MP3 player l WinAmp Media Player l UPnP Light Control Key: Demonstrate today Demonstrated at Faculty Summit Demonstrated here last year Key: Demonstrate today Demonstrated at Faculty Summit Demonstrated here last year

Brad Myers27 Generating Speech Interfaces l “Universal Speech Interface” (USI) project l Prof. Roni Rosenfeld of CMU l l Creates grammar, language model and pronunciation dictionary from PUC specification l Pronunciation from labels using phonetic rules l Can provide other pronunciations as labels for fine-tuning l Uses dependency information to help with disambiguation and explanation l Supports queries and spoken feedback l Paraphrases as confirmation l “Universal Speech Interface” (USI) project l Prof. Roni Rosenfeld of CMU l l Creates grammar, language model and pronunciation dictionary from PUC specification l Pronunciation from labels using phonetic rules l Can provide other pronunciations as labels for fine-tuning l Uses dependency information to help with disambiguation and explanation l Supports queries and spoken feedback l Paraphrases as confirmation

Brad Myers28 New Work: C# l Ported to C# with.NET Compact Framework l With support from Microsoft – Thanks! l Much faster execution than Java version on PocketPC l Got portability across PocketPC, SmartPhone, TabletPC (desktop) l Significant code reuse l 90% from PocketPC to TabletPC l 60% from PocketPC to SmartPhone l Different input model on SmartPhone requires different generator rules and different widgets l Ported to C# with.NET Compact Framework l With support from Microsoft – Thanks! l Much faster execution than Java version on PocketPC l Got portability across PocketPC, SmartPhone, TabletPC (desktop) l Significant code reuse l 90% from PocketPC to TabletPC l 60% from PocketPC to SmartPhone l Different input model on SmartPhone requires different generator rules and different widgets

Brad Myers29 New Work: High-Level Templates l Templates for “high-level” types l Play control l Time/date l Light on, dim-level, off l Issue: Highly parameterized l E.g., what options available for play group l Tape: play, pause, stop, FF, Rew l CD: play, pause, stop, FF, Rew, next track, prev. track l Live audio feed: play, stop l Answering machine: play, play new, stop, … l Templates for “high-level” types l Play control l Time/date l Light on, dim-level, off l Issue: Highly parameterized l E.g., what options available for play group l Tape: play, pause, stop, FF, Rew l CD: play, pause, stop, FF, Rew, next track, prev. track l Live audio feed: play, stop l Answering machine: play, play new, stop, …

Brad Myers30 High-Level Templates l If PUC generator doesn’t understand template, then falls back to primitive types l Primitive types also provide parameterization l If PUC generator doesn’t understand template, then falls back to primitive types l Primitive types also provide parameterization CD: Camera:

Brad Myers31 Adaptors l “Adaptors” provide the interface to existing (and future) appliances l If do not support specification language directly l Custom hardware l Custom software l Lutron l Windows Media Player l X-10 l Light switches, etc. l “Adaptors” provide the interface to existing (and future) appliances l If do not support specification language directly l Custom hardware l Custom software l Lutron l Windows Media Player l X-10 l Light switches, etc. l AV/C (standard protocol) l Sony CamCorder l HAVi l UPnP l Axis Camera

Brad Myers32 New Work: New Devices l UPnP interface to Axis Camera l Lutron light interface l General Motors automobile l Windows Media Player l Simulated elevator, etc. l UPnP interface to Axis Camera l Lutron light interface l General Motors automobile l Windows Media Player l Simulated elevator, etc.

Brad Myers33 Demonstration l New devices: l Windows Media Player through COM l Axis Camera through UPnP l Simulated Elevator l Simulated GMC Denali systems l Non-driving functions l GUI and speech l Cross platform: PocketPC, SmartPhone, TabletPC l Two way communication: l When state changes from appliance, GUI is updated and can query state with speech l New devices: l Windows Media Player through COM l Axis Camera through UPnP l Simulated Elevator l Simulated GMC Denali systems l Non-driving functions l GUI and speech l Cross platform: PocketPC, SmartPhone, TabletPC l Two way communication: l When state changes from appliance, GUI is updated and can query state with speech

Brad Myers34 Example Screens Axis CameraWindows Media Player

Brad Myers35 Future Work l Remote Control for “Experiences” l Combined control of multiple appliances l Better quality generation l New devices? l Automatic creation of specifications for UPnP devices? l Remote Control for “Experiences” l Combined control of multiple appliances l Better quality generation l New devices? l Automatic creation of specifications for UPnP devices?

EdgeWrite ™ l PhD research of Jake Wobbrock (patent pending) l PhD research of Jake Wobbrock (patent pending)

Brad Myers37 Origins l Prior work on using handhelds as input devices for people with Muscular Dystrophy l Didn’t generalize to other disabilities l Prior work on using handhelds as input devices for people with Muscular Dystrophy l Didn’t generalize to other disabilities

Brad Myers38 Idea: Use Edges for Stability l First design: use edges around entire screen l User study: moving along edges and into corners much more accurate l Especially for people with disabilities l First design: use edges around entire screen l User study: moving along edges and into corners much more accurate l Especially for people with disabilities

Brad Myers39 Initial Design l Place letters around top and sides of screen l Alphabetic layout and by letter frequency l Not effective  l Distances too long l Character sizes too small l Visual search times too large l Requires too much visual attention l Place letters around top and sides of screen l Alphabetic layout and by letter frequency l Not effective  l Distances too long l Character sizes too small l Visual search times too large l Requires too much visual attention “the woman”

Brad Myers40 Next Design: Smaller Area l Added custom template with a square hole l Designed alphabet using gestures inside of hole, along edges and into corners l First implementation: for Palm l Added custom template with a square hole l Designed alphabet using gestures inside of hole, along edges and into corners l First implementation: for Palm

Brad Myers41 Alphabet Properties l Recognizes based on sequence of corners l Users can wiggle during diagonals l Not speed dependent l Easy to customize and adapt letter forms l Even user-driven l Letters “feel” normal l Use multiple strokes along edge l Presentation (only) uses arcs l No capitalization mode l Capitals end in upper left l Recognizes based on sequence of corners l Users can wiggle during diagonals l Not speed dependent l Easy to customize and adapt letter forms l Even user-driven l Letters “feel” normal l Use multiple strokes along edge l Presentation (only) uses arcs l No capitalization mode l Capitals end in upper left

Brad Myers42 EdgeWrite Engineering l Change target area shapes/sizes l Start area vs. moving l Right-handed vs. left-handed l Plastic can’t sit on digitizer l Change target area shapes/sizes l Start area vs. moving l Right-handed vs. left-handed l Plastic can’t sit on digitizer

Brad Myers43 Current Alphabet Design l Also have alternative forms l 104 unique characters l 288 total character forms l Also have alternative forms l 104 unique characters l 288 total character forms

Brad Myers44 Initial User Study l For able-bodied novices: l Just as learnable as Graffiti l Accuracy 18% better l Speed not significantly different l For able-bodied novices: l Just as learnable as Graffiti l Accuracy 18% better l Speed not significantly different

Brad Myers45 User Study, Cont. l For handicapped users: l Succeeded with EdgeWrite when unable to use Graffiti l Parkinson’s Disease l Cerebral Palsy l Muscular Dystrophy l For example: l “The g i gbsiangu% fast” vs: “The dog is going fast” l For handicapped users: l Succeeded with EdgeWrite when unable to use Graffiti l Parkinson’s Disease l Cerebral Palsy l Muscular Dystrophy l For example: l “The g i gbsiangu% fast” vs: “The dog is going fast”

Brad Myers46 Future EdgeWrite Work l Test with Xbox joystick l Test with other devices (watch, phone, etc.) l Test long-term (expert) use l Test on unstable platforms (riding on a bus or car) l Etc…. l Test with Xbox joystick l Test with other devices (watch, phone, etc.) l Test long-term (expert) use l Test on unstable platforms (riding on a bus or car) l Etc….

Brad Myers47 For More Information l Many papers on the Pebbles site l On PUC and other work l Shortcutter and Remote Commander available for free downloading: l SlideShow Commander available to Microsoft employees: l Many papers on the Pebbles site l On PUC and other work l Shortcutter and Remote Commander available for free downloading: l SlideShow Commander available to Microsoft employees:

Brad Myers48 Thanks to Our Sponsors! Supported by grants from: DARPA NSF Microsoft Pittsburgh Digital Greenhouse General Motors NEC Foundation of America And equipment grants from: Hewlett Packard Lucent Technologies Mitsubishi Palm Computing Symbol Technologies IBM and IBM Canada SMART Technologies Supported by grants from: DARPA NSF Microsoft Pittsburgh Digital Greenhouse General Motors NEC Foundation of America And equipment grants from: Hewlett Packard Lucent Technologies Mitsubishi Palm Computing Symbol Technologies IBM and IBM Canada SMART Technologies VividLogic Synergy Solutions TDK Systems Europe, LTD Lutron, Inc. Lantronix, Inc. Handango

Brad Myers49 Thanks to Pebbles Students l This research performed by a large number of students since 1997: Rishi Bhatnagar Ben Bostwick Franklin Chen Yu Shan Chuang Karen Cross Carl Evankovich Marc Khadpe Dave Kong Chun-Kwok Lee Joonhwan Lee Mathilde Pignol Suporn Pongnumkul Jennifer Li Leo Lie Jack Lin Kevin Litwack A. Chris Long Rob Miller Jeff Nichols Choon Hong Peck Rajesh Seenichamy Herbert Stiel Jeff Stylos Claire Tokar Marsha Tjandra Adrienne Warmack Jacob O. Wobbrock Jerry Yang Sunny Yang Brian Yeung

An Update on the Pebbles Project: PocketPCs, Smartphones and TabletPCs for Universal Remote Control and A New Input Method for Small Devices Brad A. Myers July 30, 2003 Brad A. Myers July 30, 2003 Human Computer Interaction Institute School of Computer Science Carnegie Mellon University

Black slide, rest are extra

Brad Myers52 Discussion of Comparison l Our hand designed interfaces succeeded for several reasons l Good organization l Each button has one function l Good labels l Only available functions are active l Others, hidden on tabs or grayed out l Better feedback and error messages l Our hand designed interfaces succeeded for several reasons l Good organization l Each button has one function l Good labels l Only available functions are active l Others, hidden on tabs or grayed out l Better feedback and error messages

Brad Myers53 Current PUC Specification Language l XML l Full documentation for the specification language and protocol: l Contains sample specification for Audiophase stereo l XML l Full documentation for the specification language and protocol: l Contains sample specification for Audiophase stereo

Brad Myers54 Examples of Generated GUIs l Stereo and X-10