Touch and Go: Leading Touch UI with Open Source Presentation by chase.douglas@canonical.com Find us in #ubuntu-touch on Freenode https://launchpad.net/utouch

Agenda A brief history of multitouch platforms Extrapolations from single application interfaces to windowed desktop environments Gestures and an overview of the uTouch gesture stack The future of uTouch and gestures

A brief history of multitouch platforms First multitouch system in 1982, by University of Toronto Input Research Group Pierre Wellner's “Digital Desk” in 1991 revolutionarily explored multitouch use cases http://www.youtube.com/watch?v=S8lCetZ_57g Fingerworks developed multitouch technologies from 1999 to 2005 before being acquired by a fruity company And then there was one more thing...

A brief history of multitouch platforms The iPhone was the first mainstream multitouch platform Introduced consumers to intuitive two-touch gestures Pinch to zoom Drag to pan content First gaming platform with multitouch

A brief history of multitouch platforms What about computers?

A brief history of multitouch platforms Trackpads have long supported “multifinger” functionality Definition: A single point and the number of touches Used for two finger scrolling, two- and three-finger tap Apple introduced full multitouch trackpads in the late 2008 Macbook Pro models Most laptops now have some form of multifinger, semi-multitouch, or full multitouch trackpads

A brief history of multitouch platforms Touch events propagate to a window on screen and then are “grabbed” by the window until they end Works great for touchscreens Issues with touchpads: Use case: two finger scroll, lift one finger, move pointer to another window, touch second finger to scroll again The finger that remained in contact has to change propagation from one window to another

A brief history of multitouch platforms Mobile devices (so far): Touchscreen support only Only one full-screen application at a time No system-level gestures OS X: Trackpad support only Mixture of system- and application- level gestures Windows 7:

A brief history of multitouch platforms What can we learn from these implementations?

A brief history of multitouch platforms Issue: System- vs application-level gestures Apple has tried to introduce system-level gestures after release Backed down when applications broke Conclusions: Start with a mix of system and application gestures Possibly make it configurable by the application

A brief history of multitouch platforms Issue: Touch vs gesture event propagation and delivery Windows only allows a window to exclusively receive touch or gesture events As a result, applications tend to do their own gestures OS X mixes touch and gesture events in one stream An application can handle gestures and follow touches simultaneously Conclusions: Make touch and gesture events usable together

A brief history of multitouch platforms Issue: Trackpad one- and multi-touch interactions Should a touch be canceled if the focused window changes? When, if ever, can a touch change the window it propagates events to? Conclusions: TBD! (Note that in OS X, two-finger scroll is handled on the window server side before touch and gesture events are generated)

Gestures and the X11 uTouch stack Multitouch gestures are a means of simplifying touch event handling The corollary is that gesture event handling is complex! Some issues: Building on X11 Resource and power constraints Consistent “feel” Low latency of touch events Support multiple distinct simultaneous gestures

Gestures and the X11 uTouch stack uTouch is currently based on the X.org X11 window system X11: is almost 24 years old! was not designed for input other than one KB and mouse is tied to the original protocols requires thorough protocol and development review to be extended These are constraints on any X11 based multitouch gesture system

Gestures and the X11 uTouch stack Further constraints we chose for uTouch Only one gesture recognizer instance Uniform “feel” across desktop Recognition occurs only once no matter who is the client Potential for multiple distinct simultaneous gestures Support both multitouch and gestures together and separately Low latency of events

Gestures and the X11 uTouch stack First half of solution: X.org XInput 2.2 multitouch Supports touchpads and touchscreens Clients control delivery of events from given touches Must accept or reject touches they are in control of May receive events for touches they do not control yet The first touchscreen touch may be transformed into pointer motion

Gestures and the X11 uTouch stack Second half of solution: X11 client-side uTouch stack Grabs touches on root window (i.e. entire desktop) Detects all possible combinations of gestures Limited by: What is subscribed Inferences from system- vs application-level gestures If no gestures, reject touches so other clients may handle Begin sending gestures to uTouch clients uTouch clients accept or reject gestures Propagate control to XI 2.2 touch accept/reject

Gestures and the future What about Wayland?

Gestures and the Future Wayland has two key advantages over X11: It is new and has the benefit of hindsight The protocol is not set in stone Wayland 1.0 will likely be released with basic touch support Richer touch and gesture support can be added later Notably, gestures may still be added as a first-class citizen

Gestures and the Future In the X11 uTouch stack, gestures have round trips to sort out acceptance/rejection: uTouch Service X Server Application

Gestures and the Future In a Wayland uTouch stack, gestures can be interleaved with touch events: Wayland-based Compositor Application uTouch Service

Questions please Thank you