1. Paul Varcholik, Joseph J. LaViola Jr., Denise Nicholson Institute for Simulation and Training University of Central Florida pvarchol@ist.ucf.edupvarchol@ist.ucf.edu, jjl@eecs.ucf.edu, dnichols@ist.ucf.edujjl@eecs.ucf.edudnichols@ist.ucf.edu

2. Multi-Touch: Introduction

3. Framework Motivation  Primary multi-touch components 1. Physical interaction surface 2. Software system for collecting and interpreting points of contact Present a barrier to entry for researchers focused on higher-level interface issues or application development

4. Multi-Touch Origins (pre-2000)  Keyboards  Touchscreens  Pen-based computing  1984 – Bob Boie (Bell Labs) perhaps the first multi-touch screen  1985 – Bill Buxton, Multi-Touch TabletMulti-Touch Tablet  1990 – Sensor Frame, Paul McAvinney (CMU) Optical Sensor 3 fingers (some trouble with ambiguous finger positions)  1998 – Fingerworks

5. Multi-Touch Origins (2000 to present)  2001 – Paul Deitz – Mitsubishi DiamondTouch  2003 – Jazz Mutant  2004 – Andy Wilson – TouchLight  Oct. 2005 – Jeff Han – FTIR, UIST Paper  Feb. 2006 – Jeff Han – TED Talk  2006 – Jeff Han – Perceptive Pixel (CNN Magic Wall)  Spring 2007 – Microsoft Surface  Summer 2007 – Apple iPhone  2008 – Open-Source Community  2009 – Windows 7

6. Framework Requirements Essential ComponentDescription Multi-Touch SurfaceCommercial-off-the-shelf hardware, requiring near zero pressure to detect an interaction point Software Hit-TestingAbility to determine the presence and location of each point of surface contact; supporting at least four concurrent users Software Point TrackingIdentifying a continuous point of contact and reporting its velocity and duration

7. Framework Requirements Secondary ComponentDescription Application Programming Interface (API) A software system upon which multiple multi-touch applications can be developed Multi-Platform SupportThe ability to access multi-touch interaction data from different computing environments (e.g. languages, OS’s, etc.) ReconfigurationModifying the software system without recompilation Software ServiceAllowing multiple applications to access multi-touch interaction data simultaneously, including over a computer network Presentation-Layer Independence Isolating the multi-touch interaction data from the system to graphically present such data, allowing any GUI to be employed when developing multi-touch applications Mouse EmulationSupport for controlling traditional ‘Window, Icon, Menu, Pointing Device’ interaction through a multi-touch surface Tangible InterfacesThe ability to detect and interact with physical devices placed on or near the multi-touch surface Customizable Gesture SystemSupport for training arbitrary multi-touch gestures and mapping them to software events Shape Detectione.g. Finger, blob, hand

8. Multi-Touch Hardware  Frustrated Total Internal Reflection (FTIR) 1. Optical waveguide 2. Supporting structure (e.g. cabinet) 3. IR sensing camera 4. Projector and diffuser 5. IR emission source 6. Computer

9. Multi-Touch Efforts: Hardware

10. Software Framework  Image processing  Presentation layer  Multi-platform communication  Pen/writing style interaction (Ink)  Gesture recognition

11. Image Processing Pipeline WebcamRotateFlip Background Filter Threshold Scaling Blob Detection GrayScale Grayscale Image 8bpp Binary Image 1bpp Calibration Point Tracking Processed Frame Camera Frame 24bpp

12. Image Processing Raw Camera Frame Background Filtered Threshold Processed Points

13. Multi-Touch Efforts: Software

14. Mouse Emulator FunctionGesture Description Left ClickQuick tap on the surface with one finger. Left Click (alternate)While holding down a finger, tap another finger to the left side of the first. DragPerform a Left Click (alternate) but do not release the left side press. Drag both fingers to the destination and release. Right ClickWhile holding down a finger, tap another finger to the right side of the first. Double ClickTap two fingers at the same time. Mouse Wheel ScrollWhile holding down a finger, drag another finger vertically and to the right side of the first. Dragging up scrolls the mouse wheel up and vice versa. Alt-TabWhile not a mouse command, the Alt-Tab command is a useful Windows feature that switches between applications. To perform an Alt-Tab, hold down a finger and drag another finger horizontally above the first. Dragging to the left moves backward through the list of active applications and dragging to the right moves forward.

15. Multi-Touch Efforts: Video Montage

16. Multi-Touch Efforts: Additional Videos  Demo Explorer Demo Explorer  Improved Calibration Improved Calibration  Multi-Touch Starcraft Multi-Touch Starcraft  Mouse Emulator Mouse Emulator  Ink Demonstration Ink Demonstration  Particle System Demonstration Particle System Demonstration  SurfaceSimon SurfaceSimon  SurfaceCommand SurfaceCommand

17. Conclusions  The Framework: In development since Fall 2007 Open-source availability since Spring 2008 Now in its fourth major release Answers most of the specified requirements Successfully employed by UCF and colleagues from Brown University Downloaded extensively by the open-source community Provides a robust, extensible platform for research in multi-touch interaction

18. Paul Varcholik, Joseph J. LaViola Jr., Denise Nicholson Institute for Simulation and Training University of Central Florida pvarchol@ist.ucf.edupvarchol@ist.ucf.edu, jjl@eecs.ucf.edu, dnichols@ist.ucf.edujjl@eecs.ucf.edudnichols@ist.ucf.edu

19. Resources: Commercial Platforms  Microsoft Surface Microsoft Surface  Mitsubishi DiamondTouch Mitsubishi DiamondTouch  Apple iPhone & iPod Touch Apple iPhone & iPod Touch  Perceptive Pixel Perceptive Pixel  N-trig N-trig  MultiTouch OY (Finland) MultiTouch OY  Jazz Mutant Jazz Mutant  HP TouchSmart HP TouchSmart  Dell Latitude XT2 Dell Latitude XT2  TacTable TacTable

20. Resources: Open-Source Platforms  TouchLib (NUI Group) TouchLib  tBeta (NUI Group) tBeta  reacTIVision reacTIVision  Bespoke Multi-Touch Framework Bespoke Multi-Touch Framework  Sparsh UI (Iowa State) Sparsh UI  BBTouch BBTouch  Touche` Touche`

21. Resources: Online  NUI Group NUI Group  TouchKit by NOR_/D TouchKit by NOR_/D  Multi-Touch Systems that I Have Known and Loved – Bill Buxton, Microsoft Research Multi-Touch Systems that I Have Known and Loved  Jeff Han’s TED Talk Jeff Han’s TED Talk  Bespoke Software Bespoke Software