A Framework for Multi-resolution and Multi-touch Systems Presenter: 胡婷婷 多重解析度及多重觸控互動系統之開發 Advisor: 洪一平 教授

A Framework for Multi-Resolution & Multi-Touch Systems Page 2 of 41 Outline  Introduction  Related Work  i-m-Top System  i-m-Top SDK  Application  Conclusion & Future Work

A Framework for Multi-Resolution & Multi-Touch Systems Page 3 of 41 Outline IIntroduction RRelated Work ii-m-Top System ii-m-Top SDK AApplication CConclusion & Future Work

A Framework for Multi-Resolution & Multi-Touch Systems Page 4 of 41 Introduction Motivation  Multi-resolution Variable-resolution nature of human vision Cost-effective large high-res display  Multi-touch Intuitive interaction Computer-vision based approach  Software Toolkit Hide engineering challenges

A Framework for Multi-Resolution & Multi-Touch Systems Page 5 of 41 Introduction i-m-Top  interactive multi-resolution tableTop interactive: multi-touch multi-resolution: fovea + peripheral projectors i-m-Top SDK  Toolkit for rapid prototyping applications Featured on multi-resolution/multi-touch

A Framework for Multi-Resolution & Multi-Touch Systems Page 6 of 41 Overview Introduction Projectors Touch Detection Other sensors Cams User Interface i-m-Top SDK Image Processing Applications Pan-tilt Unit Multimedia

A Framework for Multi-Resolution & Multi-Touch Systems Page 7 of 41 Outline IIntroduction RRelated Work ii-m-Top System ii-m-Top SDK AApplication CConclusion & Future Work

A Framework for Multi-Resolution & Multi-Touch Systems Page 8 of 41 Related Work Multi-Resolution Display  Fovea Plus Context LCD screen (fixed) + projector Baudisch, P., Good, N., and Stewart, P. "Focus Plus Context Screens: Combining Display Technology with Visualization Techniques.“ In proceedings of ACM UIST '01

A Framework for Multi-Resolution & Multi-Touch Systems Page 9 of 41 Related Work Multi-Resolution Display  Fovea-Tablett Tablets as high-res displays Tracked with visual marker Multiple high-res is allowed Cons: physical boundary J. Geisler, R. Eck, N. Rehfeld, E. Peinsipp-Byma, C. Schutz, and S. Geggus, “Fovea- tablette: A new paradigm for the interaction with large screens," in HCI (8), 2007

A Framework for Multi-Resolution & Multi-Touch Systems Page 10 of 41 Related Work Software Development Toolkit  DiamondTouch SDK (C++) Multi-touch, multi-user Esenther, A.; Forlines, C.; Ryall, K.; Shipman, S., "DiamondTouch SDK: Support for Multi-User, Multi-Touch Applications", ACM Conference on CSCW, 2002

A Framework for Multi-Resolution & Multi-Touch Systems Page 11 of 41 Related Work Software Development Toolkit  DiamondSpin (Java): Around-the-table interaction Real-time polar to Cartesian transformation public + personal work areas rotatable circular tabletop Shen, C.; Vernier, F.D.; Forlines, C.; Ringel, M., "DiamondSpin: An Extensible Toolkit for Around-the-Table Interaction", CHI 2004

A Framework for Multi-Resolution & Multi-Touch Systems Page 12 of 41 Related Work Software Development Toolkit  T3 (Java) Multiple projectors → single high-res display Mixed-presence collaboration Philip Tuddenham and Peter Robinson. “T3: Rapid Prototyping of High- Resolution and Mixed-Presence Tabletop Applications”, TABLETOP 2007.

A Framework for Multi-Resolution & Multi-Touch Systems Page 13 of 41 Outline  Introduction  Related Work  i-m-Top System Hardware Configuration Touch Detection  i-m-Top SDK  Application  Conclusion & Future Work

A Framework for Multi-Resolution & Multi-Touch Systems Page 14 of 41 i-m-Top System Peripheral Projector Fovea Projector IR Camera IR Illuminator Mirror Pan-tilt Unit 56” 96 ppi 24 ppi

A Framework for Multi-Resolution & Multi-Touch Systems Page 15 of 41 i-m-Top System

A Framework for Multi-Resolution & Multi-Touch Systems Page 16 of 41 Touch Detection i-m-Top System Background Subtraction Opening Down- scale Opening Connected Component Palm Tracking Connected Component PCA Analysis Palm-finger Association Finger Tracking Connected Component PCA Analysis Palm Association Finger Fusion Background Subtraction - - Opening - fingerpalm

A Framework for Multi-Resolution & Multi-Touch Systems Page 17 of 41 Outline  Introduction  Related Work  i-m-Top System  i-m-Top SDK Design Principles Architecture Implementation  Application  Conclusion & Future Work

A Framework for Multi-Resolution & Multi-Touch Systems Page 18 of 41 i-m-Top SDK Design Principles  Efficient prototyping Hide engineering details Use metaphors Provide reusable components  Touch-detection module independent Vision-based & electronic-based approaches  Real-time requirement Touch event handling + UI rendering  Extensibility Integrate with other components

A Framework for Multi-Resolution & Multi-Touch Systems Page 19 of 41 i-m-Top SDK Architecture Virtual Director Scene ViewNode Scene ViewNode Scene Node Viewport / Virtual Camera Position / Textures / Event Handlers / Animation Effects Global Coordinate Local Coordinate Node Virtual Camera (Fovea View) Virtual Camera (Peripheral View)

A Framework for Multi-Resolution & Multi-Touch Systems Page 20 of 41 i-m-Top SDK Architecture Scene Touch Detection View Application Node Other Sensors Projector PTU Speaker Socket Window Message COM Message Parser OpenGL c# Device Control Virtual Director Event queue LED Light CamNode

A Framework for Multi-Resolution & Multi-Touch Systems Page 21 of 41 i-m-Top SDK Loop of virtual director Check Inputs Update States Render Scenes Multimedia Output Affiliated Timer Functions Sleep 1/fps Time spent 1/fps-T T 0

A Framework for Multi-Resolution & Multi-Touch Systems Page 22 of 41 i-m-Top SDK Sample Code  A scene with a photo InitScenes( ){ oScene scene = new oScene();//construct a scene director.AddScene(scene);//add scene to director oView view = new View(0, 0, 1280, 720);//construct a view scene.AddView(view);//add view to scene oPhoto photo = new oPhoto(0,0,-100,640,480,”sun.png”);//construct a photo scene.AddNode(photo);//add to scene }

A Framework for Multi-Resolution & Multi-Touch Systems Page 23 of 41 i-m-Top SDK Implementation  Multi-resolution Handling  Touch-event Processing  Reusable Components

A Framework for Multi-Resolution & Multi-Touch Systems Page 24 of 41 Implementation Multi-resolution Handling  Calibration Homography of fovea & peripheral projections Run-time warping  Masking  Steerable fovea projection Region/Object oriented  Multi-presented Information

A Framework for Multi-Resolution & Multi-Touch Systems Page 25 of 41 Calibration  Homography of fovea & peripheral projections Provide tools for manual calibration Save homography/ center point /pan-tilt of PTU Implementation F1F1 F2F2 F3F3 F4F4 V1V1 V2V2 V3V3 V4V4 Y X V2V2 V1V1 V3V3 V4V4 P1P1 P4P4 P2P2 P3P3 Y X F1F1 F2F2 F3F3 F4F4 V1V1 V2V2 V3V3 V4V4

A Framework for Multi-Resolution & Multi-Touch Systems Page 26 of 41 Calibration  Run-time warping Adapt OpenGL rendering pipeline Implementation Projection Matrix Perspective Division Viewport Transformation Model-View Matrix Local coordinate Global coordinate Eye coordinate Clipping coordinate Normalized device coordinate Window coordinate Warping Homography Matrix

A Framework for Multi-Resolution & Multi-Touch Systems Page 27 of 41 Masking Implementation fovea peripheralfovea

A Framework for Multi-Resolution & Multi-Touch Systems Page 28 of 41 Masking  Find maximum region in fovea Utilize Stencil buffer in OpenGL Implementation Apply arbitrary masks Set ‘0’ in mask regions Set ‘1’ in the fovea region Invert pixels in fovea region

A Framework for Multi-Resolution & Multi-Touch Systems Page 29 of 41 Implementation Steerable Fovea Projection  Region-oriented Spotlight on a specific region Steered by finger gestures  Object-oriented Spotlight on a specific object Move with object  Real-time update Steer the mirror Retrieve homography Apply mask

A Framework for Multi-Resolution & Multi-Touch Systems Page 30 of 41 Multi-presented Information  Different content in fovea and peripheral regions Eg. Google Map Automatically change Implementation oRectangle multi_presentation = new oRectangle(0, 0,-100, 200, 200); multi_presentation.LoadTexture(“context.png");//load low-resolution image multi_presentation.LoadHighTexture(“detail.png");//load high-resolution image

A Framework for Multi-Resolution & Multi-Touch Systems Page 31 of 41 Touch-event Processing Virtual Director Implementation Touch Detection Pass Message Parser Node i-m-Top SDK Touch events ReceiveCheckDispatchHandle (by event handler)

A Framework for Multi-Resolution & Multi-Touch Systems Page 32 of 41 Implementation Touch-event Processing  Receive & parse Run on background threads  Pass to director Gather all events in one detection frame before pass if rendering rate < detection rate » Keep vital events: ‘Up’, ‘Down’ » Discard other events: ‘Move’ Virtual Director Pass Message Parser Node i-m-Top SDK Receive

A Framework for Multi-Resolution & Multi-Touch Systems Page 33 of 41 Implementation Touch-event Processing  Check ‘Down’: Test if hit some node By SELECTION mode in OpenGL If yes: add to TargetTable (event id → node) ‘Move’, ‘Up’: Check if there is a target Hit TargetTable by event id ‘Up’ : Remove from TargetTable  Dispatch Dispatch events to each node Virtual Director Dispatch Message Parser Node i-m-Top SDK Check

A Framework for Multi-Resolution & Multi-Touch Systems Page 34 of 41 Implementation Touch-event Processing  Define event handler Node decides how to act Default handler Move/Scale/Rotate under ‘Move’ events Redefine handler InitScenes( ){ //…… //add new event handlers to the photo object photo.MultiFingerEvent += new MultiFingerHandler(MyMultiFingerHandler); photo.FingerClick += new FingerClickHandler(MyFingerClickHandler); } Virtual Director Handle Message Parser Node i-m-Top SDK

A Framework for Multi-Resolution & Multi-Touch Systems Page 35 of 41 P4P4 Rotation: R Implementation P1P1 P2P2 P3P3 P4P4 Translation: T Scaling: S P 12 P3P3 P4P4 P3P3 P 34 P3P3 P4P4 P1P1 P2P2

A Framework for Multi-Resolution & Multi-Touch Systems Page 36 of 41 Implementation Reusable components  Multimedia Video/audio Camera capture  UI components Virtual keyboard Text-pad Button, scrollbar…

A Framework for Multi-Resolution & Multi-Touch Systems Page 37 of 41 Outline  Introduction  Related Work  i-m-Top System  i-m-Top SDK  Application  Conclusion & Future Work

A Framework for Multi-Resolution & Multi-Touch Systems Page 38 of 41 Application Map Browser  Different content in the fovea & peripheral Photo Browser  Object/Region oriented fovea projection Emotion Player  Interactive art c

A Framework for Multi-Resolution & Multi-Touch Systems Page 39 of 41 Outline  Introduction  Related Work  i-m-Top System  i-m-Top SDK  Application  Conclusion & Future Work

A Framework for Multi-Resolution & Multi-Touch Systems Page 40 of 41 Conclusion i-m-Top  Multi-resolution Accommodating to human vision Cost-effective large high-res display  Multi-touch Intuitive interaction i-m-Top SDK  Hide engineering challenges Rapid prototyping applications Investigating new interfaces / interactions

A Framework for Multi-Resolution & Multi-Touch Systems Page 41 of 41 Future Work  Attentive computing Combine with gaze-tracking  3D interaction Surface to space  Auto calibration Vision-based Continuous pan-tilt / homography map

Thanks for your listening