1. Visual Screen: Transforming an Ordinary Screen into a Touch Screen Zhengyou Zhang & Ying Shan Vision Technology Group Microsoft Research zhang@microsoft.com

2. Motivation Transform an ordinary screen into a touch screen using an ordinary camera

3. Configuration Position a camera so it can see the whole screen

4. Fingertip Tracker Action Detector & Event Generator Mouse Events Video Input Visual Screen Fingertip Detector Homography Mapping Screen Detection Virtual Touch Screen Non-flatness correction Calibration The System

5. System Diagram

6. Calibration Mapping from image coordinate to the screen coordinate Homography if the screen is flat More accurate method required when the screen is curved

7. Plane Plus Residue Flow H

8. The actual screen coordinate can be found by the homography corrected by an interpolated residue vector Mapping

9. Plane Plus Residual Flow Before correction After correction Original calibration points Reprojected image points

10. Segmentation Images of screen pixels have some degrees of invariance in the color space Compute a color model for the screen without the indicator Compute a color model for the indicator Use standard Bayes classifier to segment the indicator from the screen background

11. Segmentation Before segmentation After segmentation

12. Locate the Finger Tip Initial location from horizontal histogram Fit the centerline of the finger Finger tip is the intersection of the centerline and the segment boundary

13. Experiments: Draw Bubbles for Fish

14. Experiments: Barney Under the Sea

15. Thank You