Visual Screen: Transforming an Ordinary Screen into a Touch Screen Zhengyou Zhang & Ying Shan Vision Technology Group Microsoft Research

Motivation Transform an ordinary screen into a touch screen using an ordinary camera

Configuration Position a camera so it can see the whole screen

Fingertip Tracker Action Detector & Event Generator Mouse Events Video Input Visual Screen Fingertip Detector Homography Mapping Screen Detection Virtual Touch Screen Non-flatness correction Calibration The System

System Diagram

Calibration Mapping from image coordinate to the screen coordinate Homography if the screen is flat More accurate method required when the screen is curved

Plane Plus Residue Flow H

The actual screen coordinate can be found by the homography corrected by an interpolated residue vector Mapping

Plane Plus Residual Flow Before correction After correction Original calibration points Reprojected image points

Segmentation Images of screen pixels have some degrees of invariance in the color space Compute a color model for the screen without the indicator Compute a color model for the indicator Use standard Bayes classifier to segment the indicator from the screen background

Segmentation Before segmentation After segmentation

Locate the Finger Tip Initial location from horizontal histogram Fit the centerline of the finger Finger tip is the intersection of the centerline and the segment boundary

Experiments: Draw Bubbles for Fish

Experiments: Barney Under the Sea

Thank You