Vision-based Interaction Scott Klemmer 11 November 2004

Vision-based Interaction cs547: Michael Shiloh Teleo: Rapid Prototyping Toolkit Making Things 11 November 2004 Vision-based Interaction

The first vision-based interface Myron Krueger used computer vision to create Responsive Environments (1970s) “Reaction is the Medium” http://www.artmuseum.net/w2vr/timeline/videoplace_video.html 11 November 2004 Vision-based Interaction

Vision-based Interaction How it works Video and background are separated in analog using chroma key techniques (think broadcast news) The first and last points of each raster are stored in the computer, and represent the person’s outline 11 November 2004 Vision-based Interaction

Vision-based UIs: “Verbs” Detecting and Tracking elements of a certain type in a scene Capturing contents of detected objects Recognizing individual members in an object class 11 November 2004 Vision-based Interaction

Vision-based UIs: “Verbs” Detecting and Tracking elements of a certain type in a scene 11 November 2004 Vision-based Interaction

Vision-based UIs: “Verbs” Capturing contents of detected objects 11 November 2004 Vision-based Interaction

Vision-based UIs: “Verbs” Recognizing individual members in a class 11 November 2004 Vision-based Interaction

Vision-based UIs: “Nouns” People (one or multiple) Bodies Faces Hands Documents Objects 11 November 2004 Vision-based Interaction

Vision-based UIs: “Nouns” People (one or multiple) Bodies Faces Hands Documents Objects 11 November 2004 Vision-based Interaction

Vision-based UIs: “Nouns” People (one or multiple) Bodies Faces Hands Documents Objects 11 November 2004 Vision-based Interaction

Background Subtraction I N F R A S T R U C T U R E Background Subtraction Two Cameras 640x480 Sony Firewire camera for video; Mounted behind board; eliminates occlusion; Used for Tracking Notes 3 megapixel Epson still USB camera; Treat as ~1fps ‘video’ camera; Mount in front, use for ink capture 11 November 2004 Vision-based Interaction

Image Moments (of Inertia) 0th moment is mass (total number of pixels) 11 November 2004 Vision-based Interaction

Image Moments (of Inertia) 1st moment is center 11 November 2004 Vision-based Interaction

Image Moments (of Inertia) 2nd moment is orientation 11 November 2004 Vision-based Interaction

Vision-based Interaction Tools for Vision apps Intel’s OpenCV C API to highly optimized image processing functions (threshold, dilate, optical flow, …) http://www.intel.com/research/mrl/research/opencv Fast to run! Slow to develop Great for vision folks; too low-level for app folks Papier-Mâché Java API (and to some extent visual UI) for vision (and other physical input) http://guir.berkeley.edu/papier-mache Fast to develop! Slow to run Great for app folks; ~5 fps can sometimes be too slow 11 November 2004 Vision-based Interaction

Vision-based Interaction Good Vision Books Computer Vision: A Modern Approach David Forsyth and Jean Ponce (2003) Fantastic book; but goal is more theoretical understanding than practical application Robot Vision Berthold Horn (1987) More focused on apps and interactive algorithms Somewhat out of date 11 November 2004 Vision-based Interaction

iRoom demo after class… 11 November 2004 Vision-based Interaction