stanford hci group / cs376 research topics in human-computer interaction Vision-based Interaction Scott Klemmer 17 November 2005

2 cs547: Blake Ross and Asa Dotzler Mozilla: Creating simple software in a geek-driven culture

3 The first vision-based interface  Myron Krueger used computer vision to create Responsive Environments (1970s)  “Reaction is the Medium”  timeline/videoplace_video.html

4 How it works  Video and background are separated in analog using chroma key techniques (think broadcast news)  The first and last points of each raster are stored in the computer, and represent the person’s outline

5 Vision-based UIs: “Verbs”  Detecting and Tracking elements of a certain type in a scene  Capturing contents of detected objects  Recognizing individual members in an object class

6 Vision-based UIs: “Verbs”  Detecting and Tracking elements of a certain type in a scene

7 Vision-based UIs: “Verbs”  Capturing contents of detected objects

8 Vision-based UIs: “Verbs”  Recognizing individual members in a class

9 Vision-based UIs: “Nouns”  People (one or multiple)  Bodies  Faces  Hands  Documents  Objects

10 Vision-based UIs: “Nouns”  People (one or multiple)  Bodies  Faces  Hands  Documents  Objects

11 Vision-based UIs: “Nouns”  People (one or multiple)  Bodies  Faces  Hands  Documents  Objects

12 Background Subtraction I N F R A S T R U C T U R E

13 Image Moments (of Inertia)  0 th moment is mass (total number of pixels)

14 Image Moments (of Inertia)  1 st moment is center

15 Image Moments (of Inertia)  2 nd moment is orientation

16 Tools for Vision apps  Intel’s OpenCV  C API to highly optimized image processing functions (threshold, dilate, optical flow, …)  encv  Fast to run! Slow to develop  Great for vision folks; too low-level for app folks  Papier-Mâché  Java API (and to some extent visual UI) for vision (and other physical input)   Fast to develop! Slow to run  Great for app folks; ~5 fps can sometimes be too slow

17 Good Vision Books  Computer Vision: A Modern Approach  David Forsyth and Jean Ponce (2003)  Fantastic book; but goal is more theoretical understanding than practical application  Robot Vision  Berthold Horn (1987)  More focused on apps and interactive algorithms  Somewhat out of date