1. stanford hci group / cs376 research topics in human-computer interaction http://cs376.stanford.edu Vision-based Interaction Scott Klemmer 17 November 2005

2. 2 cs547: Blake Ross and Asa Dotzler Mozilla: Creating simple software in a geek-driven culture

3. 3 The first vision-based interface  Myron Krueger used computer vision to create Responsive Environments (1970s)  “Reaction is the Medium”  http://www.artmuseum.net/w2vr/ timeline/videoplace_video.html

4. 4 How it works  Video and background are separated in analog using chroma key techniques (think broadcast news)  The first and last points of each raster are stored in the computer, and represent the person’s outline

5. 5 Vision-based UIs: “Verbs”  Detecting and Tracking elements of a certain type in a scene  Capturing contents of detected objects  Recognizing individual members in an object class

6. 6 Vision-based UIs: “Verbs”  Detecting and Tracking elements of a certain type in a scene

7. 7 Vision-based UIs: “Verbs”  Capturing contents of detected objects

8. 8 Vision-based UIs: “Verbs”  Recognizing individual members in a class

9. 9 Vision-based UIs: “Nouns”  People (one or multiple)  Bodies  Faces  Hands  Documents  Objects

10. 10 Vision-based UIs: “Nouns”  People (one or multiple)  Bodies  Faces  Hands  Documents  Objects

11. 11 Vision-based UIs: “Nouns”  People (one or multiple)  Bodies  Faces  Hands  Documents  Objects

12. 12 Background Subtraction I N F R A S T R U C T U R E

13. 13 Image Moments (of Inertia)  0 th moment is mass (total number of pixels)

14. 14 Image Moments (of Inertia)  1 st moment is center

15. 15 Image Moments (of Inertia)  2 nd moment is orientation

16. 16 Tools for Vision apps  Intel’s OpenCV  C API to highly optimized image processing functions (threshold, dilate, optical flow, …)  http://www.intel.com/research/mrl/research/op encv  Fast to run! Slow to develop  Great for vision folks; too low-level for app folks  Papier-Mâché  Java API (and to some extent visual UI) for vision (and other physical input)  http://guir.berkeley.edu/papier-mache  Fast to develop! Slow to run  Great for app folks; ~5 fps can sometimes be too slow

17. 17 Good Vision Books  Computer Vision: A Modern Approach  David Forsyth and Jean Ponce (2003)  Fantastic book; but goal is more theoretical understanding than practical application  Robot Vision  Berthold Horn (1987)  More focused on apps and interactive algorithms  Somewhat out of date