1. Vision-based Interaction
Scott Klemmer
11 November 2004

2. Vision-based Interaction
cs547: Michael Shiloh
Teleo: Rapid Prototyping Toolkit
Making Things
11 November 2004
Vision-based Interaction

3. The first vision-based interface
Myron Krueger used computer vision to create Responsive Environments (1970s)
“Reaction is the Medium”
11 November 2004
Vision-based Interaction

4. Vision-based Interaction
How it works
Video and background are separated in analog using chroma key techniques (think broadcast news)
The first and last points of each raster are stored in the computer, and represent the person’s outline
11 November 2004
Vision-based Interaction

5. Vision-based UIs: “Verbs”
Detecting and Tracking elements of a certain type in a scene
Capturing contents of detected objects
Recognizing individual members in an object class
11 November 2004
Vision-based Interaction

6. Vision-based UIs: “Verbs”
Detecting and Tracking elements of a certain type in a scene
11 November 2004
Vision-based Interaction

7. Vision-based UIs: “Verbs”
Capturing contents of detected objects
11 November 2004
Vision-based Interaction

8. Vision-based UIs: “Verbs”
Recognizing individual members in a class
11 November 2004
Vision-based Interaction

9. Vision-based UIs: “Nouns”
People (one or multiple)
Bodies
Faces
Hands
Documents
Objects
11 November 2004
Vision-based Interaction

10. Vision-based UIs: “Nouns”
People (one or multiple)
Bodies
Faces
Hands
Documents
Objects
11 November 2004
Vision-based Interaction

11. Vision-based UIs: “Nouns”
People (one or multiple)
Bodies
Faces
Hands
Documents
Objects
11 November 2004
Vision-based Interaction

12. Background Subtraction
I N F R A S T R U C T U R E
Background Subtraction
Two Cameras
640x480 Sony Firewire camera for video; Mounted behind board; eliminates occlusion; Used for Tracking Notes
3 megapixel Epson still USB camera; Treat as ~1fps ‘video’ camera; Mount in front, use for ink capture
11 November 2004
Vision-based Interaction

13. Image Moments (of Inertia)
0th moment is mass (total number of pixels)
11 November 2004
Vision-based Interaction

14. Image Moments (of Inertia)
1st moment is center
11 November 2004
Vision-based Interaction

15. Image Moments (of Inertia)
2nd moment is orientation
11 November 2004
Vision-based Interaction

16. Vision-based Interaction
Tools for Vision apps
Intel’s OpenCV
C API to highly optimized image processing functions (threshold, dilate, optical flow, …)
Fast to run! Slow to develop
Great for vision folks; too low-level for app folks
Papier-Mâché
Java API (and to some extent visual UI) for vision (and other physical input)
Fast to develop! Slow to run
Great for app folks; ~5 fps can sometimes be too slow
11 November 2004
Vision-based Interaction

17. Vision-based Interaction
Good Vision Books
Computer Vision: A Modern Approach
David Forsyth and Jean Ponce (2003)
Fantastic book; but goal is more theoretical understanding than practical application
Robot Vision
Berthold Horn (1987)
More focused on apps and interactive algorithms
Somewhat out of date
11 November 2004
Vision-based Interaction

18. iRoom demo after class…
11 November 2004
Vision-based Interaction