1. Perceptive Context for Pervasive Computing Trevor Darrell Vision Interface Group

2. Human-centered Interfaces
Free users from desktop and wired interfaces
Allow natural gesture and speech commands
Give computers awareness of users
Work in open and noisy environments
Outdoors -- PDA next to construction site!
Indoors -- crowded meeting room
Vision’s role: provide perceptive context

3. Perceptive Context Who is there? (presence, identity)
What is going on? (activity)
Where are they? (individual location)
Which person said that? (audiovisual grouping)
What are they looking / pointing at? (pose, gaze)

4. Virtual views
Profile silhouette:
Frontal
Head:

5. Examples: VH-generated views
Faces:
Gait:

6. Range-based stereo person tracking
Range can be insensitive to fast illumination change
Compare range values to known background
Project into 2D overhead view
Plan view
Foreground
Range
Intensity
Merge data from multiple stereo cameras..
Group into trajectories…
Examine height for sitting/standing…

7. Audio input in noisy environments
Acquire high-quality audio from untethered, moving speakers
“Virtual” headset microphones for all users

8. Vision guided microphone array
Cameras
Microphones

9. Audio-visual synchrony detection

10. AVMI Applications + Audio weighting from video (detected face)
Image localization from audio +
image variance
AVMI
Audio weighting from video (detected face)
Audio associated with left face
Audio associated with right face

11. Brightness and depth motion constraints
Parameter space
I t
I t + 1 I Z
Z t
Z t + 1
yt = yt-1

12. Track users head gaze for hands-free pointing…
Closed-loop 3D tracker
Track users head gaze for hands-free pointing…

13. Head-driven cursor Related Projects: Schiele Kjeldsen Toyama
Current application for second pointer or scrolling / focus of attention…

14. Subject not looking at SAM
“Look-to-talk”
Subject not looking at SAM
ASR turned off
Subject looking at SAM
ASR turned on