1. David Nguyen Professor John Canny MultiView A Spatially Faithful Video-Conferencing System

2. Talk Outline Motivation Prior Work Our Approach Experiment Discussion Conclusions

3. MultiView is the first practical video conferencing system that preserves many non-verbal cues in group-to-group meetings by improving spatial faithfulness

4. Spatial Faithfulness describes how attention cues (e.g., gaze and gesture) are preserved or distorted across a virtual space.

5. Motivation You’re Fired!

6. Spatial Distortions Collapsed Viewer Effect With only one camera, remote participants take on a shared and incorrect perspective warping nonverbal cues

7. Mona Lisa Effect 010203550

9. Spatial Faithfulness Different levels of Spatial Faithfulness –Mutual – Correctly perceive that attention is or is not being directed at you when it actually is or is not –Partial – Correctly perceive the general direction of attention when it is away from you –Full – Correctly perceive the specific object of attention Donald Dave increasing

10. Talk Outline Motivation Prior Work Our Approach Experiment Discussion Conclusions

11. HYDRA (Sellen et al., 1992) Group Sites Mutual Sp.Fa. Full Sp.Fa. Partial Sp.Fa.

12. GAZE-2 (Vertegaal et al., 2003) Group Sites Mutual Sp.Fa. Full Sp.Fa. Partial Sp.Fa.

13. MAJIC (Okada et al., 1994) Group Sites Mutual Sp.Fa. Full Sp.Fa. Partial Sp.Fa.

14. Talk Outline Motivation Prior Work Our Approach Experiment Discussion Conclusions

15. Unique and Correct Views Multiple Viewpoint Display “Virtually Here” Preserving a geometric relationships between virtual and actual positions provides full spatial faithfulness

16. Gaze Parallax

17. Camera Placement vs. Gaze Parallax Ideally, cameras should be placed at the position of the eyes –Occlusion (either the camera or the image) Cameras can be up to 5 o before gaze parallax is perceived (Chen, 2002) [5°-7°]

18. MultiView Directional Display Big, Bright, High Resolution Display Each view is provided by a projector The projected image is reflected directly back in the direction of the projector The image can be seen at varying heights only behind the projector

19. Construction Retroreflective Layer Reflects image back in direction of source Vertical Diffuser Diffuses image vertically to accommodate varying viewing heights Antireflective Sheet Reduces distracting glares due to glossy surface and front projection setup

20. Brightness vs. Viewing Angle 0O0O α -10 O

21. MultiView Cameras MultiView Screen Projectors

22. Cost Small fixed cost Variable costs increase linearly Alternatives increase quadratically Fixed Costs (MultiView Screen) ItemCost Retroreflective Sheet$50 Lenticular Lens$50 Anti-Glare Layer$600 Total Fixed Cost$700 Variable Costs ItemCost/Unit Camera$100 Projectors$1000

23. Easy to Join

24. Talk Outline Motivation Prior Work Our Approach Experiment Discussion Conclusions

25. Experimental Setup 3O3O

26. 23 participants: UC Berkeley graduate and undergraduate students Each Paid $10 Experiments –Exp 1: 10 Trials (230 Total) –Exp 2: 10 Trials (230 Total) –Exp 3: 30 Trials (690 Total)

27. Experiment 1 & 2: Mona Lisa Effect Each researcher was asked to look or point at one of the five targets Each participant was asked to circle which target each researcher appeared to be looking/pointing at Repeated for 10 trials

28. Experiment 1: Accuracy of Gaze Perception Confusion Matrix –Each column represents actual target stimulus –Each Row represents perceived target –Ideally, this would be a diagonal matrix. Participants were accurate in determining target of gaze –90% were at most one target off

29. Experiment 2: Accuracy of Gesture Perception Confusion Matrix –Each column represents actual target stimulus –Each Row represents perceived target –Ideally, this would be a diagonal matrix. Participants were accurate in determining target of gaze –94% were at most one target off

30. Experiment 3: Eye Contact Researchers and Participants were paired off. Researchers were asked to look at the eyes, at the cameras, below the eyes, to the left of the eyes or to the right of the eyes. Each participant was asked if they felt like the partner was looking at them in the eyes.

31. Mutual Gaze No significant difference in positive responses between different gaze positions “At Cam” yielded <100% positive response Participants noted a strong sense of eye contact in the context of conversation Yes/No “Are they looking at me?”

32. Virtual Viewing Distance Viewers sat 12’ from screen –projector throw distance Image needed to be scaled down to 66% –Scene:Screen size Resulting in a virtual viewing distance of 18’

33. Future Work New MultiView design iteration –8’ Screen –Short throw projectors for closer virtual distance (from 18’ to 8’) –True Life Size Images

34. Future Work: Shared Workspace Object Virtual Object Projector Glass Table

35. Future Work Using a new digital pipeline, we can control for aspects such as video quality and latency for low level experiments New series of higher level experiments using new screen design –Trust formation –Turn Taking –Speech Patterns

36. Conclusion Our perceptual experiments show that attention target cues can be accurately determined using our system ? Standard video conference systems have spatial distortions which can adversely effect communication Our solution, MultiView, is the first practical video conferencing system to support spatially faithful group-to-group meetings

39. Additional Slides

40. David Nguyen Professor John Canny MultiView A Spatially Faithful Video-Conferencing System

41. Position Dependant Views Participants are facing the right camera View From Left View From Center View From Right

42. Current Technologies

43. Functions of Gaze (Kendon, 1967) Monitoring –“Is my partner listening to me?” –“Does my partner want to take over?” –Planning, Current Control, Checking Regulating –Floor Control (Turn Taking) –Forestall/Demand a Response Expressing – feelings or attitudes –Power, interest, point-granting, attention, agreement

44. First Impressions: 7-38-55 Rule (Mehrabian, 1971) In forming first impressions… –7% is based on what you actually say –38% is based on the way you say it –55% is based on your appearance Suggests that a lot is at risk in forming first impressions if visual nonverbal cues are not appropriately preserved

45. Pop Quiz: First Impressions 7-38-55 Rule (Mehrabian, 1971) In forming first impressions… – is based on content – is based on voice – is based on appearance Fill in the blank with [7%, 38%, or 55%] 7% 38% 55%

46. The Video Tunnel Variants Include… Gaze-2 (Vertegaal et al., 1999, 2003) GA Display (Monk and Gale, 2002) The Teleprompter (Kahn, 1994) Clearboard (Ishii and Kobayashi, 1992) Reciprocal Video Tunnel (Buxton and Moran, 1990)

47. Computer Vision/Graphics Approach (Gemmel et al., 2000)

48. Human Accuity (Chen, 2002)

49. Next Design Iteration 8ft screen for life size images Short throw projector for closer viewing distances (8’wide @ 9’6”) essentially halving virtual distance Ghosting –Multiple elements and uncontrollable diffusion causes ghosting effects

50. Error by Viewing Position GAZE Positionμσ 10.700.65 30.630.67 50.600.70 All0.640.68 GESTURE PositionΜσ 10.550.61 30.530.60 50.650.67 All0.580.63 In standard video conferencing systems, Mona Lisa Effect would cause greater error as viewing position deviates from “virtual camera position” Finding: Viewing position had no significant effect on error MultiView does not exhibit the Mona Lisa Effect

51. 3 Site Setup

52. Vision: More Motion Freedom Three evolving technologies –Synthesize viewpoints from multiple cameras –Shrinking Projector Size and Power Consumption –Position Tracking Virtual Camera

53. Prior Work Hydra (Sellen et al., 1992) MAJIC (Okada et al., 1994) GAZE-2 (Vertigaal et al., 2003)

54. Feature Comparison

55. MultiView (Nguyen and Canny, 2005) A Spatially Faithful Video Conferencing System

56. Trust Formation (Bos et al., 2002) Trust development was delayed in audio/video Defections were more likely with video/audio than FTF communication. Little difference between video and audio

57. Errors By Stimulus Direction Edge Effects Extension of Chen’s Snap To Contact Theory –Includes other sources attention (not just gaze) –Includes third party viewers viewing two others in discussion 1 0.23 2 0.65 3 0.59 4 0.76 5 0.55 Mean Error By Gesture Direction Direction 1 0.28 2 0.79 3 0.68 4 0.73 5 0.43 Mean Error By Gaze Direction Direction

58. Acknowledgments