1. Image-Based Visual Hulls Wojciech Matusik Chris Buehler Leonard McMillan Wojciech Matusik Chris Buehler Leonard McMillan Massachusetts Institute of Technology Laboratory for Computer Science Ramesh Raskar Steven J. Gortler University of North Carolina at Chapel Hill Steven J. Gortler Harvard University

2. Motivation Real-time acquisition and rendering of dynamic scenes

3. Previous Work Virtualized Reality (Rander’97, Kanade’97, Narayanan’98) Virtualized Reality (Rander’97, Kanade’97, Narayanan’98) Visual Hull (Laurentini’94) Visual Hull (Laurentini’94) Volume Carving (Potmesil’87, Szeliski’93, Seitz’97) Volume Carving (Potmesil’87, Szeliski’93, Seitz’97) CSG Rendering (Goldfeather’86, Rappoport’97) CSG Rendering (Goldfeather’86, Rappoport’97) Image-Based Rendering (McMillan’95, Debevec’96, Debevec’98) Image-Based Rendering (McMillan’95, Debevec’96, Debevec’98)

4. Contributions View-dependent image-based visual hull representation View-dependent image-based visual hull representation Efficient algorithm for sampling the visual hull Efficient algorithm for sampling the visual hull Efficient algorithm computing visibility Efficient algorithm computing visibility A real-time system A real-time system

5. What is a Visual Hull?

6. Why use a Visual Hull? Can be computed robustly Can be computed robustly Can be computed efficiently Can be computed efficiently - =background+foregroundbackgroundforeground

7. Rendering Visual Hulls Reference 1 Reference 2 Desired

8. Build then Sample Reference 1 Reference 2 Desired

9. Build then Sample Reference 1 Reference 2 Desired

10. Build then Sample Reference 1 Reference 2 Desired

11. Build then Sample Reference 1 Reference 2 Desired

12. Build then Sample Reference 1 Reference 2 Desired

13. Sample Directly Reference 1 Reference 2 Desired

14. Sample Directly Reference 1 Reference 2 Desired

15. Sample Directly Reference 1 Reference 2 Desired

16. Sample Directly Reference 1 Reference 2 Desired

17. Sample Directly Reference 1 Reference 2 Desired

18. Sample Directly Reference 1 Reference 2 Desired

19. Sample Directly Reference 1 Reference 2 Desired

20. Sample Directly Reference 1 Reference 2 Desired

21. Direct Sampling Advantages Line interval intersections are robust Line interval intersections are robust Direct sampling gives us exact rendering Direct sampling gives us exact rendering Can be computed efficiently in image space Can be computed efficiently in image space

22. Image-Based Computation Reference 1 Reference 2 Desired

23. Observation Incremental computation along scanlines Incremental computation along scanlines Desired Reference

24. Binning Epipole Sort silhouette edges into bins Sort silhouette edges into bins

25. Binning Epipole Sort silhouette edges into bins Sort silhouette edges into bins

26. Binning Sort silhouette edges into bins Sort silhouette edges into bins Epipole Bin 1

27. Binning Sort silhouette edges into bins Sort silhouette edges into bins Epipole Bin 2 Bin 1

28. Binning Sort silhouette edges into bins Sort silhouette edges into bins Epipole Bin 3 Bin 1 Bin 2

29. Binning Sort silhouette edges into bins Sort silhouette edges into bins Epipole Bin 4 Bin 1 Bin 2 Bin 3

30. Binning Sort silhouette edges into bins Sort silhouette edges into bins Epipole Bin 5 Bin 1 Bin 2 Bin 3 Bin 4

31. Binning Sort silhouette edges into bins Sort silhouette edges into bins Epipole Bin 5 Bin 1 Bin 2 Bin 3 Bin 4

32. Scanning Epipole Bin 1

33. Epipole Bin 2 Scanning

34. Epipole Bin 2 Scanning

35. Epipole Bin 2 Scanning

36. Epipole Bin 4 Scanning

37. Epipole Bin 5 Scanning

38. Coarse-to-Fine Sampling

39. IBVH Results Approximately constant computation per pixel per camera Approximately constant computation per pixel per camera Parallelizes Parallelizes Consistent with input silhouettes Consistent with input silhouettes

40. Video of IBVH

41. Shading Algorithm A view-dependent strategy A view-dependent strategy

42. Visibility Algorithm

43. Visibility in 2D Desired view Reference view

44. Visibility in 2D Desired view Reference view Front-most Points

45. Visibility in 2D Desired view Reference view Visible

46. Visibility in 2D Desired view Reference view Coverage Mask

47. Visibility in 2D Desired view Reference view Coverage Mask Visible

48. Visibility in 2D Desired view Reference view Coverage Mask Visible

49. Visibility in 2D Desired view Reference view Coverage Mask VisibleNot

50. Visibility in 2D Desired view Reference view Coverage Mask

51. Visibility in 2D Desired view Reference view Coverage Mask Visible

52. Visibility in 2D Desired view Reference view Coverage Mask

53. Visibility in 2D Desired view Reference view Coverage Mask VisibleNot

54. Shaded Visual Hulls

55. System Server (4x 500 Mhz) Camera Client Camera Client Camera Client Camera Client

56. System Server (4x 500 Mhz) Camera Client Camera Client Camera Client Camera Client Trigger Signal

57. System Server (4x 500 Mhz) Camera Client Camera Client Camera Client Camera Client

58. System Server (4x 500 Mhz) Camera Client Camera Client Camera Client Camera Client Compressed video

59. System Server (4x 500 Mhz) Camera Client Camera Client Camera Client Camera Client Intersection

60. System Server (4x 500 Mhz) Camera Client Camera Client Camera Client Camera Client Visibility

61. System Server (4x 500 Mhz) Camera Client Camera Client Camera Client Camera Client Shading

62. More IBVH Results

63. Future Work 3D teleconferencing 3D teleconferencing Virtual sets Virtual sets Post-production camera effects Post-production camera effects Mixed reality Mixed reality

64. Summary Visual hulls with texture can provide a compelling real-time visualizations Visual hulls with texture can provide a compelling real-time visualizations Visual hulls can be computed accurately and efficiently in image space Visual hulls can be computed accurately and efficiently in image space View dependent shading with visibility View dependent shading with visibility

65. Acknowledgements DARPA ITO Grant F30602-971-0283 DARPA ITO Grant F30602-971-0283 A generous grant from Intel Corporation A generous grant from Intel Corporation NSF Career Awards 9875859 & 9703399 NSF Career Awards 9875859 & 9703399 Tom Buehler & Kari Anne Kjølass Tom Buehler & Kari Anne Kjølass Thanks to all members of the MIT Computer Graphics Group