1. The UNIVERSITY of NORTH CAROLINA at CHAPEL HILL An Incremental Weighted Least Squares Approach To Surface Light Fields Greg Coombe Anselmo Lastra

2. The UNIVERSITY of NORTH CAROLINA at CHAPEL HILL 2 Image-Based Rendering Generate new views of a scene from existing views Sample appearance from physical world Lightfields [Levoy96], Lumigraphs [Gortler96] …

3. The UNIVERSITY of NORTH CAROLINA at CHAPEL HILL 3 Surface Lightfields A surface light field [Wood00] represents the appearance of a model with known geometry and static lighting viewing direction surface position (u,v) (θ,Φ)(θ,Φ)

4. The UNIVERSITY of NORTH CAROLINA at CHAPEL HILL 4 Surface Lightfields f 1 (Θ, Φ)f 2 (Θ, Φ) f 3 (Θ, Φ) f 4 (Θ, Φ)

5. The UNIVERSITY of NORTH CAROLINA at CHAPEL HILL 5 SLF - Batch Process Surface Lightfield Construction Renderer … [Chen02, Hillesland03] disk

6. The UNIVERSITY of NORTH CAROLINA at CHAPEL HILL 6 SLF - Online Process Incremental Surface Lightfield Construction Renderer 1024x768 @ 15fps [Coombe05]

7. The UNIVERSITY of NORTH CAROLINA at CHAPEL HILL 7 Scattered Data Approximation [Coombe05] required resampling to grid Samples are at arbitrary locations in domain due to geometry and camera ? ?

8. The UNIVERSITY of NORTH CAROLINA at CHAPEL HILL 8 Scattered Data Approximation Scattered Data Approximation in lightfields Unstructured Lightfields [Buehler01] Tesselation of pure lightfield Polynomial Texture Maps [Malzbender01] Fit polynomials to set of images Radial Basis Functions [Zickler05] Interpolate sparse reflectance data

9. The UNIVERSITY of NORTH CAROLINA at CHAPEL HILL 9 Outline 1.A 2D scattered data approximation 2.Fast incremental construction 3.Realtime evaluation

10. The UNIVERSITY of NORTH CAROLINA at CHAPEL HILL 10 Our Approach Represent the surface lightfield using Weighted Least Squares approximation Modify WLS for the incremental framework Adaptive and Hierarchical CPU/GPU implementation Process each image in about 1-2 seconds Real-time rendering

11. The UNIVERSITY of NORTH CAROLINA at CHAPEL HILL 11 Bust model, 75 training images

12. The UNIVERSITY of NORTH CAROLINA at CHAPEL HILL 12 Least Squares Fitting Find the “best” polynomial approximation to the input samples “best” means minimizes sum of squared differences the coefficients are determined by solving a linear system input samples reconstruction

13. The UNIVERSITY of NORTH CAROLINA at CHAPEL HILL 13 Least Squares domain reconstructed function input samples

14. The UNIVERSITY of NORTH CAROLINA at CHAPEL HILL 14 Weighted Least Squares Problem: LS is a global approximation Solution: Divide domain into multiple LS approximations, and combine to get global approximation Use a set of low-degree polynomials Non-linear blending (Partition of Unity) Good discussion in Scattered Data Approximation, Holgar Wendland

15. The UNIVERSITY of NORTH CAROLINA at CHAPEL HILL 15 Weighted Least Squares centers domains polynomial approximations reconstructed function input samples

16. The UNIVERSITY of NORTH CAROLINA at CHAPEL HILL 16 Weighted Least Squares x x x x Θ Φ

17. The UNIVERSITY of NORTH CAROLINA at CHAPEL HILL 17 Outline 1.A 2D scattered data approximation 2.Fast incremental construction 3.Realtime evaluation

18. The UNIVERSITY of NORTH CAROLINA at CHAPEL HILL 18 Incremental WLS Feedback is important in SLF construction As each image is captured, it must be incorporated into the representation How do we determine the size of each domain?

19. The UNIVERSITY of NORTH CAROLINA at CHAPEL HILL 19 Adaptive Construction x x x x x x x x Start out with large domains Adaptive shrink as more points arrive

20. The UNIVERSITY of NORTH CAROLINA at CHAPEL HILL 20 Hierarchical Construction x x xx x x x x x xx Start out with a single domain Subdivide as more points arrive (quadtree)

21. The UNIVERSITY of NORTH CAROLINA at CHAPEL HILL 21 Hierarchical Construction, First 10 images

22. The UNIVERSITY of NORTH CAROLINA at CHAPEL HILL 22 Hierarchical and Adaptive Hierarchical 1x1 Polynomial Texture Mapping Fast at first, slows down as refines # of domains is a power of 2 Adaptive Slow at first, speeds up as domains shrink Can handle arbitrary # of domains

23. The UNIVERSITY of NORTH CAROLINA at CHAPEL HILL 23 Outline 1.A 2D scattered data approximation 2.Fast incremental construction 3.Realtime evaluation

24. The UNIVERSITY of NORTH CAROLINA at CHAPEL HILL 24 Implementation Pose Estimation Visibility / Reprojection Incremental Weighted Least Squares 1024x768 @ 15fps GPU Renderer

25. The UNIVERSITY of NORTH CAROLINA at CHAPEL HILL 25 Φ Θ GPU Implementation Φ Θ Φ Θ Φ Θ

26. The UNIVERSITY of NORTH CAROLINA at CHAPEL HILL 26 Results 29K patches 4K patches side by side, 14K patches

27. The UNIVERSITY of NORTH CAROLINA at CHAPEL HILL 27 Pitcher model, 65 images

28. The UNIVERSITY of NORTH CAROLINA at CHAPEL HILL 28 Performance ~ 0.5 - 2 seconds per image for hierarchical construction 0.5s for 4K bust model 2s for 30K pitcher model 95% is Least Squares Fitting Adaptive is 2-3x more expensive Rendering is 60fps or more

29. The UNIVERSITY of NORTH CAROLINA at CHAPEL HILL 29 Conclusion Represent the surface lightfield using Weighted Least Squares approximation Modify WLS for the incremental framework Adaptive and Hierarchical CPU/GPU implementation Process each image in about 1-2 seconds Real-time rendering

30. The UNIVERSITY of NORTH CAROLINA at CHAPEL HILL 30 Future Work Since construction is dominated by LS fitting, implement on GPU Extend to surface reflectance (4D) Change basis functions Order-of-magnitude more data

31. The UNIVERSITY of NORTH CAROLINA at CHAPEL HILL 31 Thanks NVIDIA Graduate Student Fellowship UNC Graphics Group National Science Foundation Questions?