1. Jan. 19, 1999 CS260 Winter 1999-Wittenbrink, lect. 5 1 CS 260 Computer Graphics Craig M. Wittenbrink Lecture 5: Image Based Rendering Techniques: Shade et al., Wittenbrink et al.

2. CS260 Winter 1999-Wittenbrink, lect. 5 2 Jan. 19, 1999 Overview Intro to Warping Shade et al. –Sprites with depth –Layered depth images SIGGRAPH’98 Videos Wittenbrink et al. –Karhunen-Loeve Transform –2D texture mapping from satellite images Conclusions

3. CS260 Winter 1999-Wittenbrink, lect. 5 3 Jan. 19, 1999 Intro to Warping Geometric Transformations –Castelman, Chapter 8, page 115 –Gonzalez and Woods, Section 5.9, page 296 –G. Wolberg. Digital Image Warping, Computer Society Press, 1988. –Wittenbrink and Somani, 2D&3D Optimal Parallel Image Warping Two aspects: –1) Algorithm that defines spatial transformation for image pixels –2) Algorithm for gray-level interpolation

4. CS260 Winter 1999-Wittenbrink, lect. 5 4 Jan. 19, 1999 Warping cont. Spatial transformation examples Rotation(counter) Translation Scaling (shrink) –(expand) Homogeneous coordinate

5. CS260 Winter 1999-Wittenbrink, lect. 5 5 Jan. 19, 1999 Warping cont. Numerous methods to calculate include: –Forwards (Pixel Carryover)[splatting] –Backwards (Pixel Filling)[ray casting] Backwards is often more practical T T -1 Input image Output image T T -1 Interpolate from surrounding pixels

6. CS260 Winter 1999-Wittenbrink, lect. 5 6 Jan. 19, 1999 Warping/Interpolation Types of interpolation 1) zero-order, or nearest neighbor 2) first-order, or bilinear Bilinear term Zero-order hold bilinear quadratic cubic Zero-order hold Bilinear

7. CS260 Winter 1999-Wittenbrink, lect. 5 7 Jan. 19, 1999 Shade et al. Goal is to create parallax and remove disocclusions Sprites with depth –Improve over just affine warps Layered-Depth Images (LDI) –faster than Max

8. CS260 Winter 1999-Wittenbrink, lect. 5 8 Jan. 19, 1999 Shade et al./ Texture mapping as IBR? Texture mapping is the most familiar image- based rendering method Two main challenges with texture mapping –1) sample rate of texture and scene are vastly different –2) texture mapping is compute and bandwidth intensive Implications –1) Antialiasing for textures is difficult –2) Texture mapping is key focus for hardware 3D graphics design

9. CS260 Winter 1999-Wittenbrink, lect. 5 9 Jan. 19, 1999 Shade et al. / Big Picture of IBR Grab bag of Image-based rendering primitives (from back to front, see Fig. 1) –1) Environment map –2) Planar sprites –3) Sprites with depth –4) Layered-depth image (LDI) –5) Polygons Camera Center for LDI and Sprites with depth Viewing Region

10. CS260 Winter 1999-Wittenbrink, lect. 5 10 Jan. 19, 1999 Shade et al. / Epipolar Geometry Output camera and input camera define epipolar point Back-to-front ordering defined Epipolar Point Layered Depth Image Camera Output Camera

11. CS260 Winter 1999-Wittenbrink, lect. 5 11 Jan. 19, 1999 Shade et al. / Sprites with Depth, Transformations 3D Plane equation computed for sprite Z d1 is scaled perpendicular distance to plane Add a displacement per pixel of sprite, resulting transfer function transform sprite coord to output image coord Note: d1 for z Eq 2 Eq 4

12. CS260 Winter 1999-Wittenbrink, lect. 5 12 Jan. 19, 1999 Sprite with depth, cont. transformation from sprite to image d1’s - depth with sprite H1,2 - 2D planar perspective transformation (x2,y2) - coordinate in output camera image (x1,y1) - sprite’s coordinates e1,2 - epipole (from 3rd column of T1,2) Layered Depth Image Camera Output Camera sprite

13. CS260 Winter 1999-Wittenbrink, lect. 5 13 Jan. 19, 1999 Forward Mapping/Splatting Eq 4, forward map sprite pixels to output image Or, Interchange 1 & 2 indices –both cameras are “views” and epipole is the same Now forward map displacements, and use EQ 4 with interchanged indices to do backward mapping with new view based displacements

14. CS260 Winter 1999-Wittenbrink, lect. 5 14 Jan. 19, 1999 Factoring warping equation Create intermediate space between d1 (sprite) and d2 (output image) or d3 d1’s, sprite d3’s forward warped d2’s backward warp C2, backwards warp for colors (EQ 4 reversed) (EQ6) (EQ5) Output image Sprite Step 1, forward map Step 2a, backward map Step 2b, backward map

15. CS260 Winter 1999-Wittenbrink, lect. 5 15 Jan. 19, 1999 Factoring warping equation More efficient Shortcut d3 conversion to d2 And, can be even faster by skipping d1 to d3 transform d1’s, sprite d3’s forward warped d2’s backward warp (EQ6) (EQ5) Output image Sprite (x2,y2) shortcut Even faster (approximate)

16. CS260 Winter 1999-Wittenbrink, lect. 5 16 Jan. 19, 1999 Warping by lookup table Can precompute warp factor and store in lookup table Eq 9

17. CS260 Winter 1999-Wittenbrink, lect. 5 17 Jan. 19, 1999 Sprites with depth results No parallax Incorrect using d1 Correct d3 used Better gap filling Less slanted pyramid Perspective projection Pure parallax Both parallax and perspective Original color and depth Reproduced from Shade et al. Copyright SIGGRAPH’98

18. CS260 Winter 1999-Wittenbrink, lect. 5 18 Jan. 19, 1999 Extracted Sprites with Depth Interactive 3DGraphics Symposium 5 images used Fig. 4 reproduced from Shade et al. Copyright SIGGRAPH’98 and Dayton Taylor 1 of 5 images 6 layer segmentation Recovered depth map 5 layer sprites Residual depth image for fifth layer Resynthesized 3rd image 2 novel views

19. CS260 Winter 1999-Wittenbrink, lect. 5 19 Jan. 19, 1999 Extracted Sprites Novel view with residual depth Fig 4 h Fig. 4 reproduced from Shade et al. Copyright SIGGRAPH’98 and Dayton Taylor

20. CS260 Winter 1999-Wittenbrink, lect. 5 20 Jan. 19, 1999 Layered Depth Images (LDI’s) Attempt to handle more disocclusions and large amounts of parallax contains potentially multiple depth pixels per pixel location Farther pixels help to fill holes (disocclusions) Use linked during construction, and packed for rendering Fig. 5 reproduced from Shade et al. Copyright SIGGRAPH ‘98

21. CS260 Winter 1999-Wittenbrink, lect. 5 21 Jan. 19, 1999 Creation of LDI’s 1a)Use synthetic ray tracer that provides depth per pixel 1b) Or scan conversion and read z-buffer –Choose one camera position as LDI camera and warp images to that camera 2)Use Less regular sampling with ray tracer –which rays to choose? 3)Or use Computer Vision from multiple images –Modified Seitz and Dyer algorithm, view centered voxelization

22. CS260 Winter 1999-Wittenbrink, lect. 5 22 Jan. 19, 1999 Reconstruction Common events: 1) disocclusions as viewpoint changes 2) surfaces that cover large areas of the screen Define an LDI from each cube face (they don’t appear to actually do this) Cube of possible new viewpoints

23. CS260 Winter 1999-Wittenbrink, lect. 5 23 Jan. 19, 1999 Creation with ray caster cont. Parametrize the rays Use cosine weighted direction over the hemisphere use Stratified Stochastic sampling divide uniformly into NxNxNxN strata for each stratum cast m rays N=32, m=16, gives 32^4*16=16million

24. CS260 Winter 1999-Wittenbrink, lect. 5 24 Jan. 19, 1999 Sampling with ray caster Main point, get lots of rays to cover many viewpoints Rays are in all directions

25. CS260 Winter 1999-Wittenbrink, lect. 5 25 Jan. 19, 1999 LDI from real images (not that you could tell ;) ) Seitz and Dyer dinosaur toy

26. CS260 Winter 1999-Wittenbrink, lect. 5 26 Jan. 19, 1999 Rendering Layered Depth Images Splatting used Space efficient representation –pack LDI, bottom-to-top, left-to-right in screen space, and back-to-front along ray –Store offsets for fast access –1) beginning of scanline –2) pixel in scanline

27. CS260 Winter 1999-Wittenbrink, lect. 5 27 Jan. 19, 1999 Incremental Warping Computation Given 4x4 matrix for LDI view and 4x4 matrix for desired new view You can transform a point in the LDI view to the world coordinates, and then to the new view C1 C2 a Aworld a2

28. CS260 Winter 1999-Wittenbrink, lect. 5 28 Jan. 19, 1999 Incremental Warping Reuse matrix results by factoring New start is simply incremented

29. CS260 Winter 1999-Wittenbrink, lect. 5 29 Jan. 19, 1999 Rendering of LDI For each pixel –For number of layers –result=start +z1*depth (location) –clip either behind camera or out of frustum –splat (pick appropriate splat size) –increment for next pixel on scanline Splat size chosen by projected pixel area approximiation –Put approximations in lookup table

30. CS260 Winter 1999-Wittenbrink, lect. 5 30 Jan. 19, 1999 LDI Rendering Results LDI’s max 10 layers per pixel, 1.24 average depth complexity 300x300 resolution, at 8-10 frames/second on Pentium II, 300 MHz

31. CS260 Winter 1999-Wittenbrink, lect. 5 31 Jan. 19, 1999 LDI Rendering Results Cross eyed stereo pairs. LDI’s from Rayshade raytracer. LDI has 1.1 million depth pixels. 4-10 frames/second on Pentium II @300MHz

32. CS260 Winter 1999-Wittenbrink, lect. 5 32 Jan. 19, 1999 LDI Rendering Results Cross eyed stereo pairs. LDI’s from Rayshade raytracer. LDI has 1.1 million depth pixels. 4-10 frames/second on Pentium II @300MHz

33. CS260 Winter 1999-Wittenbrink, lect. 5 33 Jan. 19, 1999 SIGGRAPH’98 Videos Image Based Rendering (24 min.) 9: The Office of the Future: …Raskar, Fuchs et al. 10: Rendering Synthetic Objects Into Real Scenes: … Debevec 11: Multiple-Center-of-Projection Images, Rademacher and Bishop 12: Recovering Photometric Properties of Architectural Scenes from Photographs, Uy and Malik

34. CS260 Winter 1999-Wittenbrink, lect. 5 34 Jan. 19, 1999 SIGGRAPH’98 Videos Image Based Rendering (cont.) 13: Visibility Sorting and Compositing Without Splitting for Image Layer Decomposition, Snyder and Lengyel 14: Layered Depth Images, Shad, Gortler, He, and Szeliski

35. CS260 Winter 1999-Wittenbrink, lect. 5 35 Jan. 19, 1999 Wittenbrink et al. SPIE’96 35 mm slides Feature extraction of clouds from GOES satellite data for integrated model measurement visualization, Craig M. Wittenbrink, Glen Langdon, Jr., and Gabriel Fernandez, In Proceedings of IS&T/SPIE Symposium on Electronic Imaging: Image and Video Processing IV 1996, Vol. 2666, pages 212-222, R. Stevenson and M.I. Sezan, San Jose, CA 1996. www.cse.ucsc.edu/research/slvg/cloud.html

36. CS260 Winter 1999-Wittenbrink, lect. 5 36 Jan. 19, 1999 Conclusions Shade et al. –Sprites with depth –Layered depth images SIGGRAPH’98 Videos: Wittenbrink et al. –Karhunen-Loeve Transform –2D texture mapping from satellite images Next time: Seitz and Dyer “Photorealistic Scene Reconstruction by Voxel Coloring” CVPR’97 Conclusions