1. Plenoptic Modeling: An Image-Based Rendering System Leonard McMillan & Gary Bishop SIGGRAPH 1995 presented by Dave Edwards 10/12/2000

2. Introduction Two Kinds of Rendering –Geometric Requires increased complexity to achieve realism Solid foundation & physical models –Image-based Photorealism is automatic

3. Problem & Solution Problem: –No consistent framework for evaluating image- based rendering techniques Solution: –Use the plenoptic function for evaluation

4. Definition of Image-Based Rendering From the paper: –“Given a set of discrete samples... from the plenoptic function, the goal of image-based rendering is to generate a continuous representation of that function.”

5. Plenoptic Modeling IBR System based on earlier definition Four main steps: –Representation –Acquisition –Determination of flow fields –Reconstruction

6. Plenoptic Sample Representation Spherical projection –Intuitive, but awkward to store Cubic projection –Easy to store, but mapping is non-uniform Cylindrical projection –Easy to acquire & store –Drawback: boundaries limit possible views

7. Image Acquisition Take a series of images by rotating a camera about a fixed axis –A portion of each image must overlap the previous image These images are related to each other by a homogenous transform: I i+1 = H i I i

8. Image Acquisition The matrix H i can be expressed as S -1 R i S –S is intrinsic to the camera –R i represents rotation between images Given our image set, we can find S & R i –This allows us to move all the images into a single reference frame

9. Image Acquisition To find R i : –R i depends only on rotation angle –Rotation by a small angle is like a translation for pixels near the center of the image –Compute optimal translation t i between I i+1 and I i –  i = arctan( t i / f ) –since   i = 2 , use Newton’s method to find f

10. Image Acquisition To find S: –S is based on 6 parameters: (C x, C y ) - pixel coordinate of optical axis  x,  z - rotation about x- and z-axes ,  - skew and aspect ratio –Find S by minimizing sum of errors between I i+1 and S -1 R i SI i for the middle third of each image

11. Image Acquisition We can now move all images into the same frame of reference: –I j = S -1 R j-1 R j-2 …R 1 R 0 SI 0 Combining all the images in this way gives us our cylindrical projection –We can store this as a planar image:

12. Flow Field Determination The next step is to find disparity values between two projections For each cylinder, we want to find: –(C x, C y, C z ) - the global coordinates of the center of the cylinder –  - the angle offset between each cylinder and a common reference angle –k - scale factor for vertical field of view –C v - central horizontal scanline (“equator”)

13. Flow Field Determination We can find these values using tiepoints –user-specified points common to both images Each tiepoint lies at the intersection of 2 vectors, one from the center of each cylinder: –p i = C 1 + sD 1 = C 2 + tD 2

14. Flow Field Determination In general, these vectors don’t intersect –There is some distance between them Minimize the sum of these distances for the set of all tiepoints –This yields an estimate for the 6 cylinder parameters mentioned earlier

15. Flow Field Determination To find the disparity between other points, use epipolar relationships –On a cylinder, epipolar curves are sinusoids: –Curves are given by v(  ) = a + bsin(  ) + ccos(  ) For each point on one reference cylinder, store the angular disparity α with the corresponding point on the second cylinder –Disparity vector is then (  + , v(  +  ))

16. Sample Reconstruction A P B V   

17. Sample Reconstruction We know  at each point on the original cylinder We can calculate  to generate a projection about some new cylinder V We can then map pixels in V into a planar map to recover an image from a new viewpoint

18. Sample Reconstruction To produce correct visibility: –Draw the points closest to the new viewing position last To fill holes: –use the color of the nearest pixel in the new projection

19. Results Sample Images: More at: http://graphics.lcs.mit.edu/~mcmillan/Publications/plenoptic.html (includes movie)

20. Conclusion IBR definition is intuitive and accurate Modeling system –Pros Models visibility and perspective correctly Suitable for interactive use –Cons Still frames have noticeable aberrations Hole-filling is obvious in interactive use