1. The “Occlusion Problem” Timothy S. Milliron CS 598d, Princeton University

2. What is the “Occlusion Problem?” zBetween two images, occlusion relationships might change unpredictably yBackground might be exposed yHoles in surfaces might result. zIBR needs more information to avoid artifacts zChen & Williams: Associate a depth-image with each RGB image.

3. Another Approach zUse depth information to generate something more like a traditional mesh zPerform a 3-D warp on the texture-mapped mesh (which looks just like the image). zTraditional Z-buffering will (almost) solve the occlusion problem.

4. Mark, McMillan, & Bishop zApplication: yIncrease frame-rates from 5 FPS to 60 FPS for expensive scenes yObtain good remote display performance zApproach: yRender a few frames traditionally (“reference” frames) per second, interpolate the rest.

5. Solving the Occlusion Problem zTreat the image as a texture-mapped triangular mesh with depth encoded in the Z coordinate. zTo interpolate, perform a standard 3-D warp using triangulation and geometry zDepth-buffering should suffice to solve occlusion and ensure no holes in surfaces zBut...

6. Rubber Sheets zAt object edges, triangles can be perpendicular to image plane, between foreground and background. yCause problems for occlusion zSolution: Tag triangles as “connected:” connected triangles are part of a surface yComplicated calculation zOkay, but …

7. Undersampling zIf a surface is oblique to the camera in one reference frame, and the derived frame shows the surface head-on, the surface is “undersampled” yHoles in the surface result zSolution: “confidence” value for each pixel yMeasures how well a given view can “see” a surface

8. View-Based Rendering zApplication: yViewing an arbitrary real-world object in 3-D from a concentric sphere around the object. xSpherical lumigraph: zApproach: yUse range-images to generate view-dependent triangular meshes to approximate the surface. yTexture-map the surface with the acquired image yComposite multiple views to form novel viewpoints and solve occlusion.

9. “Soft” Z-Buffering zTo composite the meshes: yIf the z-values are not within some threshold, use the closest yIf they are, blend them, using three weights

10. Weights (1) zFirst Weight: “Distance” from reference image (like Gouraud triangle interpolation) (corrects for image distortion)

11. Weights (2) zObliqueness to camera (corrects for holes) yPrefer “head-on” images to “oblique” images

12. Weights (3) zDistance from mesh boundary (less confident at the boundary) (corrects for polygonal error)

13. Summary zAll IBR technologies that incorporate translation must solve the occlusion problem. zThere are various approaches yPurely image-based (RGB and Z-buffer) yPartially geometry-reconstructing (meshes and 3-D warps) zSpecial considerations yUsing existing graphics hardware.