1. Appearance modeling: textures and IBR Class 17

2. 3D photography course schedule Introduction Aug 24, 26(no course) Aug.31,Sep.2(no course) Sep. 7, 9(no course) Sep. 14, 16Projective GeometryCamera Model and Calibration (assignment 1) Feb. 21, 23Camera Calib. and SVMFeature matching (assignment 2) Feb. 28, 30Feature trackingEpipolar geometry (assignment 3) Oct. 5, 7Computing FTriangulation and MVG Oct. 12, 14(university day)(fall break) Oct. 19, 21StereoActive ranging Oct. 26, 28Structure from motionSfM and Self-calibration Nov. 2, 4Shape-from-silhouettesSpace carving Nov. 9, 113D modelingAppearance Modeling Nov.12 papers (2-3pm SN115) Nov. 16, 18(VMV’04) Nov. 23, 25papers & discussion(Thanksgiving) Nov.30,Dec.2papers & discussionpapers and discussion Dec.3 papers (2-3pm SN115) Dec. 7?Project presentations

3. Papers Li Exact Voxel Occupancy with Graph Cuts Sudipta Stereo without epipolar lines Chris A graph cut based adaptive structured light approach for real-time range acquisition Nathan Space-time faces Brian Depth-from-focus … Chad Interactive Modeling from Dense Color and Sparse Depth Seon Joo Outdoor calibration of active cameras Jason spectral partitioning Sriram Linear multi-view reconstruction Christine 3D photography using dual … http://www.unc.edu/courses/2004fall/comp/290b/089/papers/

4. Projects ChrisWide-area display reconstruction NathanStructured light BrianDepth-from-focus/defocus LiVisual-hulls with occlusions ChadLaser scanner for 3D environments Seon JooCollaborative 3D tracking JasonSfM for long sequences Sudipta Combining exact silhouettes and photoconsistency SriramPanoramic cameras self-calibration Christine desktop lamp scanner

5. Multiple depth imagesVolumetric integration Volumetric 3D integration

6. Appearance Modeling Texturing Single image Multiple image Image-based rendering (Unstructured) lightfield rendering Surface lightfields

7. Texture mapping 3D model Need to estimate relative pose between camera and 3D model

8. Texture Mapping Conventional texture-mapping with texture coordinates Projective texture-mapping

9. Texture Map Synthesis I Conventional Texture- Mapping with Texture Coordinates Create a triangular texture patch for each triangle The texture patch is a weighted average of the image patches from multiple photographs Pixels that are close to image boundaries or viewed from a grazing angle obtain smaller weights Photograph Texture Map 3D Triangle

10. Texture Map Synthesis II Allocate space for texture patches from texture maps Generalization of memory allocation to 2D Quantize edge length to a power of 2 Sort texture patches into decreasing order and use First-Fit strategy to allocate space First-Fit

11. A Texture Map Packed with Triangular Texture Patches

12. Appearance Modeling texture atlas

13. Dealing with auto-exposure Photometric alignment of textures (or HDR textures) (Kim and Pollefeys, CVPR’04)

14. Image as texture Depth image Triangle mesh Texture image Textured 3D Wireframe model Affine vs. projective texture mapping (see later)

15. Lightfield literature Plenoptic function Lightfield (plane) and Lumigraph (some geometry) Unstructered lightfield (some (view-dependent) geometry) Surface lightfields (full geometry) Plenoptic sampling (trade-off geometry vs. images) (Levoy&Hanrahan,Siggraph´96 Gortler et al.,Siggraph´96) (Koch et al. ICCV´99; Heigl et al. DAGM´99; Buehler et al. Siggraph‘01) (Chai et al.,Siggraph´00) (Wood et al.,Siggraph´00, Chen et al., Siggraph‘02) (Adelson&Bergen´91; McMillan&Bishop,Siggraph´95)

16. Lightfield rendering focal surface Approximate light rays by interpolating from closest light rays in lightfield viewpoint surface Projection of viewpoint surface in virtual camera determines which views to get lightrays from Transfer from images to virtual views over focal surface determines which pixels to use

17. Unstructured lightfield rendering original viewpoints Novel view For every pixel, combine best rays from closest views (Koch et al.,ICCV´99; Heigl et al.,DAGM´99) Focal surface demo

18. Example: desk sequence 186 images recorded with hand-held camera

19. Example: desk sequence structure and motion depth images 190 images 7000points

20. Example: Desk Lightfield Planar focal surface (shadow artefacts)

21. View-dependent geometry approximation original viewpoints object surface View-dependent surface approximation Novel view depth maps

22. Adaptation of geometry with the rendering viewpoint View-dependent geometry approximation

23. Geometry subdivision original viewpoints object surface View-dependent surface approximation Novel view depth maps Note: Only necessary when depth value significantly deviates from previous approximation deviates from previous approximation

24. Viewpoint-geometry without subdivision 4 subdivisions 2 subdivisions 1 subdivision of viewpoint surface Scalable geometric approximation

25. Example: Desk lightfield Planar focal surface View-dependent geometry approximation (2 subdivisions)

26. Hardware accelerated rendering Use blending operation similar to Gouraud shading Use projective textures!

27. Demo demo

28. Extrapolation (Buehler et al., Siggraph´01) Add mesh to cover whole image (compute non-binary blending weights) Rendered image Blending field (courtesy Leonard McMillan)

29. Surface Lightfields Surface location Viewing direction Surface light field (SLF) function Chen et al., Siggraph 2002, "Light Field Mapping: Efficient Representation and Hardware Rendering of Surface Light Fields""Light Field Mapping: Efficient Representation and Hardware Rendering of Surface Light Fields" R. Grzeszczuk, Presentation on Light Field Mapping, SIGGRAPH 2002 Course Notes for Course “Image-based Modeling.”Presentation on Light Field Mapping http://www.intel.com/research/mrl/research/lfm/

30. Surface Lightfields Partition SLF across surface primitives Pi Approximate SLF for each Pi individually as Surface light field (SLF) function Light field maps: stored as 2D texture maps Surface maps View maps

31. Light Field Mapping Data Acquisition Resampling Partitioning Rendering Approximation Compression

32. Light Field Mapping Data Acquisition Resampling Partitioning Rendering Approximation Compression

33. 200-400 images captured by hand- held camera Geometry scanned with structured lighting Images registered to geometry

34. Light Field Mapping Data Acquisition Resampling Partitioning Rendering Approximation Compression

35. Partitioning Partitioning the light field data across small surface primitives Individual parts add up to original SLF Ensure continuous approximations across neighbouring surface elementsTriangle-centered: split the light field between individual triangles

36. Partitioning Triangle-centered: split the light field between individual triangles ->discontinuity  ->discontinuity  Partitioning the light field data across small surface primitives Individual parts add up to original SLF Ensure continuous approximations across neighbouring surface elements

37. Vertex-centered Partitioning Partition surface light field data around every vertex Hat function =

38. Vertex-centered Partitioning

39. Define local reference frame of the vertex Reparameterize each vertex light field to its local coordinate system Vertex light field

40. Light Field Mapping Data Acquisition Resampling Partitioning Rendering Approximation Compression

41. Resampling Goal: Generate vertex light field function Visibility computation determines unoccluded views for each triangle ring 2 steps: Normalization of texture size Resampling of viewing directions

42. Resampling Each column represents a different view 1 st view 2 nd view C i -th view

43. Resampling 1. Normalization of texture size Each texture patch has the same shape and size Bilinear interpolation 2. Resampling of viewing directions

44. Resampling 1. Normalization of texture size 2. Resampling of viewing directions Projection of original views 1 2 3 4 ……. c ….. C i 1 2 ……. m M

45. Resampling 1. Normalization of texture size 2. Resampling of viewing directions Delaunay triangulation Uniform grid of views M 1 2 3 4 ……. c ….. C i 1 2 ……. m

46. Resampling 1. Normalization of texture size 2. Resampling of viewing directions 1 2 3 4 ……. n ….. N 1 2 ……. m M

47. Light Field Mapping Data Acquisition Resampling Partitioning Rendering Approximation Compression

48. Decomposition & Approximation Rearrange 4-dimensional F into M*N matrix Decompose F using matrix factorization Truncate the sum after K terms N K<<N K Surface maps View maps

49. Decomposition & Approximation Split surface maps for triangle ring into surface maps for individual triangles 3 surface maps for each approximation term of each triangle

50. Decomposition & Approximation 1 st approximation 2 nd approximation K th approximation …. Each approximation =3 surface maps + 3 view maps f

51. Approximation methods PCA (principal component analysis) Progressive Arbitrary sign factors NMF (non-negative matrix factorization) Parts-based representation Non-negative factors Easier and faster rendering

52. Light Field Mapping Data Acquisition Resampling Partitioning Rendering Approximation Compression

53. Light Field Mapping Tiled surface maps Tiled view maps Light field maps are redundant Very high compression ratio (10000:1)

54. Light Field Mapping Data Acquisition Resampling Partitioning Rendering Approximation Compression

55. Rendering 1 st approximation 2 nd approximation K th approximation …. Each approximation =3 surface maps + 3 view maps f

56. Rendering Surface map: view-independent View map: Establish vertex coordinate system Project viewing vector onto view map hemisphere

57. Results Bust Star Turtle Buddha Horse