1. Image Based Rendering(IBR) Jiao-ying Shi State Key laboratory of Computer Aided Design and Graphics Zhejiang University, Hangzhou, China jyshi@cad.zju.edu.cn http://www.cad.zju.edu.cn/home/jyshi

2. Survey on Image Based Rendering (IBR) PART I

3. Traditional Computer Graphics 4 Use Geometry and lighting model to simulate the imaging process and generate realistic scene –No Guarantees for the rightness of the models –A lot of computation time needed

4. Use of images In Computer Graphics 4 Texture Mapping 4 Environment map How about more images?

5. Computer vision Extract Geometry model from real scene(photos) Combined with Computer Graphics: Image based rendering Bypass the “model”,driectly from real image to synthesized image

6. Image Based Rendering

7. A Framework of Image Based Rendering

8. The Key Part of IBR The data representation system is the key part of IBR, It determines the other three subsystems. - A taxonomy based on the data representation system

9. A Taxonomy of IBR 4 The Geometry based data representation 4 The Image based data representation 4 The plenoptic function based data representation

10. The Geometry based data representation 4 Geometry elements used as data representation in IBR: –polyhedra(Debevec, et. al 1996) –layers (Baker, Szeliski and Anandan 1998) –points(Shade et al. 1998) 4 Similar to Traditional Computer Graphics, except the geometry model comes from images

11. General working process

12. Image based data representation 4 data are treated as a series of images with correspondence relations 4 “optical flow” ”morphing map” “Trifocal/Trilinear tensor” are used to control the generation of novel image 4 forward/ reverse mapping;morphing Examples:. View interpolation (Chen and William,1993). View Morphing(Seitz and Dyer 1998)

13. General working process

14. Plenoptic function based data representation 4 Plenoptic function (Adelson and Bergen,1991)

15. General working process

16. Representative IBR methods based on Plenoptic Functions 4 Plenoptic Modeling: 5D 4 Light field/Lumigraph: 4D 4 Concentric Mosaics : 3D 4 Panorama: 2D

18. Conclusion 4 The progress of IBR technique is also the progress of new data representation method, We treat an image: –as texture in geometry  texture mapping –as images with correspondence relation  view interpolation /morphing –as light beams  light field –as slit image  concentric mosaics...

19. The study on slit images in image based rendering PART II

20. The concept of slit images The slit image is a kind of 1-D image with width only 1 pixel. An example of slit image

21. Previous work based on slit images 4 MCOP images 4 concentric mosaics

22. The advantage of using slit images 4 Most computer graphics technology is used to simulate human motion and observing usually only in 3 DOF: 4 The walk through task in virtual reality applications requires human motion only in 3 DOF: Left/right, forward/backward and look around.

23. The representation of slit images A slit image is identified by the camera 2D position and orientation (azimuth angle) in polar coordinates in Cartesian coordinates S(x,y,θ)

24. Slit image sets(I) 4 A scene view at position (ρ v, φ v ), with azimuth θ v and horizontal FOV ω: S v = 4 Panorama at position (ρ p, φ p ) S p =

25. Slit image sets(II) 4 Concentric mosaic with its center at origin ： S c ={S(ρ, φ,θ)|θ=-π/2 orθ=π/2 ， ρ≤R} –(camera alone normal direction) S cn ={S(ρ, φ,θ)|-ω /2<θ< ω/2 ， ρ=R} –(camera alone tangential direction) S ct ={S(ρ, φ,θ)|-ω /2 ＋ π/2 <θ< ω/2 ＋ π/2 ， ρ=R} 4 moving straight forward from origin, with horizontal FOV ω {S(x ， y,θ)|y=0, x>0, -ω /2<θ< ω/2 }

26. Slit image field 4 Slit images that captured at any position and any azimuth inside a 2D region. –Inside a circle: {S(ρ, φ ， θ)|ρ≤R} –Inside a rectangle: {S(x, y,θ)| x 1 ≤x≤ x 2, y 1 ≤y≤ y 2 } From the slit image field we can generate the walk-through scenes inside th region just by resampling

27. Analogical Slit Images

28. Relations between analogical slit images Let h d =| y d | ， h r =| y r | or

29. and let and  –analogical slit images are highly coherent –slit images can be synthesized by their analogical slit image Relations between analogical Slit Images

30. Analogical relation of slit images 4 Analogical relation of slit images is –reflexive S 1 ~S 1 –symmetric if S 1 ~S 2 ， there will be S 2 ~S 1 –transitive if S 1 ~S 2 、 S 2 ~S 3 ， there will be S 1 ~S 3 Written as S 1 ~S 2 So analogical relation of slit image is an equivalence relation

31. Analogical slit image set 4 Slit images that are analogical each other are consisted to be a analogical slit image set. 4 Analogical relation is a kind of equivalence relation  an analogical slit image set is a partition of slit image field

32. A slit image field can be obtained approximately by limited sampling 4 Each analogical slit image set can be approximated by one or a few its member silt images 4 The set of slit image sets can cover the slit image field. 4 A slit image field can be approximated by limited sampling

33. Depth correction for Concentric Mosaics -A slit image segments based approach Application of analogical slit images

34. Motivation 4 In concentric mosaics, only one slit image is captured for every analogical slit image set. And this slit image is simply used as substitution for all its analogical slit images.  Distortion caused 4 find the pixel relations between analogical slit images and correct the distortion of images

35. Slit image segments 4 Definition: a slit image segment consists of a series of adjacent pixels in one slit image which have either similar color or similar depth. Segment is used as primitive of image. 4 Applications: use segment mapping instead of pixel mapping between analogical slit images 4 Advantages: –Reduce big amount of data. –Segment is used as basic block in VQ compression

36. How to segment slit images 4 Analyze 2 analogical slit images 4 Initial segment –Find edge point of slit image 4 Warp slit image segment of one slit image to its analogical slit image, find the best segmentation and correspondence relations between two slit images.

37. Data slit image and reference slit image 4 In the 2 slit images: –one is used to synthesize novel view, called data slit image. –The other is used to find the best segmentation and define the segment mapping of data slit image, called reference slit image.

38. Implementation 4 Capturing Slit Images using normal camera 4 Calibration between concentric mosaics 4 Slit Image Segments Matching 4 Synthesizing novel view

39. Capturing slit images using normal camera inward Setupoutward Setup

40. Capturing two set of Concentric Mosaics a)Same direction setup b)Opposite direction setup

41. Possible errors 4 Lead to wrong “analogical” slit images

42. Calibration between concentric mosaics –Estimate the errors parameters so that we can find the correct analogical slit images. 4 smallΔθ is treated asΔφ for simplification. 4 Only consider the relative error e of R 。

43. Calibration between concentric mosaics 4 Method: –analogical slit images should be alike –select a set of slit images in one CM, calculate their analogical slit images in another CM with the consideration of introduced error parameters. S adj is the set of slit images in one CM for calibration use Conform() is a likelihood measurement between data and reference slit images.

44. Calibration in the Same Direction Setup Two error parameters 4 notice when |θ d | is small, e has only small effect to θ r andφ r. 4 De-coupling: Select the slit images with small |θ d |, estimate Δφ, then estimate e.

45. Calibration in the Opposite Direction Setup 4 Only need to estimate 1 error parameter

46. Preprocessing 4 Edge detection inside slit image: find the initial segment 4 warp the initial segments to its analogical slit images, find the best segmentation and correspondence relations between two analogical slit image.

47. Generate corrected image Desired Slit image Set: Generate images from known slit image segment relations between analogical slit images

48. Result

49. Panoramic mosaics of slit images with depth

50. Panorama Method (Chen, 1995) 4 Only several picture captured at a viewpoint needed, small data size and easy to sampling. although  The only off-the-shelf IBR technological for large scene although Fixed viewpoint, can only look around and zoom in / zoom out, or hop between viewpoints

51. Data size Vs. Motion range in IBR 4 Small data size  very limited DOF of virtual camera 4 more DOF  huge data size of virtual camera

52. Slit images with depth Assume a uniform depth value is used for every slit image

53. Panoramaic mosaics of Slit image with depth

54. recover or assign depth 4 recover depth from correspondence relations between analogical slit images –search correspondence points –interactive assign correspondence points  recover depth 4 Depth may be got from a known map

55. Interactive Rendering : Finding Slit Images

56. Interactive Rendering: Adjusting 4 Looming effects simulation –scale slit images uniformly 4 fill holes –fill holes using nearby slit images

57. Sample multiple panoramic mosaics of slit image with depth

58. Join multiple mosaics together 4 Join multiple mosaics together to achieve a wider motion range of virtual point

59. Specify reference points

60. Map slit images to reference point

61. Generate novel view

62. Implementation 4 Sampling –capture slit images –recover or assign depth 4 Preprocessing –mapping slit images to reference circle 4 Interactive Rendering

63. Synthesized view Move forward and backward Move left and right

64. Advantages and Disadvantages 4 Advantages –3 DOF (move left and right, forward and backward, look around)for the virtual camera with small data size –multiple mosaics can be joined up smoothly 4 Disadvantages –only fit for those scene depth variation is small along the vertical direction scene or for open scene

65. Conclusion 4 For 3 DOF motion, slit image is a good data representation for scene 4 We studied the slit image proprieties and introduced the following 3 concepts: –analogical slit images –analogical slit image set –slit image field

66. Conclusion 4 Applications of slit image concepts –Use of slit image segments to correct vertical distortion of concentric mosaics –A new IBR method: panoramic mosaics of slit image with depth