Radiometry and Photometric Stereo 1

Estimate the 3D shape from shading information Can you tell the shape of an object from these photos ? 2

White-out: Snow and Overcast Skies CAN’T perceive the shape of the snow covered terrain! CAN perceive shape in regions lit by the street lamp!! WHY? 3

Radiometry What determines the brightness of an image pixel? Light source properties Surface shape Surface reflectance properties Optics Sensor characteristics Slide by L. Fei-Fei Exposure

The journey of the light ray Camera response function: the mapping f from irradiance to pixel values –Useful if we want to estimate material properties –Shape from shading requires irradiance –Enables us to create high dynamic range images Source: S. Seitz, P. Debevec

Recovering the camera response function Method 1: Modeling –Carefully model every step in the pipeline –Measure aperture, model film, digitizer, etc. –This is really hard to get right Slide by Steve Seitz

Method 1: Modeling –Carefully model every step in the pipeline –Measure aperture, model film, digitizer, etc. –This is really hard to get right Method 2: Calibration –Take pictures of several objects with known irradiance –Measure the pixel values –Fit a function Recovering the camera response function irradiance pixel intensity = response curve Slide by Steve Seitz

Recovering the camera response function Method 3: Multiple exposures –Consider taking images with shutter speeds 1/1000, 1/100, 1/10, 1 –The sensor exposures in consecutive images get scaled by a factor of 10 –This is the same as observing values of the response function for a range of irradiances: f(E), f(10E), f(100E), etc. –Can fit a function to these successive values For more info P. E. Debevec and J. Malik. Recovering High Dynamic Range Radiance Maps from Photographs. In SIGGRAPH 97, August 1997Recovering High Dynamic Range Radiance Maps from PhotographsSIGGRAPH 97 response curve Exposure (log scale)  irradiance * time = pixel intensity = Slide by Steve Seitz

The interaction of light and matter What happens when a light ray hits a point on an object? –Some of the light gets absorbed converted to other forms of energy (e.g., heat) –Some gets transmitted through the object possibly bent, through “refraction” –Some gets reflected possibly in multiple directions at once –Really complicated things can happen fluorescence Let’s consider the case of reflection in detail –In the most general case, a single incoming ray could be reflected in all directions. How can we describe the amount of light reflected in each direction? Slide by Steve Seitz

Bidirectional reflectance distribution function (BRDF) Model of local reflection that tells how bright a surface appears when viewed from one direction when light falls on it from another Definition: ratio of the radiance in the outgoing direction to irradiance in the incident direction Radiance leaving a surface in a particular direction: add contributions from every incoming direction surface normal

BRDF’s can be incredibly complicated…

Gonioreflectometers Can add fourth degree of freedom to measure anisotropic BRDFs

Diffuse reflection Dull, matte surfaces like chalk or latex paint Microfacets scatter incoming light randomly Light is reflected equally in all directions: BRDF is constant Albedo: fraction of incident irradiance reflected by the surface Radiosity: total power leaving the surface per unit area (regardless of direction)

Viewed brightness does not depend on viewing direction, but it does depend on direction of illumination Diffuse reflection: Lambert’s law N S B: radiosity ρ: albedo N: unit normal S: source vector (magnitude proportional to intensity of the source) x

Specular reflection Radiation arriving along a source direction leaves along the specular direction (source direction reflected about normal) Some fraction is absorbed, some reflected On real surfaces, energy usually goes into a lobe of directions Phong model: reflected energy falls of with Lambertian + specular model: sum of diffuse and specular term

Specular reflection Moving the light source Changing the exponent

Example Surfaces Body Reflection: Diffuse Reflection Matte Appearance Non-Homogeneous Medium Clay, paper, etc Surface Reflection: Specular Reflection Glossy Appearance Highlights Dominant for Metals Many materials exhibit both Reflections: 17

Diffuse Reflection and Lambertian BRDF viewing direction surface element normal incident direction Lambertian BRDF is simply a constant : albedo Surface appears equally bright from ALL directions! (independent of ) Surface Radiance : Commonly used in Vision and Graphics! source intensity source intensity I 18

Specular Reflection and Mirror BRDF source intensity I viewing direction surface element normal incident direction specular/mirror direction Mirror BRDF is simply a double-delta function : Valid for very smooth surfaces. All incident light energy reflected in a SINGLE direction (only when = ). Surface Radiance : specular albedo 19

Combing Specular and Diffuse: Dichromatic Reflection Observed Image Color = a x Body Color + b x Specular Reflection Color R G B Klinker-Shafer-Kanade 1988 Color of Source (Specular reflection) Color of Surface (Diffuse/Body Reflection) Does not specify any specific model for Diffuse/specular reflection 20

Diffuse and Specular Reflection diffusespeculardiffuse+specular 21

Image Intensity and 3D Geometry Shading as a cue for shape reconstruction What is the relation between intensity and shape? –Reflectance Map 22

Surface Normal surface normal Equation of plane or Let Surface normal 23

Gradient Space Normal vector Source vector plane is called the Gradient Space (pq plane) Every point on it corresponds to a particular surface orientation 24

Reflectance Map Relates image irradiance I(x,y) to surface orientation (p,q) for given source direction and surface reflectance Lambertian case: : source brightness : surface albedo (reflectance) : constant (optical system) Image irradiance: Letthen 25

Lambertian case Reflectance Map (Lambertian) cone of constant Iso-brightness contour Reflectance Map 26

Lambertian case iso-brightness contour Note: is maximum when Reflectance Map 27

Glossy surfaces (Torrance-Sparrow reflectance model) diffuse termspecular term Diffuse peak Specular peak Reflectance Map 28

Shape from a Single Image? Given a single image of an object with known surface reflectance taken under a known light source, can we recover the shape of the object? Given R(p,q) ( (p S,q S ) and surface reflectance) can we determine (p,q) uniquely for each image point? NO Solution: Take more images Photometric stereo 29

Photometric Stereo 30

We can write this in matrix form: Image irradiance: Lambertian case: Photometric Stereo 31

Solving the Equations inverse 32

More than Three Light Sources Get better results by using more lights Least squares solution: Solve for as before Moore-Penrose pseudo inverse 33

Color Images The case of RGB images –get three sets of equations, one per color channel: –Simple solution: first solve for using one channel –Then substitute known into above equations to get –Or combine three channels and solve for 34

Computing light source directions Trick: place a chrome sphere in the scene –the location of the highlight tells you the source direction 35

For a perfect mirror, light is reflected about N Specular Reflection - Recap We see a highlight when Then is given as follows: 36

Computing the Light Source Direction Can compute N by studying this figure –Hints: use this equation: can measure c, h, and r in the image N rNrN C H c h Chrome sphere that has a highlight at position h in the image image plane sphere in 3D 37

Depth from Normals Get a similar equation for V 2 –Each normal gives us two linear constraints on z –compute z values by solving a matrix equation V1V1 V2V2 N 38

Limitations Big problems –Doesn’t work for shiny things, semi-translucent things –Shadows, inter-reflections Smaller problems –Camera and lights have to be distant –Calibration requirements measure light source directions, intensities camera response function 39

Trick for Handling Shadows Weight each equation by the pixel brightness: Gives weighted least-squares matrix equation: Solve for as before 40

Original Images 41

Results - Shape Shallow reconstruction (effect of interreflections) Accurate reconstruction (after removing interreflections) 42

Results - Albedo No Shading Information 43

Original Images 44

Results - Shape 45

Results - Albedo 46

Results 1.Estimate light source directions 2.Compute surface normals 3.Compute albedo values 4.Estimate depth from surface normals 5.Relight the object (with original texture and uniform albedo) 47

Photometric stereo example data from: 48

Alternative approach Reference object with same, but arbitrary, BRDF Use LUT to get from RGB 1 RGB 2 … vector to normal Hertzman and Seitz CVPR’03 49

Photometric stereo camera A Hand-held Photometric Stereo Camera for 3-D Modeling, ICCV’09 50

Results 51

Self-calibrating Photometric Stereo 52

Results 53

Other applications: Shape Palette Painting the “normals” for surface reconstruction A unit sphere has all normal directions User mark-up correspondents to transfer normals from sphere to images 54

Results 55

Multi-view Photometric stereo Combining multi-view stereo with photometric stereo 56

Results 57

Dynamic Shape Capture using Multi- View Photometric Stereo tometricStereo/index.htmlhttp://people.csail.mit.edu/wojciech/MultiviewPho tometricStereo/index.html 58

Suggested Reading Example-Based Photometric Stereo: Shape Reconstruction with General, Varying BRDFs, PAMI’05 Dense Photometric Stereo: A Markov Random Field Approach, PAMI’06 ShapePalettes: Interactive Normal Transfer via Sketching, Siggraph’07 Non-rigid Photometric Stereo with Colored Lights, ICCV’07 A Photometric Approach for Estimating Normals and Tangents, Siggraph Asia’08 A Hand-held Photometric Stereo Camera for 3-D Modeling, ICCV’09 Self-calibrating Photometric Stereo, CVPR’10 59

Summary Radiometry –Describe how the camera responses to the incoming lights –Radiometry calibration, estimation of the camera response function Photometric stereo –Estimate normal and surface from multiple images of same object with different lighting –We study the method for Lambertian surface –We study the method for surface reconstruction from normals –Albedo and Normal information of a surface is very useful and have many applications 60