Eyes for Relighting Extracting environment maps for use in integrating and relighting scenes (Noshino and Nayar)

Previous Work Global illumination – image based lighting Mirror Ball environment maps Eye Shape studies

Idea Find Eye in image by hand Find best fit ellipse Construct ellipsoid in 3 space from ellipse, focal length, distance to camera and previous knowledge of eyes Perform inverse ray tracing from camera to ellipsoid Use law of reflection to project ray onto unit sphere and mark color.

Idea Camera and Picture Cornea Environment Map

Implementation Mark points around eye, find best fit ellipse. Paper discusses using gradients to refine initial guess for ellipse parameters, though in most cases the results are satisfactory without.

Implementation Assume limited perspective, so that the ratio between major and minor axis represents the angle of the plane it lies in with respect to the camera. Assumes basically level camera.

Implementation Use a (t,theta) parameterization for the ellipsoid, where t is the height, and theta the radial angle. Send rays to focal point on camera and sample image (focal point in pixels) Find reflection along normal

Implementation Issues (t,theta) is not uniformly distributed, need far more rays for small t (where the ellipsoid is shallow) than large t No guarantee that whole of sphere is covered, there may be holes.

Implementation Issues To solve these ideally: compute from sphere to camera and sample image. However this is not trivial since rays are dependent upon camera position and one would need to try several camera to eye rays to reach a certain point on the sphere. The solution used just keeps adding rays until there are no gaps in the sphere image

Implementation Once sphere map is calculated, use in global illumination as light source Assumes the map approximates higher light level HDR since the reflection reduced small light to negligible levels. Still need to be careful to pick low levels for illumination Additionally no attempt is made to remove iris texture. Need to pick dark eyes.

Implementation Rendered as a sphere map in Cinema 4d 7.0 with new objects added.

Blade Runner Suppose we want to add a car to the background of this image:

Blade Runner First we find the best fit ellipse

Blade Runner Next we extract the correct environment map. (here rendered on a sphere)

Blade Runner Finally we render the object using the computed environment map (note that it is untextured)

Blade Runner Finally we composite (fiddle by hand!)

Blade Runner

Sections not implemented The map can also be used to unlight a model. Wave a light using the light distribution found on the eyes, and assumption that skin is lambertian (dot product of normal and light direction) find a normal map.

Sections not implemented Since the lighting is known and normals can be found, computing the base color is also possible. All computations take place over set of images with different lighting so best fit matches of both normals and base color can be found.

Sections not implemented Integration of the normals can be done as in previous work to find a 3d model which fits the lighting description. This can also be texture mapped with the base texture, and lit as a lambertian surface with shadows.

Sections not implemented The surface can be inserted into a new scene and lit according to the global illumination of that scene. This illumination can of course come from eyes!

Sections not implemented