Visual Perception in Realistic Image Synthesis Ann McNamara

Outline Introduction Modeling important characteristics of the human visual system (HVS) Perception based rendering Image quality metrics Tone reproduction operators Summary

Realism Architecture Stage lighting Entertainment Safety systems Archaeology

Human Visual System Physical structure well established Perceptual behaviour is a complex process

Modeling Important Characteristics of the Human Visual System

Visual Acuity How well we can see fine detail Adaptation level Rods and cones Cones

Number of grating that fall on one degree of the retina Dependent on distance Spatial Frequency

Spatial mechanisms (channels) which are used to represent the visual information at various scales and orientations as it is believed that primary visual cortex does.

Contrast Sensitivity Function Contrast sensitivity function which specifies the detection threshold for a stimulus as a function of its spatial frequencies.

Campbell-Robson contrast sensitivity chart Contrast Sensitivity

Masking Visual masking affecting the detection threshold of a stimulus as a function of the interfering background stimulus which is closely coupled in space and time.

Masking

Colour Appearance

Perceptually Based Rendering

] [Mitchell 1987] Low Sampling Densities Non-uniform sampling is less conspicuous Optimise using how the eye perceives noise as a function of contrast and colour Raytracing -> Point Samples-> Aliasing

Uniform Non-Uniform Adaptive Sampling Schemes ] [Mitchell 1987]

Low Sampling Densities Contrast Colour R 0.4 G 0.3 B 0.6

] [Mitchell 1987] Low Sampling Densities

Frequency Based Raytracing ] [Bolin &Meyer 1992] Synthesise directly into frequency domain Simple vision model to control Where to cast rays How to spawn rays

Frequency Based Raytracing ] [Bolin &Meyer 1992] Vision model Contrast sensitivity Spatial frequency Masking

Frequency Based Raytracing ] [Bolin &Meyer 1992] Specific luminance difference at low intensity more important than same luminance difference at high intensity Colour spatial frequency variations given fewer samples Decrease rays spawned in high frequency regions

Limited Color Acuity ] [Meyer & Liu1998] Colour Abberation Limited sampling of receptor Spatial acuity of opponent channels

] [Meyer & Liu1998]

Application How much computation is enough? How much reduction is too much? An objective metric of image quality which takes into account basic characteristics of the HVS could help to answer these questions without human assistance.

Questions of Appearance Preservation The Concern Is Not Whether Images Are the Same Rather the Concern Is Whether Images Appear the Same

Perceptually Based Adaptive Sampling Algorithm ] [Bolin &Meyer 1998] Image quality model embedded into image synthesis Use statistical information about spatial frequency to determine where to estimate values where samples were yet to be taken

Perceptually Based Adaptive Sampling Algorithm ] [Bolin &Meyer 1998] JND’s VDM

 = 200s  = 400s  = 800s  = 1600s Deterministic radiosity Monte Carlo radiosity Convergence Evaluation ] [ Myszkowski 1997]

 vs. reference 0.5  vs.  ] [ Myszkowski 1997] Termination Criterion

Physical Based Perceptual Metric ] [Ramasubramanian et al1999] Threshold model defines a physical error metric Handles luminance-dependent and spatially dependent processing independently Allowing pre-computation of spatially- dependent component

Physical Based Perceptual Metric ] [Ramasubramanian et al1999]

Image Quality Metrics

Image Quality Compare and validate lighting simulations Use comparisons to guide rendering more efficiently Compute less without altering perception Pixel by pixel comparison might be > 0, human might not see any difference

RMSE 9.5 RMSE 5.2 Pixel by Pixel Comparison Prikryl, 1999

Visible Differences Predictor VDP Image 2 Image 1 Psychometric Function Probability Summation Visualisation of Differences Amplitude Nonlinear. Amplitude Nonlinear. Contrast Sensitivity Function Contrast Sensitivity Function + Cortex Transform Cortex Transform Masking Function Masking Function Unidirectional or Mutual Masking [ Daly ‘93, Myszkowski ‘98]

VDP: Results StandardComparison Pixel differences: Standard - Comparison Pixel differences The VDP response: probability of perceiving the differences VDP response

Daly’s VDP: Features Predicts local differences between images Takes into account important visual characteristics: Amplitude compression Advanced CSF model Masking Uses the cortex transform, which is a pyramid-style, invertible & computationally efficient image representation Daly, 1993

Visible Discrimination Model Map of Just Noticeable Differences Point sample function to model optics Resample the image according to foveal eccentricity Band pass response Contrast pyramid steerable filters Lubin, 1997

Visible Discrimination Model Both images subjected to Identical processing Distance measure Difference in responses for Each channel and summing Them to obtain a JND Map of the two images Lubin, 1997

An Experimental Evaluation of Computer Graphics Imagery Comparing image to real-world scene An approach to image synthesis consisting of A physical module A perceptual module ] [Meyer et al, 1986]

An Experimental Evaluation of Computer Graphics Imagery ] [Meyer et al, 1986] DifferenceSimulatedMeasured

An Experimental Evaluation of Computer Graphics Imagery ] [Meyer et al, 1986]

An Experimental Evaluation of Computer Graphics Imagery ] [Meyer et al, 1986]

Image Quality Metrics ] [Rushmeier et al, 1995] Components of perceptually based metrics adapted from image compression Gervais et al 1984 Mannos et al 1974 Daly 1993

Image Quality Metrics ] [Rushmeier et al, 1995] Daly tested very well Real Room Simulated Model of Room

Visual Psychophysics Determine the relationship between the physical world and human’s subjective experience of that world Measure the response (“psycho”) to a known stimulus (“physics”)

Why Lightness ? ] [Gilchrist 1977 ]

] [McNamara et al 1998, 2000] A Psychophysical Investigation Painted 5-sided cube Objects painted with different grey paints Complex illumination, with secondary reflections

Graphic Reconstructions ] [McNamara et al 1998, 2000]

Experiment Rendered Real Scene ] [McNamara et al 1998, 2000 ]

Results ] [McNamara et al 1998, 2000]

Tone Reproduction Operators

~10 5 cd/m 2 ~10 -5 cd/m 2 Tone Reproduction ~100 cd/m 2 ~1 cd/m 2 Same Visual Response ?

Tone Reproduction for Realistic Images Mapping between radiances computed and light energy emitted from CRT Psychophysical model of brightness perception Observer model Display model Tumblin & Rushmeier, 1993

Tone Reproduction

Tone Reproduction for Realistic Images LowMediumHigh Tumblin & Rushmeier, 1993

A Contrast-based Scalefactor for Luminance Display Linear transform L d = mL W Matching contrast between real and image Ward, 1994

A Contrast-based Scalefactor for Luminance Display Min-MaxWard Ward, 1994

A Model of Visual Adaptation for Realistic Image Synthesis Threshold visibility Changes in colour appearance Visual acuity Temporal Sensitivity Ferwerda et al, 1996

A Model of Visual Adaptation for Realistic Image Synthesis Ferwerda et al, 1996

Spatially Nonuniform Scaling for High Contrast Images Incorrect to apply the same mapping to each pixel Spatial position Chiu et al, 1993

Quantization Techniques for Visualization of High Dynamic Range Pictures Similar to Chiu et al Rational rather than logarithmic Accounts for the non-linearities of both the display device and human perception The biggest advantages is speed Schlick, 1994

A Visibility Matching Tone Reproduction Operator for High Dynamic Range Scenes Preserve visibility of objects Histogram - adjusted to minimise the visible contrast distortions Also includes glare, colour sensitivity, and acuity Larson et al, 1997

A Visibility Matching Tone Reproduction Operator for High Dynamic Range Scenes Larson et al, 1997

Perceptually Driven Radiosity ] [Gibson & Hubbold. 1997] Steer computation to areas in need of most refinement A-priori estimate adaptation luminance Tone-mapping to transform luminance to display Distance between two colors in uniform colour space = numerical measure of perceived difference

Perceptually Driven Radiosity ] [Gibson & Hubbold. 1997] Stop patch refinement once the difference between successive levels becomes perceptually unnoticeable Determine the perceived importance of any shadow Optimise the mesh for faster interactive display and minimise storage

Standard shadow testing (19.33 hours) Perceptually-driven shadow testing (3.10 hours) ] [Gibson & Hubbold ] Shadow Testing

Output mesh Optimised mesh

] [ Hedley et al ] Discontinuity Meshing Throw out discontinuities that are deemed visually unimportant Tone mapping Compare colour differences along the discontinuity line

Culled discontinuities Original scene

Summary Applications of visual perception in computer graphics Efficient software Image quality evaluations Tone reproduction operators Knowledge of HVS can be used to greatly benefit the synthesis of realistic images at various stages of production

Conclusion Great deal of potential Perceptually accurate as well as physically correct Allow high level of confidence in computer imagery allowing us to demonstrate to the world that our images are faithful representations !

Thank You Ann McNamara

Extra…

Spatial and Orientation Mechanisms The following filter banks are commonly used: Gabor functions (Marcelja80), Steerable pyramid transform (Simoncelli92), Discrete Cosine Transform (DCT), Difference of Gaussians (Laplacian) pyramids (Burt83,Wilson91), Cortex transform (Watson87, Daly93).

Cortex Transform: Organization of the Filter Bank

Orientation Bands Cortex Transform: Orientation Bands Input image

Spatiovelocity CSF Contrast sensitivity data for traveling gratings of various spatial frequencies were derived in Kelly’s psychophysical experiments (1960). Daly (1998) extended Kelly’s model to account for target tracking by the eye movements. log visual sensitivity log velocity [deg/sec] log spatial frequency [cycles/deg] Temporal frequency [Hz]

Visual Masking Masking is strongest between stimuli located in the same perceptual channel, and many vision models are limited to this intra-channel masking. The following threshold elevation model is commonly used: