How to Make Printed and Displayed Images Have High Visual Quality Brian L. Evans Embedded Signal Processing Laboratory The University of Texas at Austin Austin TX 78712-1084 http:://www.ece.utexas.edu/~bevans Ph.D. Graduates: Niranjan Damera-Venkata (HP Labs) Thomas D. Kite (Audio Precision) Graduate Student: Vishal Monga Support: HP Labs National Science Foundation

UT Center for Perceptual Systems Outline Introduction Grayscale halftoning for printing Screening Error diffusion Direct binary search Linear human visual system model Color halftoning for display Optimal design Conclusion UT Center for Perceptual Systems

Need for Digital Image Halftoning Many devices incapable of reproducing grayscale (e.g. eight bits/pixel or gray levels from 0 to 255) Laser and inkjet printers Facsimile machines Low-cost liquid crystal displays Grayscale imagery binarized for these devices Halftoning tries to reproduce full range of gray while preserving quality and spatial resolution Screening methods are fast and simple Error diffusion gives better results on some media UT Center for Perceptual Systems

Digital Halftoning Methods Clutered Dot Screening AM Halftoning Dispersed Dot Screening FM Halftoning Error Diffusion FM Halftoning 1975 Blue-noise Mask FM Halftoning 1993 Green-noise Halftoning AM-FM Halftoning 1992 Direct Binary Search FM Halftoning 1992 UT Center for Perceptual Systems

Screening (Masking) Methods Periodic array of thresholds smaller than image Spatial resampling leads to aliasing (gridding effect) Clustered dot screening produces a coarse image that is more resistant to printer defects such as ink spread Dispersed dot screening has higher spatial resolution Blue noise masking uses large array of thresholds UT Center for Perceptual Systems

Grayscale Error Diffusion Shape quantization noise into high frequencies Design of error filter key to quality Not a screening technique Error Diffusion Spectrum + _ e(m) b(m) x(m) difference threshold compute error shape error u(m) current pixel 3/16 7/16 5/16 1/16 weights 2-D sigma-delta modulation UT Center for Perceptual Systems

Simple Noise Shaping Example Two-bit output device and four-bit input words Going from 4 bits down to 2 increases noise by ~ 12 dB Shaping eliminates noise at DC at expense of increased noise at high frequency. 4 2 Average output = 1/4(10+10+10+11)=1001 Input words To output device 4-bit resolution at DC! 2 2 Added noise 1 sample delay Assume input = 1001 constant 12 dB (2 bits) Time Input Feedback Sum Output 1 1001 00 1001 10 2 1001 01 1010 10 3 1001 10 1011 10 4 1001 11 1100 10 f Periodic If signal is in this band, you are better off UT Center for Perceptual Systems

Modeling Grayscale Error Diffusion Sharpening caused by a correlated error image [Knox, 1992] Floyd- Steinberg Jarvis UT Center for Perceptual Systems Error images Halftones

Modeling Grayscale Error Diffusion Apply sigma-delta modulation analysis to 2-D Linear gain model for quantizer in 1-D [Ardalan and Paulos, 1988] Linear gain model for grayscale image [Kite, Evans, Bovik, 2000] Signal transfer function (STF), noise transfer function (NTF) { us(m) Ks us(m) Ks Signal Path Q(.) u(m) b(m) n(m) Simple linear equalizers are easy to implement but enhance noise Complex equalizers such as the DFE is required (how many taps has an DFE ?) The computational complexity of DFE increases very fast with bit rate un(m) Kn un(m) + n(m) + Noise Path 1 – H(z) is highpass so H(z) is lowpass UT Center for Perceptual Systems

Problems with Error Diffusion Objectionable artifacts Scan order affects results “Worminess” visible in constant graylevel areas Image sharpening Error filters due to [Jarvis, Judice & Ninke, 1976] and [Stucki, 1980] reduce worminess and sharpen edges Sharpening not always desirable: may be adjustable by prefiltering based on linear gain model [Kite, Evans, Bovik, 2000] Computational complexity Larger error filters require more operations per pixel Push towards simple schemes for fast printing UT Center for Perceptual Systems

UT Center for Perceptual Systems Direct Binary Search Minimize mean-squared error between lowpass filtered versions of grayscale and halftone images Lowpass filter is based a linear model of the human visual system (contrast sensitivity function) Iterative method that gives a practical upper bound on the achievable quality for a halftone of an original image Each iteration visits every pixel At each pixel, consider changing state of the pixel (toggle) or swapping it with each of its 8 nearest neighbors that differ in state from it Terminate when if no pixels are changed in an iteration UT Center for Perceptual Systems

UT Center for Perceptual Systems Direct Binary Search Advantages Significantly improved image quality over screening and error diffusion methods Quality of final solution is relatively insensitive to the choice of starting point (initial halftone) Has application in off-line design of threshold arrays for screening methods Disadvantages Computational cost and memory usage is very high in comparison to error diffusion and screening methods Increase complexity makes it unsuitable for real-time applications such as printing UT Center for Perceptual Systems

Contrast Sensitivity Function Contrast needed at a particular spatial frequency for visibility Angular dependence modeled with a cosine function Modify it at low frequency to be lowpass, which gives better correlation with psychovisual results Useful in image quality metrics for optimization and performance evaluation UT Center for Perceptual Systems

Color Halftoning by Error Diffusion for Display Input image has a vector of values (e.g. Red-Green-Blue) at each pixel Error filter has matrix-valued coefficients Algorithm for adapting matrix coefficients [Akarun, Yardimci, Cetin 1997] + _ e(m) b(m) x(m) difference threshold compute error shape error u(m) t(m) The frequency domain equalizer is just a complex division per subchannel Channel shortening equalizer is an 20-30 tap FIR filter My focus is on channel shortening UT Center for Perceptual Systems

Matrix Gain Model for the Quantizer Replace scalar gain with a matrix Noise uncorrelated with signal component of quantizer input Convolution becomes matrix–vector multiplication in frequency domain u(m) quantizer input b(m) quantizer output The frequency domain equalizer is just a complex division per subchannel Channel shortening equalizer is an 20-30 tap FIR filter My focus is on channel shortening Noise component of output Signal component of output UT Center for Perceptual Systems

Optimum Color Noise Shaping Vector color error diffusion halftone model We use the matrix gain model [Damera-Venkata and Evans, 2001] Predicts signal frequency distortion Predicts shaped color halftone noise Visibility of halftone noise depends on Model predicting noise shaping Human visual system model (assume linear shift-invariant) Formulation of design problem Given HVS model and matrix gain model find the color error filter that minimizes average visible noise power subject to certain diffusion constraints UT Center for Perceptual Systems

Linear Color Vision Model Pattern-Color separable model [Poirson and Wandell, 1993] Forms the basis for S-CIELab [Zhang and Wandell, 1996] Pixel-based color transformation B-W R-G The frequency domain equalizer is just a complex division per subchannel Channel shortening equalizer is an 20-30 tap FIR filter My focus is on channel shortening B-Y Opponent representation Spatial filtering UT Center for Perceptual Systems

Linear Color Vision Model Undo gamma correction on RGB image Color separation Measure power spectral distribution of RGB phosphor excitations Measure absorption rates of long, medium, short (LMS) cones Device dependent transformation C from RGB to LMS space Transform LMS to opponent representation using O Color separation may be expressed as T = OC Spatial filtering is incorporated using matrix filter Linear color vision model The frequency domain equalizer is just a complex division per subchannel Channel shortening equalizer is an 20-30 tap FIR filter My focus is on channel shortening where is a diagonal matrix UT Center for Perceptual Systems

Original Image Sample Images and optimum coefficients for sRGB monitor available at: http://signal.ece.utexas.edu/~damera/col-vec.html Original Image UT Center for Perceptual Systems

UT Center for Perceptual Systems Floyd-Steinberg Optimum Filter UT Center for Perceptual Systems

UT Center for Perceptual Systems Conclusions Design of “optimal” color noise shaping filters We use the matrix gain model [Damera-Venkata and Evans, 2001] Predicts shaped color halftone noise HVS could be modeled as a general LSI system Solve for best error filter that minimizes visually weighted average color halftone noise energy Future work Above optimal solution does not guarantee “optimal” dot distributions Tone dependent error filters for optimal dot distributions Improve numerical stability of descent procedure UT Center for Perceptual Systems

Designing the Error Filter Backup Slides Designing the Error Filter Eliminate linear distortion filtering before error diffusion Optimize error filter h(m) for noise shaping Subject to diffusion constraints where The frequency domain equalizer is just a complex division per subchannel Channel shortening equalizer is an 20-30 tap FIR filter My focus is on channel shortening linear model of human visual system matrix-valued convolution UT Center for Perceptual Systems

Generalized Optimum Solution Backup Slides Generalized Optimum Solution Differentiate scalar objective function for visual noise shaping w/r to matrix-valued coefficients Write norm as trace and differentiate trace using identities from linear algebra The frequency domain equalizer is just a complex division per subchannel Channel shortening equalizer is an 20-30 tap FIR filter My focus is on channel shortening UT Center for Perceptual Systems

Generalized Optimum Solution (cont.) Backup Slides Generalized Optimum Solution (cont.) Differentiating and using linearity of expectation operator give a generalization of the Yule-Walker equations where Assuming white noise injection The frequency domain equalizer is just a complex division per subchannel Channel shortening equalizer is an 20-30 tap FIR filter My focus is on channel shortening Solve using gradient descent with projection onto constraint set UT Center for Perceptual Systems

Implementation of Vector Color Error Diffusion Backup Slides Implementation of Vector Color Error Diffusion Hgr Hgg + Hgb UT Center for Perceptual Systems