Halftoning-Inspired Methods for Foveation in Variable Acuity Superpixel Imager Cameras Thayne R. Coffman 1,2 Prof. Brian L. Evans 1 (presenting) Prof. Alan C. Bovik 1 1 Center for Perceptual Systems Department of Electrical and Computer Engineering The University of Texas at Austin The University of Texas at Austin st Century Technologies, Inc. Austin, Texas Austin, Texas November 2, 2005, IEEE Asilomar Conference on Signals, Systems, and Computers

Motivation: Foveated Imagery Foveated imagery has variable spatial resolution Foveated imagery has variable spatial resolution Human visual system Human visual system Provides simultaneous Provides simultaneous Wide field of view Wide field of view High resolution on regions of interest High resolution on regions of interest Low bandwidth Low bandwidth 19% bandwidth means 19% of “superpixels” 19% bandwidth means 19% of “superpixels” No compression in talk No compression in talk Full resolution (100% bandwidth) Variable resolution (19% bandwidth)

Motivation: VASI™ Cameras Variable Acuity Superpixel Imager (VASI) cameras Variable Acuity Superpixel Imager (VASI) cameras Generate foveated images by sharing charges on focal plane array Generate foveated images by sharing charges on focal plane array Achieve frames/sec (e.g. to measure engine RPMs) Achieve frames/sec (e.g. to measure engine RPMs) Pixel sharing reconfigured to achieve a particular frame rate Pixel sharing reconfigured to achieve a particular frame rate Use of 1x1, 2x2, and 4x4 pixel sharing [McCarley et al., 2002] Use of 1x1, 2x2, and 4x4 pixel sharing [McCarley et al., 2002] VASI is a trademark of Nova Sensors, Inc. Images from [McCarley et al., 2002]

The Catch Desired spatial acuity (resolution) is usually specified as a continuous amplitude function on the range (0,1] Desired spatial acuity (resolution) is usually specified as a continuous amplitude function on the range (0,1] Translate desired resolution function to VASI™ binary share/no-share control signal at very high frame rates Translate desired resolution function to VASI™ binary share/no-share control signal at very high frame rates Foveation like the human eye (left pixelation) Two fovea (right pixelation)

Halftoning for VASI Control Signals Select a small number of test images Select a small number of test images Manually specify desired resolution (using Gaussians) Manually specify desired resolution (using Gaussians) Evaluate halftoning methods to control signal translation Evaluate halftoning methods to control signal translation Figures of merit to predict object recognition performance Figures of merit to predict object recognition performance  Peak SNR (PSNR) Weighted SNR (WSNR) Weighted SNR (WSNR) Universal Quality Index (UQI) Universal Quality Index (UQI) Percentage of Bandwidth (PBW) Percentage of Bandwidth (PBW) Control signal for charge sharing at a pixel X X Shared up Shared left

Halftoning Methods Explored Classical screening Classical screening 9-level clustered dot 9-level clustered dot 9-level dispersed dot 9-level dispersed dot Block error diffusion Block error diffusion Floyd-Steinberg error diffusion Floyd-Steinberg error diffusion Blue noise dithering Blue noise dithering White noise White noise Specialized (non- general) methods Specialized (non- general) methods vasiHalftone vasiHalftone vasiHalftone2 vasiHalftone2 Dispersed dot screening F-S error diffusion White noise vasiHalftone

Specialized Methods Generate semi-regularly spaced squares Generate semi-regularly spaced squares Square size varies with inverse of desired bandwidth Square size varies with inverse of desired bandwidth Side is 2 K in vasiHalftone & unconstrained in vasiHalftone2 Side is 2 K in vasiHalftone & unconstrained in vasiHalftone2 Full-resolution image Binarized control (sharing) signal Foveated image Continuous desired resolution signal

Nontrivial Translation of Control Signal Halftoning algorithms aim to achieve a specific ratio of white or black pixels, e.g. Halftoning algorithms aim to achieve a specific ratio of white or black pixels, e.g. For constant I(r,c)=0.1, 10% of pixels will be white (“don’t share”) For constant I(r,c)=0.1, 10% of pixels will be white (“don’t share”) For constant I(r,c)=0.8, 80% of pixels will be white (“don’t share”) For constant I(r,c)=0.8, 80% of pixels will be white (“don’t share”) But bandwidth and resolution are functions of geometry also But bandwidth and resolution are functions of geometry also Example 1 Example 2 50% of pixels don’t share charge: 1% bandwidth 46% of pixels don’t share charge: 15% bandwidth Control signal Resulting image Control signal Resulting image

Nontrivial Translation of Control Signal Relationship between percent of “don’t share” pixels and bandwidth is different for every halftoning method Relationship between percent of “don’t share” pixels and bandwidth is different for every halftoning method Eliminate nonlinearity by applying an inverse function Eliminate nonlinearity by applying an inverse function Implemented with lookup tables storing x = f -1 (y) Implemented with lookup tables storing x = f -1 (y) Given target bandwidth and halftoning method, find average value (x-axis) to use in continuous control signal Given target bandwidth and halftoning method, find average value (x-axis) to use in continuous control signal Stairstep patterns in relationship limit control over bandwidth Stairstep patterns in relationship limit control over bandwidth Floyd-Steinberg gives piecewise linear map and best bandwidth control Floyd-Steinberg gives piecewise linear map and best bandwidth control

Nontrivial Translation of Control Signal Results are greatly improved Results are greatly improved Better bandwidth control Better bandwidth control Better foveation results Better foveation results Floyd-Steinberg (F-S) results below Floyd-Steinberg (F-S) results below Desired bandwidth =11.9% from ideal control signal Uncompensated control signal Achieved bandwidth = 2.6% Compensated control signal Achieved bandwidth = 12.5%

Results: F-S Error Diffusion Good performance and good bandwidth control Good performance and good bandwidth control Good SNR in foveae means accurate object recognition Good SNR in foveae means accurate object recognition Good SNR in periphery means good object detection Good SNR in periphery means good object detection Good bandwidth control means precise VASI frame rate control Good bandwidth control means precise VASI frame rate control Original Sharing Signal Resulting Image PSNR = 17.5 dB (33.3 dB in ROI) WSNR = 16.4 dB (33.8 dB in ROI) Desired BW = 11.6% Actual BW = 12.1% Inflation = 4%

Results: vasiHalftone and vasiHalftone2 For a given desired resolution signal, methods consistently For a given desired resolution signal, methods consistently Had better PSNR & WSNR than other methods Had better PSNR & WSNR than other methods Overshot desired bandwidth by ~30-100% Overshot desired bandwidth by ~30-100% Essentially “cheating” by using extra bandwidth Essentially “cheating” by using extra bandwidth Original Sharing Signal Resulting Image PSNR = 13.3 dB WSNR = 16.9 dB Desired BW = 9.6% Actual BW = 18.8% Inflation = 97%

Results: Other Halftoning Methods Method Performance (SNR) Bandwidth control Block error diffusion PoorGood Classical screening DecentPoor Stochastic methods Poor “Catastrophic gray-out” Original Blue noise Block error diffusion Original Clustered dot Dispersed dot White noise

Conclusions Floyd & Steinberg error diffusion gives the best results while still being able to control bandwidth precisely Floyd & Steinberg error diffusion gives the best results while still being able to control bandwidth precisely vasiHalftone and vasiHalftone2 vasiHalftone and vasiHalftone2 Consistently the best PSNR, WSNR Consistently the best PSNR, WSNR Poor bandwidth control – overshot specifications by % Poor bandwidth control – overshot specifications by % Bandwidth inflation means it’s not a fair comparison (they’re cheating) Bandwidth inflation means it’s not a fair comparison (they’re cheating) Stochastic methods (white & blue noise) perform poorly Stochastic methods (white & blue noise) perform poorly Outperformed by deterministic approaches Outperformed by deterministic approaches Susceptible to “catastrophic gray-out” Susceptible to “catastrophic gray-out” Classical screening performs marginally and has poor bandwidth control Classical screening performs marginally and has poor bandwidth control

Recent Work vasiHalftone3 and vasiHalftone4 vasiHalftone3 and vasiHalftone4 Extensions to eliminate simplifying assumption that VASI™ shareUp and shareLeft signals are equal Extensions to eliminate simplifying assumption that VASI™ shareUp and shareLeft signals are equal This eliminates single-pixel artifacts in non-foveal regions This eliminates single-pixel artifacts in non-foveal regions Eliminated lookup table (LUT) in F-S approach by determining closed-form inverse relationship Eliminated lookup table (LUT) in F-S approach by determining closed-form inverse relationship Significant speedup Significant speedup Greatly shrank LUT in vasiHalftone & vasiHalftone2 approaches Greatly shrank LUT in vasiHalftone & vasiHalftone2 approaches Leveraged “stairstep” form of inverse relationship Leveraged “stairstep” form of inverse relationship 10x speedup in vasiHalftone, 4x speedup in vasiHalftone2 10x speedup in vasiHalftone, 4x speedup in vasiHalftone2 21 st Century Technologies and Nova Sensors are actively collaborating on further work 21 st Century Technologies and Nova Sensors are actively collaborating on further work Sponsored by U.S. Air Force Research Laboratory Sponsored by U.S. Air Force Research Laboratory

Background References B.E. Bayer, “An optimum method for two level rendition of continuous-tone pictures,” Proc. IEEE Int. Conf. on Communications, Conf. Rec., pp. (26-11)-(26-15), B.E. Bayer, “An optimum method for two level rendition of continuous-tone pictures,” Proc. IEEE Int. Conf. on Communications, Conf. Rec., pp. (26-11)-(26-15), R. Floyd and L. Steinberg, “An adaptive algorithm for spatial grayscale,” Proc. SID’76, pp , R. Floyd and L. Steinberg, “An adaptive algorithm for spatial grayscale,” Proc. SID’76, pp , P. McCarley, M. Massie, and J.P. Curzan, “Large format variable spatial acuity superpixel imaging: visible and infrared systems applications,” Proc. SPIE, Infrared Technology and Applications XXX, vol. 5406, pp , Aug P. McCarley, M. Massie, and J.P. Curzan, “Large format variable spatial acuity superpixel imaging: visible and infrared systems applications,” Proc. SPIE, Infrared Technology and Applications XXX, vol. 5406, pp , Aug V. Monga, N. Damera-Venkata, and B.L. Evans, Halftoning Toolbox for Matlab. Version 1.1 released November 7, Available online at V. Monga, N. Damera-Venkata, and B.L. Evans, Halftoning Toolbox for Matlab. Version 1.1 released November 7, Available online at Toolbox for Matlab Toolbox for Matlab R.A. Ulichney, “Dithering with blue noise,” Proc. IEEE, vol. 76, pp , Jan R.A. Ulichney, “Dithering with blue noise,” Proc. IEEE, vol. 76, pp , Jan Z. Wang, A.C. Bovik, and L. Lu, “Wavelet-based foveated image quality measurement for region of interest image coding,” Proc. IEEE Int. Conf. Image Proc., vol. 2, pp , Oct Z. Wang, A.C. Bovik, and L. Lu, “Wavelet-based foveated image quality measurement for region of interest image coding,” Proc. IEEE Int. Conf. Image Proc., vol. 2, pp , Oct 2001.