Image and Video Processing: An Introduction and Overview 9/12/2018 Image and Video Processing: An Introduction and Overview UMCP ENEE631 Slides (created by M.Wu © 2001)

Why Do We Process Images? 9/12/2018 Why Do We Process Images? Enhancement and restoration Remove artifacts and scratches from an old photo/movie Improve contrast and correct blurred images Composition (for magazines and movies), Display, Printing … Transmission and storage images from oversea via Internet, or from a remote planet Information analysis and automated recognition Providing “human vision” to machines Medical imaging for diagnosis and exploration Security, forensics and rights protection Encryption, hashing, digital watermarking, digital fingerprinting … UMCP ENEE631 Slides (created by M.Wu © 2001) M. Wu: ENEE631 Digital Image Processing (Spring'09)

UMCP ENEE631 Slides (created by M.Wu © 2001) 9/12/2018 Why Digital? “Exactness” Perfect reproduction without degradation Perfect duplication of processing result Convenient & powerful computer-aided processing Can perform sophisticated processing through computer hardware or software Even kindergartners can do some! Easy storage and transmission 1 CD can store hundreds of family photos! Paperless transmission of high quality photos through network within seconds UMCP ENEE631 Slides (created by M.Wu © 2001) M. Wu: ENEE631 Digital Image Processing (Spring'09)

Examples of Digital Image & Video Processing 9/12/2018 Examples of Digital Image & Video Processing Compression Manipulation and Restoration Restoration of blurred and damaged images Noise removal and reduction Morphing Applications Visual mosaicing and virtual views Face detection Visible and invisible watermarking Error concealment and resilience in video transmission UMCP ENEE631 Slides (created by M.Wu © 2001) M. Wu: ENEE631 Digital Image Processing (Spring'09)

UMCP ENEE631 Slides (created by M.Wu © 2001) 9/12/2018 Compression Color image of 600x800 pixels Without compression 600*800 * 24 bits/pixel = 11.52K bits = 1.44M bytes After JPEG compression (popularly used on web) only 89K bytes compression ratio ~ 16:1 Movie ~ Image Sequence 720x480 per frame, 30 frames/sec, 24 bits/pixel Raw video ~ 243M bits/sec DVD ~ about 5M bits/sec Compression ratio ~ 48:1 UMCP ENEE631 Slides (created by M.Wu © 2001) “Library of Congress” by M.Wu (600x800) M. Wu: ENEE631 Digital Image Processing (Spring'09)

UMCP ENEE631 Slides (created by M.Wu © 2001) 9/12/2018 Denoising UMCP ENEE631 Slides (created by M.Wu © 2001) From X.Li http://www.ee.princeton.edu/~lixin/denoising.htm M. Wu: ENEE631 Digital Image Processing (Spring'09)

UMCP ENEE631 Slides (created by M.Wu © 2001) 9/12/2018 Deblurring UMCP ENEE631 Slides (created by M.Wu © 2001) http://www.mathworks.com/access/helpdesk/help/toolbox/images/deblurr7.shtml M. Wu: ENEE631 Digital Image Processing (Spring'09)

Special Effects: Morphing 9/12/2018 Special Effects: Morphing UMCP ENEE631 Slides (created by M.Wu © 2001) Use web browser to show this Princeton CS426 face morphing examples http://www.cs.princeton.edu/courses/archive/fall98/cs426/assignments/morph/morph_results.html M. Wu: ENEE631 Digital Image Processing (Spring'09)

UMCP ENEE631 Slides (created by M.Wu © 2001) 9/12/2018 Face Detection UMCP ENEE631 Slides (created by M.Wu © 2001) Face detection in ’98 @ CMU CS, http://www.cs.cmu.edu/afs/cs/Web/People/har/faces.html Image enhancement, feature extractions, and statistical modeling are often important steps in computer vision tasks See more image understanding examples by Prof. Chellappa’s research group (http://www.cfar.umd.edu/~rama/research.html) M. Wu: ENEE631 Digital Image Processing (Spring'09)

9/12/2018 Here are several pairs of pictures. We process the top ones to get the lower ones. What were done here? These pictures are taken from a thesis here. You can also get more information from the paper. “General Illumination Correction and Its Application to Face Normalization”, J. Zhu et al, ICASSP 2003 M. Wu: ENEE631 Digital Image Processing (Spring'09)

Data Hiding for Annotating Binary Line Drawings 9/12/2018 Data Hiding for Annotating Binary Line Drawings UMCP ENEE631 Slides (created by M.Wu © 2001) marked w/ “01/01/2000” pixel-wise difference original M. Wu: ENEE631 Digital Image Processing (Spring'09)

9/12/2018 Error Concealment 25% blocks in a checkerboard pattern are corrupted corrupted blocks are concealed via edge-directed interpolation (a) original lenna image (c) concealed lenna image (b) corrupted lenna image UMCP ENEE631 Slides (created by M.Wu © 2001) Examples were generated using the source codes provided by W.Zeng. M. Wu: ENEE631 Digital Image Processing (Spring'09)

So What’s a Digital Image After All? 9/12/2018 So What’s a Digital Image After All? UMCP ENEE408G Slides (created by M.Wu & R.Liu © 2002) M. Wu: ENEE631 Digital Image Processing (Spring'09)

UMCP ENEE631 Slides (created by M.Wu © 2007) 9/12/2018 What is an Image? What we perceive as a grayscale image is a pattern of light intensity over a 2-D plane (aka “image plane”) Described by a nonnegative real-valued function I(x,y) of two continuous spatial coordinates on an image plane. I(x,y) is the intensity of the image at the point (x,y). An image is usually defined on a bounded rectangle for processing I: [0, a]  [0, b]  [0, inf ) Color image Can be represented by three functions: R(x,y) for red, G(x,y) for green, B(x,y) for blue. UMCP ENEE631 Slides (created by M.Wu © 2007) Now, what is an image? We call this image an analog image, because … For the time being, we consider a digital image to be a set of numbers. gray scale image. x y M. Wu: ENEE631 Digital Image Processing (Spring'09)

Different Ways to View an Image 9/12/2018 Different Ways to View an Image (More generally, to view a 2-D real-valued function) Intensity visualization over 2-D (x,y) plane In 3-D (x,y, z) plot with z=I(x,y); red color for high value and blue for low Example from B. Liu – EE488 F’06 Princeton Equal value contour in (x,y) plane M. Wu: ENEE631 Digital Image Processing (Spring'09)

Sampling and Quantization 9/12/2018 Sampling and Quantization Computer handles “discrete” data. Sampling Sample the value of the image at the nodes of a regular grid on the image plane. A pixel (picture element) at (i, j) is the image intensity value at grid point indexed by the integer coordinate (i, j). Quantization Is a process of transforming a real valued sampled image to one taking only a finite number of distinct values. Each sampled value in a 256-level grayscale image is represented by 8 bits. => Stay tuned for the theories on these in future weeks. UMCP ENEE631 Slides (created by M.Wu © 2001) 0 (black) 255 (white) M. Wu: ENEE631 Digital Image Processing (Spring'09)

Recall: 1-D Sampling Theorem 9/12/2018 Recall: 1-D Sampling Theorem 1-D Sampling Theorem A 1-D signal x(t) bandlimited within [-B,B] can be uniquely determined by its samples x(nT) if s > 2B (i.e. sample fast enough). Using the samples x(nT), we can reconstruct x(t) by filtering the impulse version of x(nT) by an ideal low pass filter Sampling below Nyquist rate (2B) cause Aliasing => Will extend sampling theorem to 2-D later in the course UMCP ENEE631 Slides (created by M.Wu © 2002) Xs() with s > 2B  Perfect Reconstructable s=2/T B -s (hidden) Xs() with s < 2B  Aliasing s=2/T B M. Wu: ENEE631 Digital Image Processing (Spring'09)

UMCP ENEE631 Slides (created by M.Wu © 2001) 9/12/2018 Examples of Sampling 256x256 64x64 16x16 UMCP ENEE631 Slides (created by M.Wu © 2001) M. Wu: ENEE631 Digital Image Processing (Spring'09)

Examples of Quantizaion  9/12/2018 Examples of Quantizaion  8 bits / pixel 4 bits / pixel 2 bits / pixel UMCP ENEE631 Slides (created by M.Wu © 2001) M. Wu: ENEE631 Digital Image Processing (Spring'09)

An Ancient Example of Digital Image  9/12/2018 An Ancient Example of Digital Image  An Old “Digital” Picture (from a small church in Crete Island, Greece) => Colored tiles as “pixels” Digital Image is not new. There have been digital images since the ancient times. This is a picture in a small church in Crete. It is a digital picture, made of colored tiles Slide from B. Liu – EE488 F’06 Princeton M. Wu: ENEE631 Digital Image Processing (Spring'09)

UMCP ENEE408G Slides (created by M.Wu & R.Liu © 2002) 9/12/2018 Color of Light Perceived color depends on spectral content (wavelength composition) e.g., 700nm ~ red. “spectral color” A light with very narrow bandwidth A light with equal energy in all visible bands appears white UMCP ENEE408G Slides (created by M.Wu & R.Liu © 2002) “Spectrum” from http://www.physics.sfasu.edu/astro/color.html M. Wu: ENEE631 Digital Image Processing (Spring'09)

Perceptual Attributes of Color 9/12/2018 Perceptual Attributes of Color Value of Brightness (perceived luminance) Chrominance Hue specify color tone (redness, greenness, etc.) depend on peak wavelength Saturation describe how pure the color is depend on the spread (bandwidth) of light spectrum reflect how much white light is added RGB  HSV Conversion ~ nonlinear UMCP ENEE408G Slides (created by M.Wu & R.Liu © 2002) HSV circular cone is from online documentation of Matlab image processing toolbox http://www.mathworks.com/access/helpdesk/help/toolbox/images/color10.shtml M. Wu: ENEE631 Digital Image Processing (Spring'09)

Questions for Today (QFT) 9/12/2018 “Seeing yellow” figure is from B.Liu ELE330 S’01 lecture notes @ Princeton; primary color figure is from Chapter 6 slides at Gonzalez/ Woods DIP book website Questions for Today (QFT) Why “seeing yellow without yellow”? mix green and red light to obtain the perception of yellow, without shining a single yellow light 520nm 630nm 570nm = M. Wu: ENEE631 Digital Image Processing (Spring'09)