M. Wu: ENEE631 Digital Image Processing (Spring'09) Digital Image and Video Processing – An Introduction Spring ’09 Instructor: Min Wu Electrical and Computer Engineering Department University of Maryland, College Park   bb.eng.umd.edu (select ENEE631 S’09)   ENEE631 Spring’09 Lecture-1 (1/26/2009)

M. Wu: ENEE631 Digital Image Processing (Spring'09) Lec1 – Introduction [2] ENEE631 Logistics – Spring 2009 Lectures – Monday and Wednesday 11am-12:15pm, CSI 2120 Assignments and Projects –Matlab will be used for many assignments; C/C++ may also be involved in some. –Kim Lab #2107 ~ image/video related software installed for EE408G u students are encouraged to make use of them in public lab hours. Office Hours – Dr. Min Wu u Wednesday 12:30 – Kim 2142, or by appointment Regularly check the course web page bb.eng.umd.edu

M. Wu: ENEE631 Digital Image Processing (Spring'09) Lec1 – Introduction [3] Scope of ENEE631 First graduate course on image/video processing Prerequisites: ENEE620 and 630, or by permission –Not assume you have much exposure on image processing at undergraduate level –Require and build on background in random process and DSP Emphasis on fundamental concepts –Provide theoretical foundations on multi-dimensional signal processing built upon pre-requisites –Coupled with assignments and projects for hands-on experience and reinforcement of the concepts –Follow-up courses u image analysis, computer vision, pattern recognition u multimedia communications and security

M. Wu: ENEE631 Digital Image Processing (Spring'09) Lec1 – Introduction [4] Textbooks and References R.C. Gonzalez and R.E. Woods: Digital Image Processing, Prentice Hall, 3 rd Edition, (yellow cover) Related technical publications (will be announced in class) Other related textbooks –Y. Wang, J. Ostermann, Y-Q. Zhang: Digital Video Processing and Communications, Prentice Hall, –A.K. Jain: Fundamentals of Digital Image Processing, Prentice Hall, –John W. Woods: Multidimensional Signal, Image, and Video Processing and Coding, Academic Press, –A.Bovik: Handbook Of Image & Video Processing, 2 nd Edition, Academic Press, 2005.

M. Wu: ENEE631 Digital Image Processing (Spring'09) Lec1 – Introduction [5] ENEE631 Course Organization Grading – Assignments and class participation20% – Projects45% – Exams35% Assignments: theoretical problems + computer components –Involves Matlab or C/C++ programming and tasks with image/video tools to reinforce concepts –Grading is based mainly on completeness; encourage further explorations and discussions Projects –Put theories and principles in use and learn from doing; critical thinking Exams –In-class mid-term exam: on basic concepts, theories, and approaches –Final exam: apply theories and principles to image/video proc tasks

M. Wu: ENEE631 Digital Image Processing (Spring'09) Lec1 – Introduction [6] ENEE631 Detailed Grading Plan –Assignments20% u 4 Hw/Lab, 5% each u Class participation 2-5% –Project45% u One on image compression/wavelet coding (~20%) u One on an elective topic (~25%) proposal, 2% presentation, 8% report and overall, 15% – Two exams35% u In-class midterm exam (~20%) u Final exam (in class or take home; ~15%)

M. Wu: ENEE631 Digital Image Processing (Spring'09) Lec1 – Introduction [7] ENEE631 Course Policies No late submission will be accepted –Start early! Plan wisely and prepare for unforeseen hurdles –Inform instructor of special circumstances with documentation Independent work vs. discussions –Write up your solutions INDIVIDUALLY –Discussions with classmates on assignments and projects are encouraged (unless otherwise noted) Computer codes –You should write your own codes unless otherwise stated –DO NOT COPY other students’ codes –Clearly state the code modules obtained elsewhere and consult instructor for permission to use in your project Academic integrity: cheating, plagiarism, fabrication of results, …

M. Wu: ENEE631 Digital Image Processing (Spring'09) Lec1 – Introduction [8] Image and Video Processing: An Introduction and Overview UMCP ENEE631 Slides (created by M.Wu © 2001)

M. Wu: ENEE631 Digital Image Processing (Spring'09) Lec1 – Introduction [9] A picture is worth 1000 words. Rich info. from visual data Examples of images around us natural photographic images; artistic and engineering drawings scientific images (satellite, medical, etc.) “Motion pictures” => video movie, TV program; family video; surveillance and highway/ferry camera A video is worth 1000 sentences? UMCP ENEE631 Slides (created by M.Wu © 2001) JPL Mars’ Panorama captured by the Opportunity

M. Wu: ENEE631 Digital Image Processing (Spring'09) Lec1 – Introduction [10] Increasing Use of Images – A Glimpse from Encyclopedia Britannica First Edition ( ) “A dictionary of arts and sciences” by “a Society of Gentlemen in Scotland” –3 volumes, ~ 2600 pages –illustrated with 160 copperplates 11th Edition (1911) –“last time to encapsulate ALL human knowledge” –one picture every 4 pages 1999 Edition –32 volumes; in CD and DVD –73,000 articles; 30,000 photos and illustrations Now online: (From B. Liu EE488 F’06 at Princeton)

M. Wu: ENEE631 Digital Image Processing (Spring'09) Lec1 – Introduction [11] Why Do We Process Images? Enhancement and restoration –Remove artifacts and scratches from an old photo/movie –Improve contrast and correct blurred images Composition (for magazines and movies), Display, Printing … Transmission and storage –images from oversea via Internet, or from a remote planet Information analysis and automated recognition –Providing “human vision” to machines Medical imaging for diagnosis and exploration Security, forensics and rights protection –Encryption, hashing, digital watermarking, digital fingerprinting … UMCP ENEE631 Slides (created by M.Wu © 2001)

M. Wu: ENEE631 Digital Image Processing (Spring'09) Lec1 – Introduction [12] Why Digital? “Exactness” –Perfect reproduction without degradation –Perfect duplication of processing result Convenient & powerful computer-aided processing –Can perform sophisticated processing through computer hardware or software –Even kindergartners can do some! Easy storage and transmission –1 CD can store hundreds of family photos! –Paperless transmission of high quality photos through network within seconds UMCP ENEE631 Slides (created by M.Wu © 2001)

M. Wu: ENEE631 Digital Image Processing (Spring'09) Lec1 – Introduction [13] Examples of Digital Image & Video Processing Compression Manipulation and Restoration –Restoration of blurred and damaged images –Noise removal and reduction –Morphing Applications –Visual mosaicing and virtual views –Face detection –Visible and invisible watermarking –Error concealment and resilience in video transmission UMCP ENEE631 Slides (created by M.Wu © 2001)

M. Wu: ENEE631 Digital Image Processing (Spring'09) Lec1 – Introduction [14] Compression Color image of 600x800 pixels –Without compression u 600*800 * 24 bits/pixel = 11.52K bits = 1.44M bytes –After JPEG compression (popularly used on web) u only 89K bytes u compression ratio ~ 16:1 Movie ~ Image Sequence –720x480 per frame, 30 frames/sec, 24 bits/pixel –Raw video ~ 243M bits/sec –DVD ~ about 5M bits/sec –Compression ratio ~ 48:1 “Library of Congress” by M.Wu (600x800) UMCP ENEE631 Slides (created by M.Wu © 2001)

M. Wu: ENEE631 Digital Image Processing (Spring'09) Lec1 – Introduction [15] Denoising From X.Li UMCP ENEE631 Slides (created by M.Wu © 2001)

M. Wu: ENEE631 Digital Image Processing (Spring'09) Lec1 – Introduction [16] Deblurring UMCP ENEE631 Slides (created by M.Wu © 2001)

M. Wu: ENEE631 Digital Image Processing (Spring'09) Lec1 – Introduction [17] Special Effects: Morphing Princeton CS426 face morphing examples UMCP ENEE631 Slides (created by M.Wu © 2001)

M. Wu: ENEE631 Digital Image Processing (Spring'09) Lec1 – Introduction [18] Visual Mosaicing –Stitch photos together without thread or scotch tape R. Radke – Princeton thesis 5/2001 UMCP ENEE631 Slides (created by M.Wu © 2001)

M. Wu: ENEE631 Digital Image Processing (Spring'09) Lec1 – Introduction [19] –Image enhancement, feature extractions, and statistical modeling are often important steps in computer vision tasks u See more image understanding examples by Prof. Chellappa’s research group ( Face Detection Face detection in CMU CS, UMCP ENEE631 Slides (created by M.Wu © 2001)

M. Wu: ENEE631 Digital Image Processing (Spring'09) Lec1 – Introduction [20] “General Illumination Correction and Its Application to Face Normalization”, J. Zhu et al, ICASSP 2003

M. Wu: ENEE631 Digital Image Processing (Spring'09) Lec1 – Introduction [21] ( May add vision / image understanding e.g. from Prof. Chellapa’s group ) UMCP ENEE631 Slides (created by M.Wu © 2004)

M. Wu: ENEE631 Digital Image Processing (Spring'09) Lec1 – Introduction [22] Visible Digital Watermarks from IBM Watson web page “Vatican Digital Library” UMCP ENEE631 Slides (created by M.Wu © 2001)

M. Wu: ENEE631 Digital Image Processing (Spring'09) Lec1 – Introduction [23] UMCP ENEE631 Slides (created by M.Wu © 2001; 2009) Invisible Watermark –Original, marked, and their amplified luminance difference –human visual model for imperceptibility: protect smooth areas and sharp edges

M. Wu: ENEE631 Digital Image Processing (Spring'09) Lec1 – Introduction [24] UMCP ENEE631 Slides (created by M.Wu © 2001) Invisible Watermark –1st & 30th Mpeg4.5Mbps frame of original, marked, and their luminance difference –human visual model for imperceptibility: protect smooth areas and sharp edges

M. Wu: ENEE631 Digital Image Processing (Spring'09) Lec1 – Introduction [25] Data Hiding for Annotating Binary Line Drawings original marked w/ “01/01/2000” pixel-wise difference UMCP ENEE631 Slides (created by M.Wu © 2001)

M. Wu: ENEE631 Digital Image Processing (Spring'09) Lec1 – Introduction [26] UMCP ENEE631 Slides (created by M.Wu © 2001) Error Concealment 25% blocks in a checkerboard pattern are corrupted corrupted blocks are concealed via edge-directed interpolation (a) original lenna image(c) concealed lenna image(b) corrupted lenna image Examples were generated using the source codes provided by W.Zeng.

M. Wu: ENEE631 Digital Image Processing (Spring'09) Lec1 – Introduction [27] 2009 International Conf. on Image Processing (ICIP) According to the Call-for-Paper (Cairo, Egypt, Nov. 2009) 16th in the series (since 1994) Research frontiers ranging from traditional image processing applications to evolving multimedia and video technologies Areas of interest include but are not limited to: –Image/Video Coding and Transmission: Still image coding, video coding, stereoscopic and 3-D coding, distributed source coding,... –Image/Video Processing: filtering, restoration, enhancement, segmentation, video segmentation and tracking, morphological processing, stereoscopic and 3-D processing, feature extraction and analysis, interpolation and super-resolution, motion detection and estimation,... –Image Formation: Biomedical imaging, remote sensing, geophysical and seismic imaging, optimal imaging, synthetic-natural hybrid image systems –Image Scanning, Display, and Printing: Scanning, sampling, quantization and halftoning, color reproduction, image representation and rendering, … –Image/Video Storage, Retrieval, and Authentication: Image/video databases, image/video indexing and retrieval, multimodality image/video indexing and retrieval, authentication and watermarking –Applications: biomedical sciences, geosciences and remote sensing,...

M. Wu: ENEE631 Digital Image Processing (Spring'09) Lec1 – Introduction [28] So What’s a Digital Image After All? UMCP ENEE408G Slides (created by M.Wu & R.Liu © 2002)

M. Wu: ENEE631 Digital Image Processing (Spring'09) Lec1 – Introduction [29] What is An Image? Grayscale image –A grayscale image is a function I(x,y) of the two spatial coordinates of the image plane. –I(x,y) is the intensity of the image at the point (x,y) on the image plane. u I(x,y) takes non-negative values u assume the image is bounded by a rectangle [0,a]  [0,b] I: [0, a]  [0, b]  [0, inf ) Color image – – Can be represented by three functions, R(x,y) for red, G(x,y) for green, and B(x,y) for blue. y x y x I(x,y) UMCP ENEE408G Slides (created by M.Wu & R.Liu © 2002)

M. Wu: ENEE631 Digital Image Processing (Spring'09) Lec1 – Introduction [30] What is an Image? What we perceive as a grayscale image is a pattern of light intensity over a 2-D plane (aka “image plane”) –Described by a nonnegative real-valued function I(x,y) of two continuous spatial coordinates on an image plane. –I(x,y) is the intensity of the image at the point (x,y). –An image is usually defined on a bounded rectangle for processing I: [0, a]  [0, b]  [0, inf ) Color image –Can be represented by three functions: R(x,y) for red, G(x,y) for green, B(x,y) for blue. x y UMCP ENEE631 Slides (created by M.Wu © 2007)

M. Wu: ENEE631 Digital Image Processing (Spring'09) Lec1 – Introduction [31] Different Ways to View an Image (More generally, to view a 2-D real- valued function) Example from B. Liu – EE488 F’06 Princeton In 3-D (x,y, z) plot with z= I (x,y); red color for high value and blue for low Equal value contour in (x,y) plane Intensity visualization over 2-D (x,y) plane

M. Wu: ENEE631 Digital Image Processing (Spring'09) Lec1 – Introduction [32] Sampling and Quantization Computer handles “discrete” data. Sampling –Sample the value of the image at the nodes of a regular grid on the image plane. –A pixel (picture element) at (i, j) is the image intensity value at grid point indexed by the integer coordinate (i, j). Quantization –Is a process of transforming a real valued sampled image to one taking only a finite number of distinct values. –Each sampled value in a 256-level grayscale image is represented by 8 bits. => Stay tuned for the theories on these in future weeks. 0 (black) 255 (white) UMCP ENEE631 Slides (created by M.Wu © 2001)

M. Wu: ENEE631 Digital Image Processing (Spring'09) Lec1 – Introduction [33] Recall: 1-D Sampling Theorem 1-D Sampling Theorem –A 1-D signal x(t) bandlimited within [-  B,  B ] can be uniquely determined by its samples x(nT) if  s > 2  B (i.e. sample fast enough). –Using the samples x(nT), we can reconstruct x(t) by filtering the impulse version of x(nT) by an ideal low pass filter Sampling below Nyquist rate (2  B ) cause Aliasing => Will extend sampling theorem to 2-D later in the course X s (  ) with  s < 2  B  Aliasing  s =2  /T BB X s (  ) with  s > 2  B  Perfect Reconstructable  s =2  /T BB -s-s UMCP ENEE631 Slides (created by M.Wu © 2002)

M. Wu: ENEE631 Digital Image Processing (Spring'09) Lec1 – Introduction [34] Extend to 2-D Sampling Bandlimited 2-D signal –Its FT is zero outside a bounded region ( |  x |>  x0, |  y |>  y0 ) in spatial freq. domain Jain’s image proc. book Fig.4.6 UMCP ENEE631 Slides (created by M.Wu © 2002)

M. Wu: ENEE631 Digital Image Processing (Spring'09) Lec1 – Introduction [35] 2-D Sampling (cont’d) 2-D Comb function – comb(x,y;  x,  y) =  m,n  ( x - m  x, y - n  y ) – FT  COMB(  x,  y ) = comb(  x,  y ; 1/  x, 1/  y) /  x  y Sampling vs. Replication (tiling) –Nyquist rates (2  x0 and 2  y0 ) and Aliasing Jain’s Fig.4.7 UMCP ENEE631 Slides (created by M.Wu © 2002)

M. Wu: ENEE631 Digital Image Processing (Spring'09) Lec1 – Introduction [36] UMCP ENEE631 Slides (created by M.Wu © 2001) Examples of Sampling 256x256 64x64 16x16

M. Wu: ENEE631 Digital Image Processing (Spring'09) Lec1 – Introduction [37] UMCP ENEE631 Slides (created by M.Wu © 2001) Examples of Quantizaion   8 bits / pixel 4 bits / pixel 2 bits / pixel

M. Wu: ENEE631 Digital Image Processing (Spring'09) Lec1 – Introduction [38] An Ancient Example of Digital Image An Ancient Example of Digital Image   An Old “Digital” Picture (from a small church in Crete Island, Greece) => Colored tiles as “pixels” Slide from B. Liu – EE488 F’06 Princeton

M. Wu: ENEE631 Digital Image Processing (Spring'09) Lec1 – Introduction [39] Summary of Today’s Lecture Course organization and policies Background and examples of digital image processing Sampling and quantization concepts for digital image Next time –Color and Human Visual System UMCP ENEE631 Slides (created by M.Wu © 2001)

M. Wu: ENEE631 Digital Image Processing (Spring'09) Lec1 – Introduction [40] Readings and Assignment Introductory sections in Matlab Image Processing Toolbox – Gonzalez-Wood book, Chapter 1 Bovik’s Handbook – Section 1 Introduction (see course web) Go over mathematical preliminaries –Linear system and basics of 1-D signal processing –FT and ZT –Matrix and linear algebra –Probability UMCP ENEE631 Slides (created by M.Wu © 2001,2004,2009)

M. Wu: ENEE631 Digital Image Processing (Spring'09) Lec1 – Introduction [41]

M. Wu: ENEE631 Digital Image Processing (Spring'09) Lec1 – Introduction [42] Color of Light Perceived color depends on spectral content (wavelength composition) –e.g., 700nm ~ red. –“spectral color” u A light with very narrow bandwidth A light with equal energy in all visible bands appears white “Spectrum” from UMCP ENEE408G Slides (created by M.Wu & R.Liu © 2002)

M. Wu: ENEE631 Digital Image Processing (Spring'09) Lec1 – Introduction [43] Perceptual Attributes of Color Value of Brightness (perceived luminance) Chrominance –Hue u specify color tone (redness, greenness, etc.) u depend on peak wavelength –Saturation u describe how pure the color is u depend on the spread (bandwidth) of light spectrum u reflect how much white light is added RGB  HSV Conversion ~ nonlinear HSV circular cone is from online documentation of Matlab image processing toolbox /helpdesk/help/toolbox/images/col or10.shtml UMCP ENEE408G Slides (created by M.Wu & R.Liu © 2002)

M. Wu: ENEE631 Digital Image Processing (Spring'09) Lec1 – Introduction [44] Questions for Today (QFT) Why “seeing yellow without yellow”? –mix green and red light to obtain the perception of yellow, without shining a single yellow light 520nm630nm 570nm = “Seeing yellow” figure is from B.Liu ELE330 S’01 lecture Princeton; primary color figure is from Chapter 6 slides at Gonzalez/ Woods DIP book website

M. Wu: ENEE631 Digital Image Processing (Spring'09) Lec1 – Introduction [45] Reference Documentation of Matlab Image Processing Toolbox,