1. ENEE631 Digital Image Processing (Spring'04) Sampling Issues in Image and Video Spring ’04 Instructor: Min Wu ECE Department, Univ. of Maryland, College Park   www.ajconline.umd.edu (select ENEE631 S’04)   minwu@eng.umd.edu UMCP ENEE631 Slides (created by M.Wu © 2004) Based on ENEE631 Spring’04 Section 15

2. ENEE631 Digital Image Processing (Spring'04) Lec24 – 2-D and 3-D Sampling [2] Overview and Logistics Last Time: –Motion analysis –Geometric relations and manipulations Today: –High-order nonlinear spatial warping –2-D sampling at Rectangular grid –Lattice theory for multidimensional sampling at non-rectangular grid –Sampling and resampling for video UMCP ENEE631 Slides (created by M.Wu © 2004)

3. ENEE631 Digital Image Processing (Spring'04) Lec24 – 2-D and 3-D Sampling [3] Sampling: From 1-D to 2-D and 3-D UMCP ENEE631 Slides (created by M.Wu © 2004)

4. ENEE631 Digital Image Processing (Spring'04) Lec24 – 2-D and 3-D Sampling [4] Review: 1-D Sampling Time domain –Multiply continuous-time signal with periodic impulse train Frequency domain –Duality: sampling in one domain  tiling in another domain u FT of an impulse train is an impulse train (proper scaling & stretching) Review Oppenheim “Sig. & Sys” Chapt.7 (Sampling) Chapt.3,4,5 (FS,FT,DFT) x(t) p(t) =  k  ( t - kT) T x s (t) P(  ) =  k  (  - 2k  /T) *2  /T 2  /T X(  )  Xs()Xs() 2  /T UMCP ENEE631 Slides (created by M.Wu © 2001)

5. ENEE631 Digital Image Processing (Spring'04) Lec24 – 2-D and 3-D Sampling [5] Review: 1-D Sampling Theorem 1-D Sampling Theorem –A 1-D signal x(t) bandlimited within [-  B,  B ] can be uniquely determined by its samples x(nT) if  s > 2  B (sample fast enough). –Using the samples x(nT), we can reconstruct x(t) by filtering the impulse version of x(nT) by an ideal low pass filter Sampling below Nyquist rate (2  B ) cause Aliasing X s (  ) with  s < 2  B  Aliasing  s =2  /T BB X s (  ) with  s > 2  B  Perfect Reconstructable  s =2  /T BB -s-s UMCP ENEE631 Slides (created by M.Wu © 2001)

6. ENEE631 Digital Image Processing (Spring'04) Lec24 – 2-D and 3-D Sampling [6] Extend to 2-D Sampling with Rectangular Grid Bandlimited 2-D signal –Its FT is zero outside a bounded region ( |  x |>  x0, |  y |>  y0 ) in spatial freq. domain –Real-word multi-dimensional signals often exhibit diamond or football shape of support u With spectrum normalization, we will get spherical shape of support Jain’s Fig.4.6 UMCP ENEE631 Slides (created by M.Wu © 2001/2004)

7. ENEE631 Digital Image Processing (Spring'04) Lec24 – 2-D and 3-D Sampling [7] 2-D Sampling (cont’d) 2-D Comb function comb(x,y;  x,  y) =  m,n  ( x - m  x, y - n  y ) ~ separable function FT: COMB(  x,  y ) = comb(  x,  y ; 1/  x, 1/  y) /  x  y Sampling vs. Replication (tiling) –Nyquist rates (2  x0 and 2  y0 )  Aliasing Jain’s Fig.4.7 UMCP ENEE631 Slides (created by M.Wu © 2001)

8. ENEE631 Digital Image Processing (Spring'04) Lec24 – 2-D and 3-D Sampling [8] Examples of Aliasing Zig-zag / staircase at boundary Moiré effects UMCP ENEE631 Slides (created by M.Wu © 2004)

9. ENEE631 Digital Image Processing (Spring'04) Lec24 – 2-D and 3-D Sampling [9] 2-D Sampling: Beyond Rectangular Grid Sampling at nonrectangular grid –M ay give more efficient sampling density when spectrum region of support is not rectangular u Sampling density measured by #samples needed per unit area –E.g. interlaced grid for diamond- shaped region of support u equiv. to rotate 45-deg. of rectangular grid u spectrum rotate by the same degree From Wang’s book preprint Fig.4.2 UMCP ENEE631 Slides (created by M.Wu © 2001/2004)

10. ENEE631 Digital Image Processing (Spring'04) Lec24 – 2-D and 3-D Sampling [10] 2-D Sampling Beyond Rectangular Grid Nonrectangular Grid Sampling and Interlacing –Sampling grids other than rectangular may give more efficient sampling density for non-rectangular region of support u Sampling density measured by #samples needed per unit area –E.g. interlaced grid for diamond-shaped region of support Jain’s Fig.4.9 UMCP ENEE631 Slides (created by M.Wu © 2001)

11. ENEE631 Digital Image Processing (Spring'04) Lec24 – 2-D and 3-D Sampling [11] General Sampling Lattice Lattice  in K-dimension space R K –A set of all possible vectors represented as integer weighted combinations of K linearly independent basis vectors Generating matrix V (sampling matrix) V = [v 1, v 2, …, v k ] => lattice points x = V n e.g., identity matrix V ~ square lattice Voronoi cell of a lattice –A “unit cell” of a lattice, whose translations cover the whole space –Consists of vectors that are closer to the origin than to other lattice points u cell boundaries are equidistant lines between surrounding lattice points Sampling density d(  ) = 1 / |det(V)| –|det(V)| measures volume of a cell; d(  ) is # lattice points in unit volume From Wang’s book preprint Fig.3.1 UMCP ENEE631 Slides (created by M.Wu © 2001/2004)

12. ENEE631 Digital Image Processing (Spring'04) Lec24 – 2-D and 3-D Sampling [12] Sampling Density: d1 = 1 d2 = 2 /  3 From Wang’s book preprint Fig.3.1 Example of Lattices UMCP ENEE631 Slides (created by M.Wu © 2004)

13. ENEE631 Digital Image Processing (Spring'04) Lec24 – 2-D and 3-D Sampling [13] Frequency Domain View & Reciprocal Lattice Reciprocal lattice  # for a lattice  (with generating matrix V) –Generating matrix of  # is U = (V T ) -1 –Basis vectors for  and  # are orthonormal to each other: V T U = I –Denser lattice  has sparser reciprocal lattice  # : det(U) = 1 / det(V) Frequency domain view of sampling over lattice –Sampling in spatial domain  Repetition in freq. Domain –Repetition grid in freq. domain can be described by reciprocal lattice –Intuition for “reciprocal” [e.g.] rectangular grid that sample faster horizontally than vertically =>the repetition in frequency domain is slower horizontally than vertically Aliasing and prefiltering to avoid aliasing –Aliasing happens when Voronoi cell of reciprocal lattice can’t completely cover signal spectrum UMCP ENEE631 Slides (created by M.Wu © 2001/2004)

14. ENEE631 Digital Image Processing (Spring'04) Lec24 – 2-D and 3-D Sampling [14] Generalized Sampling Theorem –Wang’s book Sec.3.2.2, Theorem 3.6 –(see also Sec.2.1-2.2 on multi-dimensional FT and DFT) UMCP ENEE631 Slides (created by M.Wu © 2004)

15. ENEE631 Digital Image Processing (Spring'04) Lec24 – 2-D and 3-D Sampling [15] Sampling Efficiency Consider spherical signal spectrum support –Most real-world signals have symmetric freq. contents in many directions –The multi-dim spectrum can be approximated well by a sphere (with proper scaling spectrum support) Voronoi cell of reciprocal lattice need to cover the sphere to avoid aliasing –Tighter fit of the Voronoi cell to the sphere requires less sampling density What lattice gives the best sphere-covering capability? Sampling Efficiency  = volume(unit sphere) / d(  )  prefer close to 1 From Wang’s book preprint Fig.4.2 & 3.5 UMCP ENEE631 Slides (created by M.Wu © 2001/2004)

16. ENEE631 Digital Image Processing (Spring'04) Lec24 – 2-D and 3-D Sampling [16] Sampling Lattice Conversion From Wang’s book preprint Fig.4.4 Intermediate Original Targeted UMCP ENEE631 Slides (created by M.Wu © 2001)

17. ENEE631 Digital Image Processing (Spring'04) Lec24 – 2-D and 3-D Sampling [17] Recall: 1-D Upsample and Downsample From Crochiere-Rabiner “Multirate DSP” book Fig.2.15-16 UMCP ENEE631 Slides (created by M.Wu © 2001)

18. ENEE631 Digital Image Processing (Spring'04) Lec24 – 2-D and 3-D Sampling [18] General Procedures for Sampling Rate Conversion From Wang’s book preprint Fig.4.1 UMCP ENEE631 Slides (created by M.Wu © 2001)

19. ENEE631 Digital Image Processing (Spring'04) Lec24 – 2-D and 3-D Sampling [19] Example: Frame Rate Conversion Video sampling: formulate as a 3-D sampling problem Note: different signal characteristics and visual sensitivities along spatial and temporal dimensions (see Wang’s Sec.3.3 on video sampling) General Approach to frame rate conversion –Upsample => LPF => Downsample Interlaced 50 fields/sec  60 fields/sec –Analyze in terms of 2-D sampling lattice (y, t) –Convert odd field rate and even field rate separately u do 25  30 rate conversion twice u not fully utilize info. in the other fields –Deinterlace first then convert frame rate u do 50  60 frame rate conversion: 50  300  60 –Simplify 50  60 by converting 5 frames  6 frames u each of output 6 frames is from two nearest frames of the 5 originals u weights are inversely proportional to the distance between I/O –May do motion-interpolation for hybrid-coded video UMCP ENEE631 Slides (created by M.Wu © 2001/2004)

20. ENEE631 Digital Image Processing (Spring'04) Lec24 – 2-D and 3-D Sampling [20] From Wang’s book preprint Fig.4.3 UMCP ENEE631 Slides (created by M.Wu © 2001)

21. ENEE631 Digital Image Processing (Spring'04) Lec24 – 2-D and 3-D Sampling [21] Case Studies on Sampling and Resampling in Video Processing Reading Assignment: Wang’s book Chapter 4 UMCP ENEE631 Slides (created by M.Wu © 2004)

22. ENEE631 Digital Image Processing (Spring'04) Lec24 – 2-D and 3-D Sampling [22] Video Format Conversion for NTSC  PAL Require both temporal and spatial rate conversion –NTSC 525 lines per picture, 60 fields per second –PAL 625 lines per picture, 50 fields per second Ideal approach (direct conversion) –525 lines 60 field/sec  13125 line 300 field/sec  625 lines 50 field/sec 4-step sequential conversion –Deinterlace => line rate conversion => frame rate conversion => interlace UMCP ENEE631 Slides (created by M.Wu © 2001)

23. ENEE631 Digital Image Processing (Spring'04) Lec24 – 2-D and 3-D Sampling [23] Video Format Conversion for NTSC  PAL Require both temporal and spatial rate conversion –NTSC 525 lines per picture, 60 fields per second –PAL 625 lines per picture, 50 fields per second Ideal approach (direct conversion) –525 lines 60 field/sec  13125 line 300 field/sec  625 lines 50 field/sec 4-step sequential conversion –Deinterlace => line rate conversion => frame rate conversion => interlace –Food for thought: compare the computation of ideal and sequential u Hint: filtering done for X lines @ Y field/sec => examine X & Y UMCP ENEE631 Slides (created by M.Wu © 2001)

24. ENEE631 Digital Image Processing (Spring'04) Lec24 – 2-D and 3-D Sampling [24] From Wang’s book preprint Fig.4.9 UMCP ENEE631 Slides (created by M.Wu © 2001)

25. ENEE631 Digital Image Processing (Spring'04) Lec24 – 2-D and 3-D Sampling [25] Simplified Video Format Conversion 50 field/sec  60 field/sec –Simplified after deinterlacing to 5 frames  6 frames –Conversion involves two adjacent frames only 625 lines  525 lines –Simplified to 25 lines  21 lines –Conversion involves two adjacent lines only UMCP ENEE631 Slides (created by M.Wu © 2001)

26. ENEE631 Digital Image Processing (Spring'04) Lec24 – 2-D and 3-D Sampling [26] UMCP ENEE631 Slides (created by M.Wu © 2001)

27. ENEE631 Digital Image Processing (Spring'04) Lec24 – 2-D and 3-D Sampling [27] Interlaced Video and Deinterlacing Interlaced video Odd field at 0  Even field at  t  Odd field at 2  t  Even field at 3  t … Deinterlacing –Merge to get a complete frame with odd and even field Examples from http://www.geocities.com/lukesvideo/interlacing.html UMCP ENEE631 Slides (created by M.Wu © 2001)

28. ENEE631 Digital Image Processing (Spring'04) Lec24 – 2-D and 3-D Sampling [28] De-interlacing: Practical Approaches Spatial interpolation –Vertical interpolation within the same field (1-D upsample by 2) –Line averaging ~ average the line above and below D=(C+E)/2 Temporal interpolation –2-frame field merging => artifacts –3-frame field averaging D=(K+R)/2 u fill in the missing odd field by averaging odd fields before and after Spatial-temporal interpolation –Line-and-field averaging D=(C+E+K+R)/4 UMCP ENEE631 Slides (created by M.Wu © 2001)

29. ENEE631 Digital Image Processing (Spring'04) Lec24 – 2-D and 3-D Sampling [29] De-interlacing: Practical Approaches Spatial interpolation –Vertical interpolation within the same field (1-D upsample by 2) –Line averaging ~ average the line above and below D=(C+E)/2 u interpolation filter: [ ½, 1, ½ ] T u DTFT =  x(n) e -j2  n  y fy = 1 + ( e -j2  y fy + e j2  y fy ) / 2 = 1 + cos 2  y f y Temporal interpolation –2-frame field merging => artifacts –3-frame field averaging D=(K+R)/2 u fill in the missing odd field by averaging odd fields before and after Spatial-temporal interpolation –Line-and-field averaging D=(C+E+K+R)/4 UMCP ENEE631 Slides (created by M.Wu © 2001)

30. ENEE631 Digital Image Processing (Spring'04) Lec24 – 2-D and 3-D Sampling [30] Motion-Compensated De-interlacing Stationary video scenes –Temporary deinterlacing approach yield good result Scenes with rapid temporal changes –Artifacts incurred from temporal interpolation –Spatial interpolation alone is better than involving temporal interpolation Switching between spatial & temporal interpolation modes –Based on motion detection result –Hard switching or weighted average –Motion-compensated interpolation UMCP ENEE631 Slides (created by M.Wu © 2001)

31. ENEE631 Digital Image Processing (Spring'04) Lec24 – 2-D and 3-D Sampling [31] Summary of Today’s Lecture Geometric warping on images via high-order mapping Sampling and resampling issues in 2-D and 3-D –Sampling lattice and frequency-domain interpretation –Sampling rate conversion Next Lecture: –Introduction to digital watermarking for image and video Readings –Wang’s book: Sec. 3.1-3.3, 3.5; Chapter 4 –“Computer Graphics” Chapter 5 (Hearn-Baker, Prentice-Hall, 2 nd Ed) UMCP ENEE631 Slides (created by M.Wu © 2004)

32. ENEE631 Digital Image Processing (Spring'04) Lec24 – 2-D and 3-D Sampling [32]

33. ENEE631 Digital Image Processing (Spring'04) Lec24 – 2-D and 3-D Sampling [33] Resource UMCP ENEE631 Slides (created by M.Wu © 2004)

34. ENEE631 Digital Image Processing (Spring'04) Lec24 – 2-D and 3-D Sampling [34] How Today’s Lecture Operates Same team as for course project Introducing the topics (by instructor) and Q&A Step-1 (5 minutes) Independent thinking Step-2 (15 minutes) Team discussion Step-3 (25 minutes) Class discussion –Each team presents their reasoning Step-4 (15 minutes) Independent writeup  Do not refer to textbooks!

35. ENEE631 Digital Image Processing (Spring'04) Lec24 – 2-D and 3-D Sampling [35] Topic-1: Error Concealment for Images Image may be corrupted due to communication and storage errors –Corruption is often in the form of blocks or stripes for JPEG Questions –(1) How to recover corrupted block for the best visual quality? –(2) How about for the best balance between recovered quality and computation complexity? –(3) Will you be able to well conceal any lost block? Explain. –(4) What can the error concealment tool be useful beyond concealing errors? => Hint: may take advantage of info. in non-lost blocks

36. ENEE631 Digital Image Processing (Spring'04) Lec24 – 2-D and 3-D Sampling [36] Topic-2: Deinterlacing Interlaced video –Odd field at 0  Even field at  t  Odd field at 2  t  Even field at 3  t … Deinterlacing –Merge to get a complete frame with odd and even field Question –(1) How to do deinterlacing? Find as many ways as possible. –(2) What is the frequency domain interpretation of your deinterlacing methods? –(3) Compare various aspects of your solutions (complexity, deinterlaced quality, etc.) => Hint: may observe sampling grid pattern in (y,t)-plane (From Wang’s book preprint Fig.4.5)

37. ENEE631 Digital Image Processing (Spring'04) Lec24 – 2-D and 3-D Sampling [37] Examples from http://www.geocities.com/lukesvideo/interlacing.html

38. ENEE631 Digital Image Processing (Spring'04) Lec24 – 2-D and 3-D Sampling [38] Topic-3: Frame Rate Conversion Different frame rate –Movie ~ 24 fps –NTSC ~ 30 fps (equivalent) –PAL ~ 25 fps (equivalent) Question –(1) Find good ways to change frame rate: Movie  NTSC  PAL –(2) Find simplified solutions to reduce computation complexity –(3) What is the frequency domain interpretation of your solutions? –(4) How to convert field rate for interlaced video? u NTSC 60 fields per sec  PAL 50 fields per sec.