1. Overview of Digital Video 101 June 28, 2008

2. Aspect Ratio: 16:9 (Widescreen) CRT Type: PureFlat HDTV Display Capability: 480p/1080i Digital Comb Filter Motion-Adaptive: 3D-Y/C Edge Correction: Horizontal/Vertical Edge Correction Digital Velocity Modulated Scan Fine Pitch Picture Tube : (Hi-Res) Color Temperature Control Color Transient Improvement Invar Mask Progressive Cinema Scan (3/2 Pulldown) Progressive Scan Doubler Feature list of a high quality Television

3. Display Format Video Processing Group Two display formats Interlaced – Alternate fields OddEven Non-Interlaced or Progressive

4. Sampling in 2D –Creates 2D spectral replicas –Interlaced format is known as Quincunx Sampling t-1tt+1 Current field + + + + + + + + + + + Quincunx Sampling Orthogonal Sampling

5. Basic Spectral Analysis of Interlaced Luminance Signal Line and field repetition produces a 2D periodic signal Line and field repetition produces a 2D periodic signal For a still image, the 1D spectrum consists of spectral lines at the harmonics of the line (f h ) and frame (f v ) frequencies. For a still image, the 1D spectrum consists of spectral lines at the harmonics of the line (f h ) and frame (f v ) frequencies. Blown up detail of luminance signal spectrum for a typical scene with motion

6. Basic Spectral Analysis of Interlaced Luminance Signal 2D spectra for quincunx sampled video Temporal Freq Vert. Freq v f t f

7. NTSC Overview The first color TV system developed in United States on Jan 23, 1954. For economic reasons it was decided that B&W receivers must be able to display the B&W portion of color broadcast and color receiver should be able to display the B&W broadcast.

8. Bandwidth of baseband NTSC video 1.0 2.03.04.04.2 Allocated bandwidth was (0-4.2MHz).Luminance information is allowed the full bandwidth and color information uses 3.58MHz sub-carrier. This is called Composite Video. Freq (MHz) Amplitude YI Q Q I I

9. What are other Video Formats ? S – Video Separate Y and C, with individual ground CY GND Component Video Three different components, Y (Luminance), U (Hue) and V (Saturation) VGA RGB components, Sync Information, Ground and I 2 C interface

10. Luma and Chroma Interleaving FhFh F h /2 YY 0 v f h f h f2 I,Q

11. NTSC 2D Spectrum Y has quincunx V-T spectrum Y has quincunx V-T spectrum U, V also have quincunx spectrum, but off-setted. U, V also have quincunx spectrum, but off-setted. v t Temporal Freq Vert. Freq

12. Luma-Chroma Separation Many approaches trading off complexity, cost and performance Many approaches trading off complexity, cost and performance Basic approaches Basic approaches Low pass luma / high pass chroma Low pass luma / high pass chroma Notch luma / bandpass chroma Notch luma / bandpass chroma Advanced approaches Advanced approaches 2D passive line comb filter 2D passive line comb filter 2D adaptive line comb filter 2D adaptive line comb filter 3D (spatio-temporal) comb filter 3D (spatio-temporal) comb filter Good decoding requires some black magic (art) because luma and chroma spectrums overlap in real motion video Good decoding requires some black magic (art) because luma and chroma spectrums overlap in real motion video

13. Comb filtering Uses delay elements to create filter with frequency response with many peaks and nulls - like teeth in a comb Uses delay elements to create filter with frequency response with many peaks and nulls - like teeth in a comb 0 v f h f h f2 Peaks coincide with spectral lines that are to be passed through, nulls coincide with spectral lines that are to be eliminated Peaks coincide with spectral lines that are to be passed through, nulls coincide with spectral lines that are to be eliminated

14. Two line Y/C separator DD + 0.5x + Bandpass Filt 2.3-4.9MHz + + + - - Y C Twice amplitude composite video Twice amplitude Chroma Comp Video

15. V-T Spectral Analysis of 2D Comb Filter Chrominance rejection is shown cross- hatched Chrominance rejection is shown cross- hatched Temporal Freq Vert. Freq

16. 3D Y/C Filter Inter-field Y/C separator for No- Motion Intra-field Y/C separator for Motion Motion Detect MUX C0 C1 0/1 + + - C Y Comp Video

17. Color Transient Improvement  Bandwidth expansion sharpens large transitions to create perception of depth and vividness of colors Original Chroma Transition Enhanced Transition Before After

18. HDTV Technical Overview Video: Video: MPEG2 Main Profile @ High Level (MP@HL) MPEG2 Main Profile @ High Level (MP@HL) 18 formats: 6 HD, 12 SD 18 formats: 6 HD, 12 SD Audio: Audio: Dolby AC-3 Dolby AC-3 Transport: Transport: Subset of MPEG2 Subset of MPEG2 Fixed length 188-byte packets Fixed length 188-byte packets RF/Transmission: RF/Transmission: Terrestrial: Terrestrial: 8-VSB (Vestigial Side Band) with Trellis coding 8-VSB (Vestigial Side Band) with Trellis coding effective payload of ~19.3 Mb/s (18.9 Mb/s used for video) effective payload of ~19.3 Mb/s (18.9 Mb/s used for video) Cable: Cable: 16-VSB 16-VSB effective payload of ~38.6 Mb/s effective payload of ~38.6 Mb/s

19. HDTV/DTV System Layers MPEG-2 packets MPEG-2 video and Dolby AC-3 compression syntax Multiple Picture Formats and Frame Rates Multiple Picture Formats and Frame Rates 8-VSB Video packet Audio packet Aux data Data Headers Motion Vectors Chroma and Luma DCT Coefficients Variable Length Codes Flexible delivery of data and future extensability Flexible delivery of data and future extensability 19.3 Mb/s Packet Headers 996 Mb/s 1920 x 1080 @60I

20. ATSC Formats 18 formats: 6 HD, 12 SD – 720 horizontal lines and above considered High Definition – Different picture rates depending on motion content of application 24 frames/sec for film 30 frames/sec for news and live coverage 60 fields/sec, 60 frames/sec for sports and other fast action content 1920 x 1080 @ 60 frames/sec not included because it requires ~100:1 compression to fit in 19.3 Mb/s terrestrial channel, which cannot be done at high quality with MPEG2

21. Aspect Ratios Two options: 16:9 and 4:3 4:3 standard aspect ratio for US TV and computer monitors HD formats are 16:9 – –better match with cinema aspect ratio – –better match for aspect ratio of human visual system – –better for some text/graphics tasks allows side-by-side viewing of 2 pages 4:3 aspect ratio 16:9 aspect ratio 600800800

22. Scaling Aspect ratio conversion will be required – –4:3 material on 16:9 monitor – –16:9 material on 4:3 monitor Several options (shown below) Full Zoom Full ZoomSqueeze 16 x 9 Display Modes4 x 3 Display Modes Squeeze Variable Expand Variable Shrink (j)(d) (b)(a) 4 3 (e) (f) (g) (i) (h) 16 9 (c) Video Transmission Format

23. Scaling / Warping Scaling is special case of warping – –process of resampling a signal after it has undergone some transformation Major issues – –Forward versus reverse mapping – –Separable versus non-separable – –Nyquist versus super-Nyquist – –Up versus down scaling

24. Text-book scaling by the ratio M/N is done by – –Upsampling the signal by M – –Filter intermediate Mx stream with “ideal” reconstruction/band-limiting filter – –Decimate by dropping N-1 out of every N sample Scaling by M/N Actual scalars are implemented using polyphase interpolator MN H(Z) X(n)Y(n)

25. Poly-Phase Scalar M Taps N Phases

26. Poly-Phase Scalar Simple 2x interpolation using 4 tap-2 phase filter ABCD Interpolation sample for A = -0.21

27. Poly-Phase Scalar ABCD Interpolation sample value for B = 0.64

28. Poly-Phase Scalar ABCD Interpolation sample value for C = 0.64

29. Poly-Phase Scalar ABCD Interpolation sample value for D = -0.21 Overall Contribution = -0.21*A+0.64*B+0.64*C-0.21*D

30. Poly-Phase Scalar We can design more complex interpolator with 4 tap- 5 phase filter ABCD We can similarly find the co-efficient for each phase Phase 0 Phase 1 Phase 2 Phase 3

31. Downscaling More difficult than upscaling Each new scaling factor needs cutoff frequency of reconstruction filter to be altered. Inverse relationship between time (space) and frequency requires filter length to grow proportionately to shrink factor. Aliasing and lost information can be very visible when a fixed low-order filter is used Grid downscaled using fixed 2-tap filter Grid downscaled using filter with dynamic taps

32. Good vrs Bad Scalar Upscaling simpler than downscaling Upscaling simpler than downscaling However, poor reconstruction filter can introduce jaggies and Moire’ However, poor reconstruction filter can introduce jaggies and Moire’

33. De-interlacing Vertical -Temporal Progression 1 3 2 4 5 6 7 8 Lines +  + + +  + +  + +  + + + + t-1tt+1 Current field +  = missing line = original line (current field) = original line (adjacent fields) The aim is to interpolate the missing lines from the original lines

34. Methods of De-interlacing Spatial interpolation (“Bob”) Spatial interpolation (“Bob”) Temporal interpolation (“Weave”) Temporal interpolation (“Weave”) Spatio-temporal interpolation Spatio-temporal interpolation Median filtering Median filtering Motion-adaptive interpolation Motion-adaptive interpolation Motion-compensated interpolation Motion-compensated interpolation Inverse 3-2 and 2-2 pulldown (for film) Inverse 3-2 and 2-2 pulldown (for film) Other (statistical estimation, model-based etc) Other (statistical estimation, model-based etc)

35. Popular method of de-interlacing Motion-adaptive Interpolation Interlaced Odd fieldEven field Progressive Motion detectedNo Motion detected Merging of same polarity field Merging of unequal polarity field Switching of vertical resolution due to motion

36. A look at some common chipset Chipsets De-interlacing methods DVD player Brands Genesis gmVLX1A-X Spatio-temporal filtering Sony, Panasonic Silicon Image Sil503 Motion-adaptive interpolation Arcam, Camelot, EAD National Semi NDV8501 Spatial InterpolationPioneer, JVC Sage/Faroudja FLI2200 Motion-adaptive interpolation Denon, Kenwood, Philips, Panasonic, Yamaha

37. Video De-interlacing Artifact - Feathering Feathering – caused by improper handling of motion Feathering – caused by improper handling of motion

38. Edge-adaptive De-interlacing Moving edges are interpolated cleanly by adjusting the direction of interpolation at each pixel to best match the predominant local edge

39. Format Conversion 1 / 6 second Real time Video frame 1 Even field Odd field Movie frame 1 Odd field 1 / 24 second Video frame 2 Movie frame 2 Even field Odd field 1 / 24 second Video frame 4 Even field Video frame 3 Movie frame 3 Odd field Even field 1 / 24 second Movie frame 4 Video frame 5 Odd field Even field 1 / 24 second Conversion of 24 frames/sec into 60 fields/sec: 4 movie frames mapped to 5 video frames In this process, one movie frame is mapped into 3 video fields, the next into 2, etc... Referred to as “3:2 Pulldown” Similar process used to convert 25 frames/sec to 50 fields/sec and 30 frames/sec to 60 fields/sec (“2:2 pulldown”)

40. From movie frame 1 Odd field Odd 1+ Even 1 Odd 1+ Even 1 Odd 1+ Even 1 Even field Odd field Reverse Pull Down Even field From movie frame 2 Odd 2+ Even 2 Odd field Odd 2+ Even 2 Even field Odd field From movie frame 3 Odd 3+ Even 3 Even field Odd 3+ Even 3 Odd 3+ Even 3 Odd field Even field From movie frame 4 Odd 4+ Even 4 Odd and even fields generated from the same original movie frame can be combined with no motion artifacts Odd and even fields generated from the same original movie frame can be combined with no motion artifacts “3:2 Pulldown” sequence detection is necessary “3:2 Pulldown” sequence detection is necessary Done by analysis of motion content Done by analysis of motion content

41. Bad Edit Detection From movie frame 1 Odd field Odd field Odd 1+ Even 1 Odd 1+ Even 1 Even field Odd 1+ Even 1 Edit There are 25 potential edit breaks There are 25 potential edit breaks 2 Good edits 2 Good edits 23 distinct disruptions of the film chain that cause visual bad edits 23 distinct disruptions of the film chain that cause visual bad edits Sequence has to be continuously monitored Sequence has to be continuously monitored Even field From movie frame 3 Odd 3+ Even 3 Odd 3+ Even 3 Even field Odd field Even field From movie frame 4 Odd 4+ Even 4 Odd field Odd 4+ Even 3 Error

42. Aspect Ratio: 16:9 (Widescreen) CRT Type: PureFlat HDTV Display Capability: 480p/1080i Digital Comb Filter Motion-Adaptive: 3D-Y/C Edge Correction: Horizontal/Vertical Edge Correction Digital Velocity Modulated Scan Fine Pitch Picture Tube : (Hi-Res) Color Temperature Control Color Transient Improvement Invar Mask Progressive Cinema Scan (3/2 Pulldown) Progressive Scan Doubler Feature list Revisited

43. References Image/Video/Television – –“Fundamentals of Video” N. Balram, Short Course S-4, SID International Symposium, 2000. – –“Video Demystified: A Handbook for the Digital Engineer”, K. Jack, HighText Publications, 1993. – –“The Art of Digital Video”, J. Watkinson, Focal Press, 1994. HDTV/DTV – –“HDTV Status and Prospects”, B. Lechner, SID 1997 Seminar M-10. detailed history of development of HDTV – –www.atsc.org web site for Advanced Television Systems Committee Linear Systems – –“Signals and Systems”, Oppenheim, Willsky, Young, Prentice Hall.