1. ISPASS 2004 © 2004 Marilyn Wolf Multimedia Algorithms Marilyn Wolf Dept. of Electrical Engr. Princeton University

2. ISPASS 2004 © 2004 Marilyn Wolf Outline zCompact disc player. zVideo compression.

3. ISPASS 2004 © 2004 Marilyn Wolf The multimedia processing funnel Data volume Data abstraction pixel processing principal component analysis, hidden Markov models Edge extraction

4. ISPASS 2004 © 2004 Marilyn Wolf CD/MP3 player Audio CPU amp Jog memory Error corrector Servo CPU Analog in Analog out FE, TE, amp focus, tracking, sled, motor head drive memory display DAC I2SI2S

5. ISPASS 2004 © 2004 Marilyn Wolf CD medium zRotational speed: 1.2-1.4 m/s (CLV). zTrack pitch: 1.6 microns. zDiameter: 120 mm. zPit length: 0.8 -3 microns. zPit depth:.11 microns. zPit width: 0.5 microns. zLaser wavelength: 780 nm.

6. ISPASS 2004 © 2004 Wayne Wolf CD mechanism zLaser, lens, sled: laser CD detectors diffraction grating sled track focus

7. ISPASS 2004 © 2004 Marilyn Wolf Laser focus zFocus controlled by vertical position of lens. zUnfocused beam causes irregular spot: In focusOut of focus

8. ISPASS 2004 © 2004 Marilyn Wolf Laser pickup A B C D F E Side spot detectors Level: A+B+C+D Focus error: (A+C)-(B+D) Tracking error: E-F

9. ISPASS 2004 © 2004 Marilyn Wolf Servo control zFour main signals: yfocus (laser) @ 245 kHz; ytracking (laser) @ 245 kHz; ysled (motor): @ 800 Hz; yDisc motor. Optical pickup

10. ISPASS 2004 © 2004 Marilyn Wolf EFM zEight-to-fourteen modulation: yFourteen-bit code guarantees a maximum distance between transitions. 0000001100100100000000

11. ISPASS 2004 © 2004 Marilyn Wolf Error correction zCD capacity: 6.99 GB raw, 700 MB formatted. zReed-Solomon code:  g(x) = (x-  ) (x-  2 ) … (x-  n-k-1 ) (x-  n-k ) zProduces data, erasure bits. zTime to solve varies greatly depending on noise. zCD interleaves Reed-Solomon blocks to reduce effects of large data gaps.

12. ISPASS 2004 © 2004 Marilyn Wolf Control and error correction zSkips caused by physical disturbance. yWait for disturbance to subside. yRetry. zRead errors caused by disc/servo problems. yDetect error. yChoose location for retry. yRetry. yFail and interpolate.

13. ISPASS 2004 © 2004 Marilyn Wolf Audio output zAudio CD output straightforward. yMay perform D/A filtering in software. zMP3 decode is relatively straightforward. y10% of ARM7. zFile system support for data CD is complex: yPC/Mac. yArbitrary file structure.

14. ISPASS 2004 © 2004 Marilyn Wolf MPEG audio standards zLayer 1: yLossless compression of subbands + optional simple masking model zLayer 2: yMore advanced masking model. zLayer 3: yAdditional processing for lower bit rates.

15. ISPASS 2004 © 2004 Marilyn Wolf MPEG audio rates zInput sampling rates: y32, 44.1, 48 kHz. zOutput bit rates: y23, 48, 64, 96, 112, 128, 192, 256, 384 kbits/sec. zOutput can be mono, dual-channel (bilingual, etc.), stereo.

16. ISPASS 2004 © 2004 Marilyn Wolf Other standards zDolby Digital (AC-3): yUses modified discrete cosine transform. zATRAC (MiniDisc): yUses subband + modified DCT. zMPEG-2 AAC.

17. ISPASS 2004 © 2004 Marilyn Wolf Software implementations zMany standards with complex code. yAbout 1 million lines of code required to implement all major standards. zTechniques are similar but details vary. yVariations from codec to codec. yParameter changes at run time---window size, etc.

18. ISPASS 2004 © 2004 Wayne Wolf MPEG Layer 1 z384 samples/block at all frequencies. yEquals 8 ms at 48 kHz. zOptional masking model. yDriven by separate FFT for better accuracy.

19. ISPASS 2004 © 2004 Wayne Wolf MPEG Layer 1 data frame zBit allocation codes specify word length in each subband. zScale factors give gain for each band. headerCRC bit allocation scale factors subband samples aux data

20. ISPASS 2004 © 2004 Wayne Wolf MPEG Layer 1 encoder Filter bank Choose Scale factor Masking model requantize * FFT mux 0101..

21. ISPASS 2004 © 2004 Marilyn Wolf MPEG Layer 1 decoder 0101.. demux Scale factor ** Step size Inverse filter bank inverse quantize expand

22. ISPASS 2004 © 2004 Marilyn Wolf MP3 zDecoding is easier than encoding, but requires: ydecompression; yfiltering. zBasic CD standard for data discs. zNo standards for MP3 disc file structure: player must understand Windows, Mac, Unix discs.

23. ISPASS 2004 © 2004 Marilyn Wolf Basic principles of MPEG- style compression zDiscrete cosine transform (DCT) used to select perceptually significant information from blocks. zMotion estimation identifies temporal redundancy in frames. zLossless (channel) coding reduces bit rate.

24. ISPASS 2004 © 2004 Marilyn Wolf MPEG-style compression engine motion estimator + DCTQ variable length coder buffer Q -1 DCT -1 + picture store/ predictor

25. ISPASS 2004 © 2004 Marilyn Wolf Spatial frequency in 1D highlow

26. ISPASS 2004 © 2004 Marilyn Wolf DCT zDiscrete cosine transform:  v(k) =  (k)  u(t) cos[  (2t+1)k/2N] z2-D DCT can be computed from two 1-D DCTs. z1-D DCT can be computed in O(N log N) time.

27. ISPASS 2004 © 2004 Marilyn Wolf 8-point DCT flowgraph (Lee) x0 x7 x1 x6 x3 x4 x2 x5 y0 y2 y4 y6 y1 y3 y5 y7 C1 C3 C7 C5 C2 C6 C2 C6 C4

28. ISPASS 2004 © 2004 Marilyn Wolf DCT and quantization DCTQ

29. ISPASS 2004 © 2004 Wayne Wolf DCT coefficient quantization zDCT is used to throw out high spatial frequencies in an 8 x 8 block: 335310000 86420000 6 4 2 0 1 0 2100000 1100000 0000000 0000000 0000000 0000000

30. ISPASS 2004 © 2004 Marilyn Wolf Channel coding zLossless encoding is applied to final bit stream to reduce bit rate. zHuffman-style encoding is used: yvariable-length code for common symbols; yescape code + fixed-length code for less- common symbols.

31. ISPASS 2004 © 2004 Marilyn Wolf Block motion estimation 1 6 3 45 2 1 6 3 45 2

32. ISPASS 2004 © 2004 Marilyn Wolf Motion estmation, cont’d. Two-dimensional correlation of a 16 x 16 macroblock within search range. Best fit:  abs(p b - p s ) Results in a motion vector which shows displacement of macroblock in search area.

33. ISPASS 2004 © 2004 Marilyn Wolf Search process

34. ISPASS 2004 © 2004 Marilyn Wolf ODFS and PLS algorithms

35. ISPASS 2004 © 2004 Marilyn Wolf CBAS and FE2SS Center-biased adaptive search Fast and efficient 2 step search

36. ISPASS 2004 © 2004 Marilyn Wolf 3SS related algorithms z E3SS differs from N3SS in that: 1. A small diamond patter is used instead of a square in the central area 2. Unrestricted search step for the small diamond rather than a single movement for the small square. 3. Test sequences: Coastguard, Football, Salesman, Suzie 4. FS 3SS 4SS N3SS DS E3SS (1) Large search window: 31*31, E3SS performs better in terms of MSE and search points than any other non-full search algorithms (2) Small window: 15*15, E3SS is similar like DS and N3SS

37. ISPASS 2004 © 2004 Marilyn Wolf 4SS related algorithms z4SS 1. Three 5*5 search windows and a final 3*3 window. First step uses 9 points. Second/third step uses three or five points. Final step uses 8 points. 2. Smaller search window 5*5 in the first step of 4SS VS 9*9 in 3SS related algorithms. 3. More regular search pattern than N3SS. 4. 4SS has similar or worse image quality than N3SS but less searching points zOther 4SS related algorithms: E4SS Average Search points: E4SS<4SS<N3SS<3SS MSE performance is similar like N3SS.

38. ISPASS 2004 © 2004 Marilyn Wolf MPEG-1/2 frame types I frame t P frame t+3 BB frame t+1frame t+2