1. Dalí A Multimedia Software Library Wei Tsang Ooi Brian Smith, Hsi Haye Chan, Matthew Chiu, Sugata Mukhopadhyay, Dan Rubinovitz, Jiesang Song, Steve Weiss.

2. Motivations

3. Current Software Solution Black Box C Source

4. MPEG to RealVideo Black Box Real Video Encoder RGB MPEGReal OpenFile(f) GetNextFrame()

5. MPEG to RealVideo RGB Decoder YUV to RGB Encoder RGB to YUV RealMPEG YUV compressed data OpenFile(f) GetNextFrame() YUV

6. MPEG to RealVideo YUV Decoder YUV to RGB Encoder RGB to YUV RealMPEG compressed data

7. Decode I-Frame only Black Box RGB MPEG ? OpenFile(f) GetNextFrame()

8. Decode I-Frame only RGB MPEG ? Decoder YUV to RGB compressed data YUV OpenFile(f) GetNextFrame()

9. Decode I-Frame only RGB MPEG ? Decoder YUV to RGB I Frame filter compressed data I-frames

10. Others Motivations.. Black Box n Read from network ? n Tune I/O ? n Run on handheld ?

11. Current Software Solution Black Box C Source

12. Dalí is in between.. Black Box C Source Dalí

13. Example

14. Example : n Process a subset of the gray scale version of each I-frame

15. MPEG structures GOP hdrsequence hdrpicture pic hdrpic data

16. What we want to do parse sequence header

17. What we want to do parse sequence header while not eof do - find & parse picture header

18. What we want to do parse sequence header while not eof do - find & parse picture header - if it is an I-frame then - parse the picture into DCT domain - IDCT into spatial domain

19. What we want to do parse sequence header while not eof do - find & parse picture header - if it is an I-frame then - parse the picture into DCT domain - IDCT into spatial domain - take the gray scale version of the frame and process it

20. Abstractions

21. n For input/output n For representing elements in MPEG n For storing images in DCT domain n For storing images in spatial domain

22. BitStream & BitParser BitStream File BitParser

23. n Parse n Encode MPEG Headers MpegGopHdrMpegSeqHdrMpegPicHdrPicture Data n Find n Skip n Dump

24. n Parse n Encode Find MpegGopHdrMpegSeqHdrMpegPicHdrPicture Data n Find n Skip n Dump

25. n Parse n Encode Skip MpegGopHdrMpegSeqHdrMpegPicHdrPicture Data n Find n Skip n Dump

26. n Parse n Encode Dump MpegGopHdrMpegSeqHdrMpegPicHdrPicture Data n Find n Skip n Dump MpegGopHdr

27. n Parse n Encode Parse MpegGopHdrMpegSeqHdrMpegPicHdrPicture Data n Find n Skip n Dump width: 320 height: 240 pic rate: 30 fps etc

28. n Parse n Encode Encode MpegSeqHdr n Find n Skip n Dump width: 320 height: 240 pic rate: 30 fps etc

29. ScImage ScBlock :- { short dc; char index[63]; int ac[63]; } represents 8x8 pixels

30. ByteImage unsigned char 0..255 represents a pixel

31. The Code

32. Example MpegSeqHdr *seqHdr = MpegSeqHdrNew (); MpegPicHdr *picHdr = MpegPicHdrNew (); BitStream *bs = BitStreamNew (65536); BitStream

33. Example BitParser *bp = BitParserNew (); BitParserAttach (bp, bs); BitStream BitParser

34. Example BitStreamReadFromFile (bs, f, 0); BitStream BitParser File MpegSeq Hdr MpegGop Hdr MpegPic Hdr

35. Example MpegSeqHdrFind (bp); BitParser MpegSeq Hdr MpegGop Hdr MpegPic Hdr

36. Example MpegSeqHdrParse (bp, seqHdr); w = (seqHdr->width + 7)/8; h = (seqHdr->height + 7)/8; BitParser MpegSeq Hdr MpegGop Hdr MpegPic Hdr

37. Example ScImage *y = ScNew (w, h); ScImage *u = ScNew (w/2, h/2); ScImage *v = ScNew (w/2, h/2); y u v

38. Example status = MpegPicHdrFind (bp); while (status != NOT_FOUND) { MpegPicHdrParse (bp, picHdr); status = MpegPicHdrFind (bp); }

39. Example status = MpegPicHdrFind (bp); while (status != NOT_FOUND) { MpegPicHdrParse (bp, picHdr); if (picHdr->type == I_FRAME) { } status = MpegPicHdrFind (bp); }

40. Example status = MpegPicHdrFind (bp); while (status != NOT_FOUND) { MpegPicHdrParse (bp, picHdr); if (picHdr->type == I_FRAME) { MpegPicIParse ( bp,seqHdr,picHdr, scy,scu,scv); : } status = MpegPicHdrFind (bp); }

41. Example status = MpegPicHdrFind (bp); while (status != NOT_FOUND) { MpegPicHdrParse (bp, picHdr); if (picHdr->type == I_FRAME) { MpegPicIParse ( bp,seqHdr,picHdr, scy,scu,scv); : } BitStreamFill(bp, f); status = MpegPicHdrFind (bp); }

42. Refill the BitStream BitParser

43. Refill the BitStream BitParser

44. Refill the BitStream BitParser File

45. Memory Mapped BitStream bs = BitStreamMMapReadNew(“tennis.mpg”); File

46. Operate on a subset

47. width: 8 height: 6 x: 0 y: 0 parentWidth: 8 isVirtual: 0 firstByte * Physical ByteImage

48. width: 5 height: 4 x: 1 y: 1 parentWidth: 8 isVirtual: 1 firstByte * Virtual ByteImage

49. Example ScImage *virt = ScClip (y,6,2,20,20); 6 20 blocks

50. Example ScImage *virt = ScClip (y,6,2,20,20); ByteImage *byte = ByteNew (160, 160); ScToByte (virt, byte); 6 20 blocks

51. Implementation

52. n C library n Tcl bindings available

53. Organization n Code are organized into packages n Basic –ByteImage, ScImage, etc. n MPEG –MPEG-1 Video/Audio/System n Vision –Useful vision routines

54. Other Packages n AVI n WAVE n JPEG n GIF

55. Availability n http://www.cs.cornell.edu/dali n open source n binaries for linux/win32/sunos n examples n documentations

56. The End