1. Dali Virtual Machine Wei Tsang Ooi (With Brian, Sugata, Tibor, Steve, Haye, Matthew, Dmitriy)

2. Motivations zWe know how to encode, transmit and store multimedia data today zPeople start looking into ways to process multimedia data

3. Video Editing zConcat, cross-fade, overlay, cut and paste

4. Gateway zConvert from one format to another zChange compression rate transcoder (format A) format B input video output video

5. DBMS “ Find all movies in the database that contain this image “

6. Set-top Boxes zSubtitling and close captioning zMixing of incoming movies

7. Lecture Browser zMatch video to slide zSwitch between video streams

8. Current Solutions zBlack box C code (mpeg_play) yHard to reuse/adapt/break apart yUnpredictable performance zStandard library (PPM, IJG) yLeast common denominator (RGB for all) xVideo frames --> RGB --> gray

9. History zRivl - high level scripting language for multimedia processing zStill suffers from “black box” problem

10. Experience zReal cost of processing yInherently complex operations xProjective transform yLayered operations xVideo decoding ySimple operations but lots of data xCopy

11. Dali Architecture Dali Code Dali Compiler DVM Code Dali Virtual Machine User Expert User MMXTriMedia C

12. Today’s Talk DVM Code Dali Virtual Machine

13. DVM Code : Design Goal zSmall number of composable abstractions zSimple, predictable opcodes zHigh performance zTarget for compiler zEasy to extend

14. Abstractions zByte image zBitstream zMPEG zJPEG zBit masks zPCM

15. Abstractions zByte image zBitstream zMPEG zJPEG zBit masks zPCM

16. Byte Image zTwo dimensional array of byte zCan be either physical or virtual physical image virtual image

17. Byte Image Can Represent zGray scale image zRGB image zYUV image zAlpha channel

18. Example Code : Fractal

19. smaller = byte_new (neww, newh); byte_shrink_2x2(image,smaller); image smaller neww newh

20. image target = byte_clip(image,0,0,dx,2*dy); byte_set(target,0); target = byte_clip(image,3*dx,0,dx,2*dy); byte_set(target,0); target dx 2dy

21. target = byte_clip(image, dx, 0, neww, newh); byte_copy(smaller, target); target image smaller newh neww dx

22. target = byte_clip(image, 2*dx, 2*dy, neww, newh); byte_copy(smaller, target); target = byte_clip(image, 0, 2*dy, neww, newh); byte_copy(smaller, target); image smaller neww newh dx 2dy

23. DVM Code Dx = 0.25 * byte_width(image); dy = 0.25 * byte_height(image); neww = 0.5 * byte_width(image); newh = 0.5 * byte_height(image); smaller = byte_new (neww,newh); for (i = 0; i < n; i++) { byte_shrink_2x2(image,smaller); Target = byte_clip(image,0,0,dx,2*dy); byte_set(target,0); target = byte_clip(image,3*dx,0,dx,2*dy); byte_set(target,0); target = byte_clip(image, dx, 0, neww, newh); byte_copy(smaller, target); target = byte_clip(image, 2*dx, 2*dy, neww, newh); byte_copy(smaller, target); target = byte_clip(image, 0, 2*dy, neww, newh); byte_copy(smaller, target); }

24. General Dali Strategies zSpecific instruction ybyte_shrink_2x2, byte_shrink_4x4, etc zExplicit memory allocation ybyte_new zReduce data ybyte_clip zComposable abstraction ybyte image

25. Performance run fractal with n = 4 on 800x600 gray scale image about 21 frames/sec for 320x240 video

26. Abstractions zByte image zBitstream zMPEG zJPEG zBit masks zPCM

27. MPEG zGetting important and pervasive zComplex format zComplex code (mpeg_play) zMost decoders provides GetNextFrame() interface zRandom access, direct transfer difficult

28. Abstraction for MPEG Video... seq hdr gop hdr gop hdr gop seq end

29. Abstraction for MPEG Video... seq hdr gop hdr gop hdr gop seq end... pic hdr pic hdr pic

30. DVM Code Examples  Skip to the n-th frame for (i = 0; i < n; i++) { mpeg_pic_hdr_find(bs); mpeg_pic_hdr_skip(bs); }

31. DVM Code Examples  Skip to the n-th frame for (i = 0; i < n; i++) { mpeg_pic_hdr_find(bs); mpeg_pic_hdr_skip(bs); }

32. DVM Code Examples  Skip to the n-th frame for (i = 0; i < n; i++) { mpeg_pic_hdr_find(bs); mpeg_pic_hdr_skip(bs); }

33. DVM Code Examples  Skip to the n-th frame for (i = 0; i < n; i++) { mpeg_pic_hdr_find(bs); mpeg_pic_hdr_skip(bs); }

34. inbs1 inbs2 outbs hdr gop Concat 2 Video Sequences

35. DVM Code zConcat two sequences While not end_of(inbs1) { gop_hdr_dump(inbs1, outbs); gop_dump(inbs1, outbs); } While not end_of(inbs2) { gop_hdr_dump(inbs2, outbs); gop_dump(inbs2, outbs); }

36. Performance Analysis zDecode MPEG sequences to PPM zNo output zRun on 3 different sequences

37. Performance

39. MPEG Strategies zExpose structure yGop header, gop instead of just frames zBreak up layer yParse, decode, color space conversions

40. Show Time ! zDecode a MPEG sequence zFor each frame scale it to “stamp” size zCreate a contact sheet

41. Abstractions zByte image zBitstream zMPEG zJPEG zBit masks zPCM

42. What Is Bitstream ?  Skip to the n-th frame for (i = 0; i < n; i++) { mpeg_pic_hdr_find(bs); mpeg_pic_hdr_skip(bs); }

43. Bitstream parser 1011001 chunk of memory

44. Mode of Operations zFile i/o static memory fread/fwrite parser

45. Mode of Operations zNetwork static memory socket parser

46. Mode of Operations zMemory map parser entire file

47. inbsvideo audio mpeg system stream

48. filter videobs inbsvideo audio mpeg system stream

49. filter videobs inbsvideo audio parser mpeg system stream

50. Bitstream Strategies zPredictable performance yExplicit I/O

51. Recap: Current Solutions zBlack box C code (mpeg_play) yHard to reuse/adapt/break apart yUnpredictable performance

52. Recap: Cost of Processing zInherently complex operations zLayered operations zSimple operations but lots of data

53. How We Solve the Problems zDifficult to reuse/adapt/break apart yMinimum set of abstractions yAbstractions are simple ySimple reusable opcodes zUnpredictable performance yExplicit I/O yExplicit memory allocations

54. How We Solve the Problems zInherently complex operations ySpecial case yClipping and masking zLayered operations yExpose structure zSimple operations but lots of data ySpecial case yClipping and masking

55. zExtending with new formats : JPEG, WAV, AVI etc zTriMedia chip implementation zApplication : lecture browser Work in Progress

56. Future Work zMultithreading zMMX implementation zCompiler