1. MPEG-4: Demos, BIFS, SNHC Visual Lihang Ying Department of Computing Science University of Alberta, Edmonton, Canada These slides are available online: www.cs.ualberta.ca/~lihang/Share/mpeg4

2. Outline MPEG-4 Demos and Overview Demos Overview BIFS-How to Organize MPEG-4 Contents Examples Study Synthetic and Natural Hybrid Coding(SNHC) – Visual Part 2D Mesh Coding 3D Mesh Coding

3. Demos EnvivioTV: http://www.envivio.com/products/etv/content/technical.jsp

4. It’s a plug-in for realplayer, media player or quicktime

5. What can we learn from the demos? MPEG-4 vs MPEG-1/2 Not merely video and audio Interactive Object-based

6. MPEG-4 Overview Why MPEG-4? Interoperability: Create once, use everywhere Multi-network Delivery Scalability Different capacity of different devices

7. What’s MPEG-J? API: org.iso.mpeg.mpegj org.iso.mpeg.mpegj.scene org.iso.mpeg.mpegj.resource org.iso.mpeg.mpegj.decoder org.iso.mpeg.mpegj.net Implement MPEG-4 Decoder conveniently with MPEG-J API Create once, run on all kinds of devices and platforms

8. Interactive  Multi-network Delivery  Decoder: Using MPEG-J  Scalability: Different Capacity  Profile/Level  Not merely audio/video  Object-based  Interoperability

9. Outline MPEG-4 Demos and Overview Demos Overview  BIFS-How to Organize MPEG-4 Contents Examples Study Synthetic and Natural Hybrid Coding(SNHC) – Visual Part 2D Mesh Coding 3D Mesh Coding

10. How to Organize Contents Scene Description Assemble objects into audiovisual scene Scene description format — binary format for MPEG-4 scenes (BIFS) Object Descriptor Describe objects

11. initial object description ES_Descriptor 1 ES_Descriptor 2 scene descriptor stream BIFS update (replace scene) scene description scene description Video Source Audio Source object descriptor stream object descriptor update object descr. object descr. ES_Descr 1 ES_Desc 2 visual stream (base layer) visual stream (e.g. temporal enhancement layer) audio stream ES_ID 1 ES_ID 2 ES_D 1 ES_ID c ES_ID b ES_ID a ES_IDi ES_IDii

12. Scene Description - BIFS Represented by XMT-A Format: Similar to XML Express bitstream syntax in document Enable easy generation of bitstream parser BIFS Examples: …

13. BIFS Example(1) –Trivial Scene(MPEG-2/DVD) Scene Tree Layer2D Sound2D AudioSource Shape Bitmap Appearance MovieTexture

14. BIFS Example(1) –Trivial Scene(MPEG-2/DVD)

15. BIFS Example(2) –Movie with Subtitles

16. BIFS Example(3) –Buttons Event Response

17. Object Description Syntactic Description Language (SDL) Express bitstream syntax in document Enable easy generation of bitstream parser SDL Example: …

18. Syntactic Description Language ObjectDescriptor class ObjectDescriptor extends ObjectDescriptorBase: bit(8) tag=ObjectDescrTag { bit(10) ObjectDescriptorID; bit(1) URL_Flag; const bit(5) reserved=0b1111.1; if (URL_Flag) { bit(8) URLlength; bit(8) URLstring(URLlength); } else { ES_Descriptor esDescr[1..255]; OCI_Descriptor ociDescr[0..255]; IPMP_DescritporPointer ipmpDescriPtr[0..255]; } ExtensionDescriptor extDescr[0..255]; }

19. Summary of Object Descriptor ObjectDescriptor ObjectDescriptorID URL_Flag ES_Descriptor // Elementary Streaming ES_ID, streamDependenceFlag, URL_Flag, OCRstreamFlag, streamPriority, DecoderConfigDescriptor, SLConfigDescriptor, IPI_DescrPointer, IP_IdentificationDataSet, IPMP_DescriptorPointer, LanguageDescriptor, QoS_Decriptor...DecoderConfigDescriptor OCI_Descriptor // Object Content Information ContentClassificationDescriptor, KeywordDescriptor, RatingDecriptor, LanguageDescriptor, ShortTextualDescriptor, ExpandedTextualDescriptor, ContentCreatorNameDescriptor, ContentCreationDataDescriptor, OCICreatorNameDescriptor, OCICreationNameDescriptor, SmpteCameraPositionDescriptor, MediaTimeDescriptor,... IPMP_DescriptorPointer // Intellectual Property Management and Protection Applications of OCI/IPMP–eDonkey’s problems

21. MPEG-4 Objects and Tools Audio Natural Audio Synthetic and Natural Hybrid Coding(SNHC) TTS/Structured Audio/ … Visual Natural Video Object-based/Scalability SNHC 2D/3D Mesh Object/Face and Body Animation Image Text …

22. Outline MPEG-4 Demos and Overview Demos Overview BIFS-How to Organize MPEG-4 Contents Examples Study  Synthetic and Natural Hybrid Coding(SNHC) – Visual Part 2D Mesh Coding 3D Mesh Coding

23. [2D Mesh Coding] Natural Video Coding Block-based textual and motion coding Shape information coding 2D Mesh Coding Designed for video manipulation Natural images and video mapped on 2D meshes Applications: Object tracking, Content-based video retrieval(e.g. motion-based queries), 2D animation, Augmented reality, …

24. Example (a)original frame (b)Mesh generated (c)Text overlaid on video:Text moves along with the fish ’ s meshes

25. Architecture of 2D Mesh Coding

26. 2D Mesh Object Support video coding by moving the vertices of the mesh Topology of the mesh does not change in one session Mesh Data includes: Connectivity: how vertices are connected Geometry: 2D coordinates of vertices Motion: temporal difference of vertices ’ positions

27. 2D Mesh Decoding Scheme

28. Coding of Mesh Motion Mesh Traversal: 1) Start from top-left, breadth-first 2) Right(Next counterclockwise) 3) Left This order remain unchanged(intact) until next I-MOP is decoded Mesh Motion Coding Encoded based on previously encoded two neighboring vertices, e.g.

29. [3D Mesh Coding] 2D Mesh Coding: supports to map natural images and video mapped on 2D meshes 3D Mesh Coding: Represent and compress 3D objects onto which images and videos may be mapped Compress static 3D models, not their animation

30. Functionalities High compression 2%-4% of VRML ASCII file Incremental rendering Building the model with part bitstream Error resilience Suffer less from network errors Hierarchical buildup Scalable bitstream with different resolutions, depending on viewing distance

31. Incremental Rendering

32. Data of 3D Mesh Object Connectivity: how vertices are connected Geometry: 3D coordinates of vertices Photometry Colors Normals Texture

33. Bitstream of 3D Mesh Coding Connectivity Data Vertex graph Triangle tree Triangle Data (Geometry&Photometry) Contains: geometry coordinates, colors, normals, texture coordinates Largest part of the bitstream

34. Bitstream of 3D Mesh Coding Connectivity Data is packed separately and before the Triangle Data. Benefits: Incremental rendering: Could decode Triangle Data incrementally since full Connectivity(topology) Data is already available Shorten the latency Error resilience: Can form 3D structure even with some missing Triangle Data

35. Decoding Scheme of 3D Mesh

36. Vertex Graph

37. Triangle Tree

38. Coding of Geometry and Photometry Data 1) Quantization 2) Differential Coding 3) Adaptive Arithmetic Entropy Coding Code the differential values

39. 3D Mesh Coding Modes Error-Resilience Mode To minimize the impact of errors, divide into partition or packet Render partitions independently Progressive Transmission Mode Scalable coding One base layer One or more enhancement layers Provide Forest Split operations Contains face forest, triangle tree, triangle data

40. Forest Split Operation (a) Cut through the edges of vertex tree (b) Open the dotted line (c) Triangulate the opening to form a triangle tree (d) Refined mesh

41. References Books: Major Reference: Major Reference: Fernando Pereira,Touradj Ebrahimi,The MPEG-4 Book, Prenticle Hall PTR, 2002 Natural Video Coding Technology: Joan L.Mitchell,etc. MPEG Video Compression Standard, Chapman&Hall, 1996 MPEG Official Websites: Overview: http://mpeg.telecomitalialab.com/standards.htm Resources Resources: http://www.m4if.org/resources.php Demos: http://www.envivio.com/products/etv/content/technical.jsp http://www.ivast.com/aboutmpeg4/index.html MPEG-4 Series Slides, Course Presentation of C640/2003 Winter, U. of Alberta: http://www.cs.ualberta.ca/~anup/Courses/604/604_3D.htm

42. The End Acknowledgements Yongjie Liu Michael Closson Questions and Comments?