1. Concepts of Multimedia Processing and Transmission IT 481, Lecture #11 Dennis McCaughey, Ph.D. 20 November, 2006

2. 08/28/2006 IT 481, Fall 2006 2 Broadcast Environment

3. 08/28/2006 IT 481, Fall 2006 3 The MPEG-4 Layered Model Compression Layer Sync Layer Delivery Layer Media Aware Delivery Unaware MPEG-4 Visual MPEG-4 Audio Media Unaware Delivery Unaware MPEG-4 Systems Media Unaware Delivery Aware MPEG-4 DMIF Elementary Stream Interface (ESI) DMIF Application Interface (DAI)

4. 08/28/2006 IT 481, Fall 2006 4 MPEG-4: Delivery Integration of Three Major Technologies Internet, ATM, etc… Cable, Satellite, etc… CD, DVD, etc… The Broadcast Technology The Disk Technology The Interactive Network Technology

5. 08/28/2006 IT 481, Fall 2006 5 MPEG-4: DMIF Communication Architecture

6. 08/28/2006 IT 481, Fall 2006 6 MPEG-4: DMIF Communication Architecture DMIF (Delivery Multimedia Integration Framework) –It is a session protocol for the management of multimedia streaming over generic delivery technologies. –In principle it is similar to FTP. The only (essential!) difference is that FTP returns data, DMIF returns pointers to where to get (streamed) data When FTP is run, –Very first action it performs is the setup of a session with the remote side. –Later, files are selected and FTP sends a request to download them, the FTP peer will return the files in a separate connection. When DMIF is run, –Very first action it performs is the setup of a session with the remote side. –Later, streams are selected and DMIF sends a request to stream them, the DMIF peer will return the pointers to the connections where the streams will be streamed, and then also establishes the connection themselves.

7. 08/28/2006 IT 481, Fall 2006 7 DMIF Computational Model

8. 08/28/2006 IT 481, Fall 2006 8 DMIF Service Activation The Originating Application request the activation of a service to its local DMIF Layer – –a communication path between the Originating Application and its local DMIF peer is established in the control plane (1) The Originating DMIF peer establishes a network session with the Target DMIF peer – –a communication path between the Originating DMIF peer and the Target DMIF Peer is established in the control plane (2) The Target DMIF peer identifies the Target Application and forwards the service activation request – –a communication path between the Target DMIF peer and the Target Application is established in the control plane (3) The peer Applications create channels (requests flowing through communication paths 1, 2 and 3). –The resulting channels in the user plane (4) will carry the actual data exchanged by the Applications. DMIF is involved in all four steps above.

9. 08/28/2006 IT 481, Fall 2006 9DAI Compared to FTP, DMIF is both a framework and a protocol. –The functionality provided by DMIF is expressed by an interface called DMIF-Application Interface (DAI), and translated into protocol messages. These protocol messages may differ based on the network on which they operate. –The DAI is also used for accessing broadcast material and local files, this means that a single, uniform interface is defined to access multimedia contents on a multitude of delivery technologies.

10. 08/28/2006 IT 481, Fall 2006 10DNI The DMIF Network Interface (DNI)- is introduced to emphasize what kind of information DMIF peers need to exchange; It is an additional module ("Signaling mapping" in the figure) takes care of mapping the DNI primitives into signaling messages used on the specific Network. Note that DNI primitives are only specified for information purposes, and a DNI interface need not be present in an actual implementation,.

11. 08/28/2006 IT 481, Fall 2006 11 MPEG-4 Video Bitstream Logical Structure Layer 1 Layer 2

12. 08/28/2006 IT 481, Fall 2006 12 Motion Compensation Three steps –Motion Estimation –Motion-compensation-based-prediction –Coding of the prediction error MPEG-4 defines a bounding box for each VOP Macroblocks entirely within the VOP are referred to as interior macroblocks Macroblocks straddling the VOP boundary are called boundary macroblocks Motion compensation for interior macroblocks is the same as MPEG-1&2 Motion compensation for boundary macroblocks requires padding –Help match every pixel in the target VOP –Enforce rectangularity for block DCT encodeing

13. 08/28/2006 IT 481, Fall 2006 13 MPEG-4: Motion Estimation Block-based techniques in MPEG-1 and MPEG-2 have been adopted to MPEG-4 VOP structure –I-VOP: Intra VOP –P-VOP: Predicted VOP based on previous VOP –B-VOP: Bidirectional Interpolated VOP predicted based on past and future VOP Motion estimation (ME) only necessary for P-VOPs and B-VOPs –Differentially coded from up to three Motion Vectors –Variable length coding used for encoding MVs

14. 08/28/2006 IT 481, Fall 2006 14 MPEG-4 Texture Coding VOP texture information is in luminance and chrominance for I-VOP For P-VOP and B-VOP, texture information represents residual information remaining after motion compensation Standard 8x8 block-based DCT used –Coefficients quantized, predicted, scan and variable length encoded –DC and AC coefficient prediction based on neighboring blocks to reduce energy of quantized coefficients

15. 08/28/2006 IT 481, Fall 2006 15 Bounding Box & Boundary Macroblocks

16. 08/28/2006 IT 481, Fall 2006 16Padding For all boundary macroblocks in the reference VOP –Horizontal Repetitive Padding –Vertical Repetitive Padding For all exterior macroblocks outside the VOP, but adjacent to one or more boundary macroblocks –Extended padding

17. 08/28/2006 IT 481, Fall 2006 17 Horizontal Repetitive Padding Algorithm begin for all rows in Boundary macroblocks in the reference VOP if there exists a boundary pixel in the row for all interval outside the VOP if interval is bounded by only one boundary pixel b assign the value b to all pixels in interval elseif interval is bounded by two boundary pixels b1 and b2 assign the value (b1+ b2)/2 to all pixels in interval end

18. 08/28/2006 IT 481, Fall 2006 18 Vertical Repetitive Padding Algorithm Horizontal algorithm applied to the columns

19. 08/28/2006 IT 481, Fall 2006 19 Original Pixels Within the VOP

20. 08/28/2006 IT 481, Fall 2006 20 Horizontal Repetitive Padding

21. 08/28/2006 IT 481, Fall 2006 21 Vertical Repetitive Padding

22. 08/28/2006 IT 481, Fall 2006 22 Shape Adaptive Texture Coding for Boundary macroblocks

23. 08/28/2006 IT 481, Fall 2006 23Considerations Total number of DCT coefficients equals the number of grayed pixels which is less than 8x8 –Fewer computations than an 8x8 DCT During decoding translations must be reversed so a binary mask of the original shape must be provided

24. 08/28/2006 IT 481, Fall 2006 24 MPEG-4 Shape Coding Binary shape coding Grayscale shape coding

25. 08/28/2006 IT 481, Fall 2006 25 Static Texture Coding

26. 08/28/2006 IT 481, Fall 2006 26 Sprite Coding

27. 08/28/2006 IT 481, Fall 2006 27 Global Motion Compensation

28. 08/28/2006 IT 481, Fall 2006 28 MPEG-4 Scalability Spatial and temporal scalability implemented using VOLs (video object layers) –Base and enhancement layers

29. 08/28/2006 IT 481, Fall 2006 29Scalability There are several scalable coding schemes in MPEG-4 Visual: Spatial Scalability –Spatial scalability supports changing the texture quality (SNR and spatial resolution). Temporal Scalability Object-Based Spatial Scalability. –Extends the 'conventional' types of scalability towards arbitrary shape objects, so that it can be used in conjunction with other object-based capabilities. –This makes it possible to enhance SNR, spatial resolution, shape accuracy, etc, only for objects of interest or for a particular region, which can even be done dynamically at play-time.

30. 08/28/2006 IT 481, Fall 2006 30 Base and Enhancement Layer Behavior (Spatial Scalability)

31. 08/28/2006 IT 481, Fall 2006 31 Two Enhancement Types in MPEG-4 Temporal Scalability 1. Type I: The enhancement-layer improves the resolution of only a portion of the base-layer 2. Type II: The enhancement-layer improves the resolution of the entire base-layer. In enhancement type I, only a selected region of the VOP (i.e. just the car) is enhanced, while the rest (i.e. the landscape) is not. In enhancement type II, enhancement is applicable only at entire VOP level.

32. 08/28/2006 IT 481, Fall 2006 32 Subset of MPEG-4 Video Profiles and Levels Profile and Level Typical scene size Bitrate (bit/sec) Maximum number of objects Total mblk memory (mblk units) Simple Profile L1QCIF64 k4198 L2CIF128 k4792 L3CIF384 k4792 Core Profile L1QCIF384 k4594 L2CIF2 M162376 Main Profile L2CIF2 M162376 L3ITU-R 60115 M329720 L41920x108838.4 M3248960

33. 08/28/2006 IT 481, Fall 2006 33 MPEG-4 Natural & Synthetic Video Coding Synthetic 2D and 3D objects represented by meshes and surface patches –Synthetic VOs are animated by transforms and special-purpose animation techniques –Representation of synthetic VOs based on Virtual Reality Modeling Language (VRML) standard For natural objects, a large portion of materials used in movie and TV production is blue-screened, making it easier to capture objects against a blue background

34. 08/28/2006 IT 481, Fall 2006 34 Integration of Face Animation with Natural Video Three types of facial data: Facial Animation Parameters (FAP), Face Definition Parameters (FDP) and FAP Interpolation Table (FIT) –FAP allows the animation of a 3D facial model available at the receiver –FDP allows one to configure the 3D facial model to be used at the receiver –FIT allows one to define the interpolation rules for the FAP at the decoder

35. 08/28/2006 IT 481, Fall 2006 35 Integration of Face Animation and Text- to-Speech (TTS) Synthesis Synchronization of a FAP stream with TTS synthesizer possible only if encoder sends timing information Decoder TTS Stream Propriety Speech Synthesizer Phoneme/ Bookmark to FAP Converter Face Renderer Compositor Audio Video

36. DVB-H

37. 08/28/2006 IT 481, Fall 2006 37 DVB-H in a DVB-T Network NOKIA

38. 08/28/2006 IT 481, Fall 2006 38 DVB-H Receiver

39. 08/28/2006 IT 481, Fall 2006 39 DVB-H System (Sharing a Mux with MPEG-2 Services)

40. 08/28/2006 IT 481, Fall 2006 40 Detail MPE-FEC

41. 08/28/2006 IT 481, Fall 2006 41 DVB-T/H Transmitter NOKIA

42. 08/28/2006 IT 481, Fall 2006 42 DVB-H Standards Family NOKIA

43. 08/28/2006 IT 481, Fall 2006 43References “MPEG-4 Natural Video Coding - An overview” Touradj Ebrahimi* and Caspar Horne** J. Henriksson, “DVB-H, Standards Principles and Services”, Nokia HUT Seminar T-111.590 Helsinki Finland 2.24.2005