1. MPEG-4 & Wireless Multimedia Streaming
11/12/2018
MPEG-4 & Wireless Multimedia Streaming
CIS 642
Dimosthenis Anthomelidis

2. Overview Math Background MPEG Family MPEG-4 Overview
Packet-Video Technology
11/12/2018

3. Math Background - DCT
Discrete Cosine Transform (DCT) is key method of MPEG compression standard
DCT helps separate the image into parts of differing importance
Similar to DFT: transforms an image from the spatial domain to the frequency domain
2-dimensional DCT on 16x16-pixel sub-blocks of the source picture
11/12/2018

4. Math Background – DCT (2)
A is the input image, A(i,j) is the intensity of the pixel
11/12/2018

5. Math Background – DCT (3)
Coefficients for the output “image” B:
B(k1,k2) is the DCT coefficient
Signal energy lies at low frequencies
These appear in the upper left corner of the DCT
Increasing Horizontal Frequency
11/12/2018

6. MPEG Family Motion Pictures Expert Group (MPEG):
Experts dedicated to standards for digital audio and video
History:
MPEG-1, MPEG-2 have given rise to:
DVD
Digital TV
Digital Audio Broadcasting
MP3 codecs (coder-decoder)
MPEG-4
More to come: MPEG-7 (Content Description)
11/12/2018

7. MPEG-4 Overview Formally ISO/IEC international standard 14496
Audio-visual coding standard
Versions 1 & 2
Builds on success of:
Digital TV, Interactive graphics
Adopts object-based audiovisual representation model
Satisfy:
Authors (reusability, owner rights)
End-users (interaction with content, multimedia to mobile users)
11/12/2018

8. MPEG-4 Parts Part 1: Systems Part 2: Visual Part 3: Audio
Part 4: Conformance Testing
Part 5: Reference Software
Part 6: DMIF (Delivery Multimedia Integration Framework)
Part 7: Optimised software for MPEG-4 tools
11/12/2018

9. Major Forces Scene is modeled as a composition of objects
Coding: units of audio, visual as media objects
Object-oriented paradigm
Integration: natural and synthetic AV objects
Scene is modeled as a composition of objects
Multiplexing, synchronization of data associated with media objects
Interactivity: locally at the receiver or via a back channel
High Compression
Mobility (low bit-rate) & Real-time data
Identification and Protection of intellectual property
11/12/2018

10. Convergence of 3 worlds
Convergence
11/12/2018

11. Functionalities Content-based interactivity Compression
User is able to select one object in the scene
Hybrid natural and synthetic data coding
Compression
Improved coding efficiency
Multiple concurrent data streams
3D natural ‘objects’, virtual reality
Universal access
Robustness in error-prone environments
Content-based scalability
Fine granularity in content
11/12/2018

12. Part 1:Systems
Framework for integrating natural and synthetic components of complex multimedia scenes.
11/12/2018

13. Audiovisual Interactive
Part 1:Systems (2)
Decoding
Primitive
AV Objects
DAI
Audiovisual Interactive
Scene
Composition and
Rendering
Network
TransMux
...
...
Elementary
Streams
FlexMux
Display & local
user interaction
Ex: MPEG-2
Transport
Scene Description
Information
Object Descriptor
11/12/2018

14. Systems Structure DAI ESI Composition 11/12/2018 TransMux Layer
FlexMux
Tool
Sync.
Compression
Composition
DAI
ESI
11/12/2018

15. Media Objects
Content-based AV representation
AVO (AV objects)
VOC (Video Object Component), AOC(Audio OC)
User may access it
AV scene:
composition of several media objects organized in hierarchical fashion
Leaves: primitive media objects
Still images, Video objects etc
Objects are placed in elementary streams (Ess)
VOP (Video Object Plane): 2D VOC time sample with arbitrary shape. Contains motion parameters, shape info, texture data
11/12/2018

16. Media Objects (2) Sprites: used to code unchanging backgrounds
A scalable object can have an ES for basic quality info plus one or more enhancement layers (Video Object Layer)
Visual objects in a scene are described mathematically and given a position in 2D or 3D space
Object descriptor identifies all streams associated to one media object: informs the system which ESs belong to an object
It has its own ES
BIFS (Binary Format for Scenes): language for describing and dynamically changing the scene. Borrows concepts from VRML.
11/12/2018

17. MPEG-4 scene
11/12/2018

18. Composition
Task of combining all of the separate entities that make up the scene.
Multimedia scenes are conceived as hierarchical structures represented as a graph. Each leaf is a media object. Graph structure isn’t necessarily static.
Composition info is delivered in one elementary stream
11/12/2018

19. Multiplex (1) 3-layer multiplex:
Sync Layer: adding info for timing and synchronization
FlexMux layer: multiplexing streams with different characteristics
Transmux Layer: adapting the multiplexed stream to the particular network characteristics
Elementary streams are packetized adding headers with timing info (clock references) and synchronization data (timestamps). They make up the synchronization layer
11/12/2018

20. Multiplex (2)
Flexible multiplex layer: intermediate multiplex layer. Group together several low-bit-rate streams (with similar QoS requirements).
Transport multiplex layer: it is specific to the characteristics of the transport network. No specific transport mechanism is defined:
Existing transport formats: ATM, RTP suffice
11/12/2018

21. Multiplex (3)
11/12/2018

22. Multiplex (4)
11/12/2018

23. Synchronization layer
Associate timing and synchronization
Elementary streams (ES) consist of access units: portions of the stream with a specific decoding and composition time.
ES are split into SL packets, not necessarily matching the size of the access units.
A header attached contains:
Sequence number
object clock reference- a time stamp used to reconstruct the time base for the object (speed of the encoder clock)
Decoding time stamp- identify the correct time to decode the access unit
Composition time stamp- identify the correct time to render a decoded access unit
11/12/2018

24. MP4 File Format
Reliable way for users to exchange complete files of MPEG-4 content
11/12/2018

25. MPEG-J MPEG-4 specific subset of Java
Defines interfaces to elements in the scene, network resources, terminal resources
Personal Profile: lightweight package for personal devices
Network
Scene
Resource
11/12/2018

26. Part 2 – Visual “rectangular” video objects Arbitrary shaped objects
Binary shape: an encoded pixel either is or is not part of the object in question (on/off). Useful for low-bit rate environments
Alpha shape: for higher-quality content each pixel is assigned a value for its transparency
11/12/2018

27. Visual (Cont’d)
MPEG-2 defines the decoding process. Encoding processes are left to the marketplace.
Provide users a new level of interaction with visual contents
Manipulate objects
Error robustness
Scalability: minimum subset that can be decoded – Base layer. Each of the other bitstreams is called enhancement layer
Optimized for 3 bitrate ranges:
< 64 kbps ( wireless scenario)
kbps
384-4 Mbps
11/12/2018

28. Error Resilience
Very important for mobile communications because of error burstiness
Resynchronization
Errors are localized through the use of resynchronization markers. These markers can be inserted in the bitstream. If error then decoder skips data till next marker and restarts from that point. Insertion after constant #coded bits - “video packets”.
Data partitioning - motion info seperated from texture info
If error in texture bits use decoded motion info.
Header Extension code: redundant info, vital for correct decoding video
Reversible Variable Length code: codewords decoded in forward and backward. If error it’s possible to decode portions of the corrupted bitstream in reverse order.
11/12/2018

29. Scalability Use of multiple VOLs (base layer-enhancement layer)
Spatial scalability
Enhancement layer improves spatial resolution
Temporal scalability:
Offers higher frame rate. Improves smoothness of motion (temporal resolution)
Generalized framework: a scalability preprocessor implements the desired scalability. For spatial scal., it down-samples the input VOPs to produce the base layer which is encoded by base-layer encoder. The reconstructed base layer is up-sampled by a mid-processor. The difference from original VOP is the input for enhancement encoder.
11/12/2018

30. Hold that smile
Map images onto computer-generated shapes.
11/12/2018

31. Applications Criteria Non real-time, Non-symmetric, Non-Interactive
Timing constraints
Real-time or non real-time
Symmetry of transmission facilities
Interactivity
Non real-time, Non-symmetric, Non-Interactive
Multimedia broadcasting for mobile devices
Manufacturers of mobile equipment and providers of mobile services have been adopting MPEG-4
11/12/2018

32. Mobile Interactive Multimedia
Mobile computing= portable computer + wireless comm.
Limitations:
Limited computation capacity
Narrow bandwidth
Unreliable channel
Requirements
High Compression
Error resilience
11/12/2018

33. Thinking small
Moving video possible at very low bit-rates for mobile devices. Even at 10kb/s (GSM’s data rate)
Use of scalable objects: providers need encode clips only once. A base layer conveys all the info in some basic quality
Already existing MPEG-4 hardware decoders, encoders to bring video to mobile devices (e.g Toshiba)
11/12/2018

34. Packet Video Technology
Visual communication “anywhere – anytime”
Compliant with MPEG-4 visual spec.
Optimized for single rectangular objects based on motion compensation and DCT coding of macroblocks
Scalability: allows subsets of a single bitstream to go to a receiver. You encode once and deliver to multiple decoders with different capabilities
11/12/2018

35. Video Encoding
11/12/2018

36. Rate Control Rate control: multiple layer bitstreams
Temporal scalability – adding enhancement to a base layer
Spatial scalability – adding enhancement with differential images
11/12/2018

37. Video Decoding
11/12/2018

38. PV error-resilient decoding
11/12/2018

39. Products Software-based solutions
PVPlayer: decoder application for rendering
PVServer: server application
PVAuthor: encoder, create MP4 file format bit stream
11/12/2018

40. Conclusion
Extensive tests show that MPEG-4 achieves better or similar image qualities at all bitrates targeted, with the bonus of added functionalities.
11/12/2018

41. References http://www.cselt.it/mpeg/ http://www.packetvideo.com
11/12/2018