1. MPEG-4 streams Comp- ress Comp- ress Comp- ress Comp- ress D E L I V E R Y I N T E R F A C E (DAI) Comp- ress Scene Des. decomp- ress decomp- ress decomp- ress decomp- ress D E L I V E R Y I N T E R F A C E (DAI) decomp- ress COMPOSITORCOMPOSITOR Obejct-des

2. Synchronization of elementary streams with the scene

3. Synchronization of multiple elementary streams Based on two well known concepts _______ references – Convey the speed of the encoder clock _________ stamps – Convey the time at which an event should happen Time stamps and clock references are – defined in the _____________ model – conveyed on the ______ layer

4. System Decoder Model

5. Object-based compression and delivery Comp- ress Comp- ress Comp- ress Comp- ress D E L I V E R Y I N T E R F A C E (DAI) Comp- ress Scene Des. decomp- ress decomp- ress decomp- ress decomp- ress D E L I V E R Y I N T E R F A C E (DAI) decomp- ress COMPOSITORCOMPOSITOR

6. System Decoder Model decoder Composition Memory CM1 Decoder Buffer DB1 D E L I V E R Y I N T E R F A C E (DAI) COMPOSITORCOMPOSITOR decoder Composition Memory CM1 Decoder Buffer DB1 decoder Composition Memory CM1 Decoder Buffer DB1 decoder Composition Memory CMn Decoder Buffer DBn

7. System Decoder Model Ideal model of the decoder behavior – Instantaneous ______________ Incorporates the__________ model – Decoding & composition time Manage decoder buffer resources – Useful for the _______ – Ignores delivery ______ Designed for a rate-controlled “______” scenario – Applicable also to flow-controlled “pull” scenario Defines composition memory (CM) behavior – A random access memory to the current composition unit – CM resource management not implemented

8. So, how does synchronization work?

9. Synchronization of elementary streams with time events in the scene description How are time events handled in the scene description? How is this related to time in the elementary streams? Which time base is valid for the scene description?

10. Time line (“object time base”) for the scene Scene description stream with time stamped BIFS access units Object descriptor stream with pointers to all other streams Video stream with (decoding & composition) time stamps Audio stream with (decoding & composition) time stamps Alternate time line for audio and video

14. Where are time stamps carried?

15. The synchronization layer Synchronization layer (short: sync layer or SL) SL packet = one packet of data consists of header and __________ Indicates boundaries of __________ units – AccessUnitStartFlag, AccessUnitEndFlag, AULength Provides consistency checking for ____ packets Carries object _______ reference (OCR) stamps Carries decoding and composition time stamps (DTS, CTS)

16. Elementary Stream Interface Compression Layer SL-Packetized stream Sync Layer Elementary stream Access UnitAUAccess UnitAU

17. Elementary Stream Interface Elementary stream Access UnitAUAccess UnitAU Elementary stream Access Unit Elementary stream AU

18. Elementary Stream Interface Elementary stream Access Unit ENCODERENCODER This guy knows about Access unit _________ Random access points Desired __________ time The _________ time

19. Elementary Stream Interface Access Unit ENCODERENCODER Interface may be hidden in the Encoder Access Unit

20. How is the sync layer designed?

21. The sync layer design SL Packet Access units are conveyed in SL packets

22. The sync layer design SL PSL Packet SL P Access units may use more than one SL packet

23. The sync layer design SL packets have a header to encode the information conveyed through the ESI SL packets that don’t start an AU have a smaller header SL PSL Packet SL P

24. How is the sync layer designed? As flexible as possible to be suitable for – a wide range of ______________ – a wide range of different media __________s Time stamps have – variable length – variable _______________ Same for clock reference (OCR) values – OCR may come via another stream Alternative to time stamps exists for lower bitrate Indication of ______ time and ________ of units (accessUnitDuration, compositionUnitDuration)

25. How is the sync layer flexibility signaled? SLConfigDescriptor within ES_Descriptor Determines all variable parameters Allows to select from predefined _________s  Must be known to parse an SL-packetized stream

26. SLConfigDescriptor syntax example class SLConfigDescriptor { uint (8) predefined; if (predefined==0) { bit(1) useAccessUnitStartFlag; bit(1) useAccessUnitEndFlag; bit(1) useRandomAccessPointFlag; bit(1) usePaddingFlag; bit(1) useTimeStampsFlag; uint(32) timeStampResolution; uint(32) OCRResolution; uint(6) timeStampLength; uint(6) OCRLength; if (!useTimeStamps) {................ SDL- Syntax Description Language

27. How to ship the SL packets?

28. Wrap SL packets in a suitable layer! SL Packet e.g. FlexMux e.g. RTP/UDP/IP e.g. MPEG-2 TS

29. MPEG-4 stream delivery (a sneak preview to DMIF ;-)

30. MPEG-4 delivery framework

31. The delivery layer The “delivery layer” is an abstraction to refer generically to – any ________ multiplex (a. k. a. file format) – any transmission __________ – and the associated control functionality The DMIF (Delivery Multimedia Integration Framework) part of MPEG-4 describes the _________ to any multiplex (DAI - DMIF Application Interface) Adaptations of MPEG-4 to, e.g., RTP, MPEG-2, H.223 are being defined by the responsible group (with MPEG help)

32. Adaptation of the sync layer syntax “Wrapping” of SL packets was the original design goal MPEG-2 TS adaptation (Amendment 7 to MPEG-2 Systems) and FlexMux encapsulation really work like that RTP encapsulation still in development – “Payload formats” specify the adaptation – They may use “Header compression”, i.e., – Syntax elements of SL are replaced by their RTP counterparts e.g., composition time stamp --> RTP time stamp The MPEG-4 file format stores elementary streams – conversion to SL-packetized streams during playout (if needed)

33. Multiplex of elementary streams

34. Not a core MPEG task Just respond to specific needs for MPEG-4 content ______________ – Low delay – Low overhead – Low complexity This prompted the design of the “___________” tool One ___________ file format desirable – This lead to the design of the MPEG-4 file format

35. The MPEG-4 content access procedure

36. MPEG-4 content access procedure Locate an MPEG-4 content item (e.g. by URL) and connect to it How? – Via the _______________________ – Reminder: This abstraction is being mapped to actual signaling protocols (RTSP, MPEG-2 PSI, etc.) Retrieve the _____________________ This Object Descriptor points to _____ + _____________ – Open these streams via DAI The Scene Description points to other streams through __________ Descriptors – Open the required streams via DAI Start playing!

37. A second attempt – more graphics