1. E0262 MIS - Multimedia Playback Systems Prof Pallapa Venkataram, Electrical Communication Engineering, Indian Institute of Science, Bangalore – 560012, India Multimedia Streams Synchronization

2. E0262 MIS - Multimedia Playback Systems ● Synchronization in multimedia systems referes to the temporal relations between media stream data units in multimedia system. ● Time dependent media data units: Temporal relations between consecutive units of media stream exist. ● Time independent media data unit is any kind of traditional media like text and image.

3. E0262 MIS - Multimedia Playback Systems Classification of Media Use

4. E0262 MIS - Multimedia Playback Systems Multimedia Synchronization ● The process of maintaining the temporal order of one or several media streams ● Relations relevant for audio and video exchange: – Intra-stream synchronization: Voice samples and video frames need to arrive in time at the receiver before display or playout time to maintain the continuity of playback. – Inter-stream synchronization is needed to present voice and video samples in a certain relation. – Spatial synchronization: Participants in a multimedia conference should receive audio-visual data at the same time although geographically distributed. ● Life & Synthetic synchronization

5. E0262 MIS - Multimedia Playback Systems Basic Synchronization Issues ● Content Relations: define a dependencies of media data units on some data ● Spatial Relations: usually as layout relationships define the space used for the presentation of a media data units on an output device at a certain point of time in a multimedia presentation. ● Temporal Relations: define the temporal dependencies between the more than one media stream data.

6. E0262 MIS - Multimedia Playback Systems Sources of Asynchrony ● Different delays: Assumption of independent network connections imposes different delays. ● Network jitter: asynchronous data transfer destroys synchrony. ● End-system jitter: packetizing and depacketizing of media data units with different size due to encoding introduces jitter as well as passing media units through the lower protocol layers. ● Clock drift between the clocks in the servers and in the client is present because we do not assume global clocks. ● Change of the average delay: the synchronization scheme has to be adaptive with respect to a change of the average delay. ● Server drop outs: due to process scheduling are a realistic assumption when using non-real-time operating systems.

7. E0262 MIS - Multimedia Playback Systems Types of Synchronization ● Time based synchronization ● To synchronize and to disseminate multimedia data using time as a major parameter ● Synchronization is achieved by mean of electronic time signals generated ad hoc by devices directly connected to a primary time standard and transmitted by radio or by cable. ● Time signals classes: video or sound signals (called right time signals) for low accuracy manual synchronization, digital signals for automatic synchronization with medium-high accuracy ● Passive synchronization techniques: Do not use a time signal generated ad hoc, instead it is used a signal generated for other purpose

8. E0262 MIS - Multimedia Playback Systems Types of Synchronization ● Delay based synchronization – Schemes compensate for these delays by computing well-defined starting times for each stream server of the application. – Allow to initiate the synchronized playback of a media stream that is composed of several multimedia streams. ● Jitter based synchronization – jitter is experienced by media units traveling from the source to the destination. – To smoothen out jitter, elastic buffers are required. – These schemes guarantee a smooth playback of the stream – very low buffer requirements.

9. E0262 MIS - Multimedia Playback Systems Intra-Stream Synchronization ● Audio and video streams are isochronous in nature. ● processing and network delay jitter (i.e., the variance in delay); ● variations in rates of recording and playback; and ● unreliable transmission of stream data units. ● Solutions: – buffer monitoring – feedback technique – global clock

10. E0262 MIS - Multimedia Playback Systems Inter-Stream Synchronization ● A temporal relationship may exist at the source between multiple continuous streams ● multiplexing of streams ● aggregation in one data structure ● global clocks ● synchronization marker ● synchronization channel ● feedback technique

11. E0262 MIS - Multimedia Playback Systems Spatial Synchronization ● all participants in the conference receive the audio and video data at the same time, to maintain a fair conference ● Based on global clocks synchronization channel or feedback techniques as proposed for inter-stream synchronization. ● When global clocks are available, mechanisms based on these clocks can achieve the most accurate spatial synchronization

12. E0262 MIS - Multimedia Playback Systems Synchronization Handlers ● roles : to identify the stakeholders involved, as well as the prohibition and policy concepts. ● Roles of these components in handling synchronization – Service provider – Network provider – End-user roles

13. E0262 MIS - Multimedia Playback Systems Service Provider ● Intra-stream synchronization: SP manipulates incoming audio/video streams so that outgoing streams are within the 10 ms jitter boundary; ● Inter-stream synchronization: SP will manipulate incoming audio/video streams so that related outgoing audio and video streams are within the -20 ms and +40 ms range; ● Spatial synchronization: SP is responsible for ensuring that outgoing audio/video streams are played out simultaneously at the multiple users within the 0.25 s boundary.

14. E0262 MIS - Multimedia Playback Systems Network Provider ● Compulsory network service: a transportation service with a deterministic service is provided ● Statistical reliable network service: a transportation service with a certain percentage of QoS violations is provided. ● Best effort network service: the request from a client for a certain transport service is evaluated against the current network traffic.

15. E0262 MIS - Multimedia Playback Systems End-User ● With respect to the display of audio and video it is important that the following requirements to be met: ● lip-synchronization is a well known requirement and should be in the -20 ms to +40 ms range; ● audio or video jitter should be within the range of 10 ms; ● loss of video frames or audio samples is tolerable when less than 1% of the total sent; ● spatial synchronization should be in the range of -0.25 s to +0.25 seconds.

16. E0262 MIS - Multimedia Playback Systems End-User ● compulsory end-user service: in this case the synchronization requirements must be met. ● statistical reliable end-user service: a certain percentage of violations of the synchronization requirements is allowed. ● best effort end-user service: possibilities to full the synchronization requirements are based on current processing and storage activities. ● User policies are often application dependent.

17. E0262 MIS - Multimedia Playback Systems Synchronization Methods ● Startup Synchronization ● Constant delay and zero jitter first n media units where t i indicates expected arrival time of the MDU i at the client-site and t 0 is the playout start time of the I o th stream ● Evaluation phase: round trip delays for each of the stream of the application are calculated, while ● Synchronization phase: the starting time for each server is calculated and transmitted back to the servers.

18. E0262 MIS - Multimedia Playback Systems Synchronization Methods ● Startup Synchronization ● Beginning of first phase Max of the round trip delay for all n substreams ● Second phase ● Difference between the arrival times of arbitrary media units

19. E0262 MIS - Multimedia Playback Systems Example of Startup Synchronization

20. E0262 MIS - Multimedia Playback Systems Example of Startup Synchronization

21. E0262 MIS - Multimedia Playback Systems Intra-Stream Synchronization ● Jitter based Intra-Stream Synchronization Worst case estimate for error and buffer req. ● Interdependencies:

22. E0262 MIS - Multimedia Playback Systems Inter-Stream Synchronization ● Inter-stream synchronization example that shows temporal relations in a multimedia presentation including audio, video, animation, and picture frames

23. E0262 MIS - Multimedia Playback Systems Inter-Stream Synchronizations ● Live Synchronization: Maintains the temporal relations as they existed during the capturing process ● Synthetic Synchronization: Temporal relations are artificially specified.

24. E0262 MIS - Multimedia Playback Systems Synchronization Reference Model

25. E0262 MIS - Multimedia Playback Systems

27. Synchronization Models ● The Temporal Synchronization Model

28. E0262 MIS - Multimedia Playback Systems

29. Synchronization Models ● Unix Synchronization Model

30. E0262 MIS - Multimedia Playback Systems

32. Object Data Stream Synchronization Technique ● Synchronization point is a point held by all participating media streams needing to be synchronized ● Synchronization points defined on the composite stream specify the places that synchronous presentation must be checked and maintained. ● Let ∆s i denote the maximum time interval that media m i can skip and d i denote the current delay time interval with the presentation of media stream m i. We then have the following situations:

33. E0262 MIS - Multimedia Playback Systems Deadline based Scheduling for Sync ● n MDUs in a multimedia document, set of m channels in network ● Effective bandwidth rate c j per channel and transit delay j, aggregate capacity is not sufficient to provide inter-stream and intra-stream synchronization without pre-fetching. ● If the i th MDU and playout deadline  i is scheduled for transmission on the j th channel at time  j, according to some scheduling policy, then its arrival time, a i, at the user site becomes

34. E0262 MIS - Multimedia Playback Systems Feedback Techniques for Sync ● The resynchronization policies are: ● Conservative: This scheme reacts only when playout of different streams is guaranteed to be asynchronous, ● Aggressive: In this approach the server reacts as soon as there is even a slight chance that playback is asynchronous ● Probabilistic: The server reacts on the average (assuming the network delay distributions and playout rate variations are known).

35. E0262 MIS - Multimedia Playback Systems Virtual Local Time ● Virtual local time ● Does not rely on a synchronized network clock, which is not always feasible to procure on the Internet, ● Can be implemented with a minimum overheads, and ● Can produce a reasonably good quality even when the network delay jitter is very high.

36. E0262 MIS - Multimedia Playback Systems Different Playout Strategies

37. E0262 MIS - Multimedia Playback Systems

38. Synchronized multicast media streaming framework