nus.soc.cs5248 Ooi Wei Tsang 1 Intermedia Synchronization Ooi Wei Tsang

2 Overview Why we need to sync? How close must we sync? How to sync? How to measure sync? Applications

3 Synchronization IntrastreamInterstream Local playbackNetworked

4 Problem Ensure that audio/video captured at the same instance at the source, are displayed at the same instance at the receivers.

nus.soc.cs5248 Ooi Wei Tsang 5 Why we need Synchronization?

6 Why not Pre-sync? V AVA SenderReceiver

7 Seperating A/V Flows SenderReceiver V A V A SenderReceiver Can receive audio without video Can prioritize audio over video

8 May Come from Different Sources UNLV Access Grid SenderReceiver V A V A SenderReceiver

9 or Displayed at Different Sources SenderReceiver V A SenderReceiver A V

10 More Complications Different decoding time for video and audio Network delay and jitter Different clocks

11 Bottom Line Must synchronize at the receiver

nus.soc.cs5248 Ooi Wei Tsang 12 Synchronization Requirements

13 Reference Human Perception of Jitter and Media Synchronization by Ralf Steinmetz IEEE JSAC 14(1) 1996

14 Results “Badness” A/V offset -80ms+80ms

15 What matters Video ahead of audio vs. audio ahead of video Size of head Occupancy

16 What does not matter Language Speaking or Singing Hobby Speed

17 Conclusion -80ms to +80ms acceptable -160ms to +240ms noticable but not annoying - audio > video + video > audio

nus.soc.cs5248 Ooi Wei Tsang 18 How to Synchronize?

19 Receiver needs Exact time a sample is generated Each packet needs a timestamp

20 RTP Protocol Media Timestamp RTP Hdr

21 But.. Media timestamp has a random offset

22 RTCP Protocol : Media Timestamp NTP Timestamp RTCP Hdr :

23 NTP Network Time Protocol Synchronize clock up to few milliseconds

24 Sender Audio Sender Video Receiver Audio Receiver Video Audio RTCP Video RTCP Audio RTP Video RTP

25 Sender Audio Sender Video Receiver Audio Receiver Video Audio RTCP Video RTCP Audio RTP Video RTP Synchronization Delay

26 Multiple Receivers Everyone must have the same synchronization delay

27 How to set Sync Delay? Receivers tell each other their network delay Synchronization Delay = max {Network Delays} + Buffer

28 Buffer Control

29 No Delay No Jitter

30 Delay, No Jitter

31 Delay, Jitter

32 If Buffer too Full/Empty Send message “Play faster!” or “Play slower!” to other receivers

33 Must Synchronize! Send message “Play faster at time T!” or “Play slower at time T!” to other receivers

nus.soc.cs5248 Ooi Wei Tsang 34 How do we Measure Synchronization?

35 Reference A Method and Apparatus for Measuring Media Synchronization by Brian Smhmidt etc. NOSSDAV 1995

36 System under Test Measurement System Generator

nus.soc.cs5248 Ooi Wei Tsang 37 Beyond Video Conferencing

38 HQ Video and Audio Multiple Channels Multiple Displays

39 Network Musical Performance BOS SF BOS SF ACM Multimedia ‘94

40 NASDAQ Synchronize display of stock quotes