MP4 help session Hoang Nguyen

Outline Scenarios Media Synchronization requirement FF/RWND vs. Seek Media synchronization GUI Demo steps (posted next Monday)

Scenario A Server 1 Dispatcher Client 1 Server 2 audio1 file video1 file audio2 file video2 file PLAY audio1/video1 or PLAY audio2/video2 Audio 2 streaming Video 2 streaming

Scenario B Server 1 Dispatcher Client 1 Client 2 Server 2 audio1 file video1 file audio2 file video2 file PLAY audio1/video1 PLAY audio2/video2

Scenario C (optional) Server 1 Dispatcher Client 1 Client 2 audio1 file video1 file audio2 file video2 file PLAY audio1/video1 PLAY audio2/video2

Scenario D (optional) Server 1 Dispatcher Client 1 Client 2 audio1 file video1 file PLAY audio1/video1 audio1 file video1 file

Media synchronization requirement Clips will be provided (1 week before the demo) –E.g. 1) An interview (lipsync), 2) A music clip Each clip consists of an audio stream and a video stream stored in separate files The demo will show how to –Upload/Manipulate those files (GUI or command line) –Play a synchronized clip per corresponding video file and audio file

How to create these clips? Find an video file (e.g. youtube) –E.g.: Download it (e.g. kissyoutube) –E.g.: Extract audio –E.g.: ffmpeg -i obama.mp4 -vn -acodec pcm_s16le -ar ac 2 obama.wav Extract video –E.g.: ffmpeg -i obama.mp4 -vcodec mjpeg obama.mjpeg

FF/RWND vs. Seek Only implement either FF/WRND or SEEK FF/Rewind: –requires skipping frames (similar to MP1) On PLAY  FF/RWND: –The client works on the local buffer first –The client sends the request to the server –The server sends 1-second frames for buffering FF/RWND: step of 2 67

FF/RWND vs. Seek Seek: –requires random access to frames ( av_seek ) –May require a faster/shorter buffering i Seek to frame ith

Media Synchronization Audio and video files must be able to clearly demonstrate synchronization –Speeches and singing are good examples requiring lip synchronization Audio is leading time synchronization stream –Video (and any other stream) should be synchronized with respect to audio Recommend synchronization point each second –Might need to speed up or slow down video relative to the audio

Media Synchronization audio stream 3 samples/sec video stream 2 frames/sec Synchronization Points audio = 3 video = 2 audio = 6 video = 4 audio = 9 video = 6 audio = 12 video = 8 audio = 15 video = 10

Media Synchronization The server –May maintain a sequence number –Each sent frame is assigned with a sequence number The client –May maintain a counter –Each received frame is assigned with a counter

Media Synchronization Ensure the audio stream is not starved at the client Adjust the sending rate of the video stream according to the video –If the video is AHEAD => slow down the video server –If the video is BEHIND => Speed up the video server Drop frames arriving later than the desired presentation deadline

Media Synchronization audio stream 3 samples/sec video stream 2 frames/sec Synchronization Points audio = 3 video = 2 audio = 6 video = 4 audio = 9 video = 5 audio = 12 video = 8 audio = 15 video = 10 DROPPED VIDEO FRAME 6 TO CATCH UP

Graphical User Interface GTK Requires creativity Groups are free to use their own original design Should support many operations common to video-on-demand services Should display meta-data –Dispatcher address and port –Audio/video characteristics Sample rate Frame rate

GUI Operations MENU –List all of the available A/V files –File management interface PLAY –Play the selected A/V stream STOP –Stop the selected A/V stream PAUSE –Pause the selected A/V stream FF and RWND (if implemented) –Fast-forward or rewind selected A/V stream SEEK (if implemented) –A textbox for user to enter or a slider