MP 2: Audio/ Video Streaming CS414: Multimedia System Instructor: Klara Nahrstedt March 16, 2012

Covered Aspects of Multimedia Image/Video Capture Media Server Storage Transmission Compression Processing Audio/Video Presentation Playback Audio/Video Perception/ Playback Audio Information Representation Transmission Audio Capture A/V Playback Image/Video Information Representation

Learning Goals Learning media streaming using network Learning audio video synchronization mechanism Understanding following multimedia system concepts – Resource Admission – QoS Negotiation – QoS Enforcement – Session Control – Session Adaptation Understanding network dynamics such as end-to-end delay, network jitter, and synchronization skew server client

System Components Server Client Server – Capture Audio and Video at a fixed rate – Video: 30 fps, Audio: 8000Hz Client – Requests for Video and/or Audio – Works in Two modes: Active Mode and Passive Mode Control Channel Data Channel Active Mode: Media type: Audio, Video Video Rate: 15 to 25 fps Audio Rate: 8000Hz Video Resolution: 640X480 Passive Mode: Media type: Video Video Rate: 10 fps Video Resolution: 320X240

Functional Diagram: Connection Establishment Client Server Stream Request: Stream Request Accept/ Reject Resource admission RTP/ UDP QoS Enforcem ent Take user-inputs (mode) Take user-inputs (mode) Server listening Resour ce admissi on AV Synchronization resource.txt Session Monitor Client listening Resourc e Reservat ion

Functional Diagram: Control Playback Client Server Control Command: PAUSE RTP/ UDP QoS Enforceme nt User requests PAUSE Server listening AV Synchronization Session Monitor Client is streaming from the Server Stop streaming [does not close connection states] User requests RESUME Control Command: RESUME Resume streaming from current position Control Command: STOP Clear Resource Reservatio n Then new connection will require the connection setup phase User requests STOP

Functional Diagram: Session Adaptation Client Server User Change Bandwidth Server always listening resource.txt Resource admission Stream Request: Stream Request Accept/ Reject RTP/ UDP QoS Enforcem ent Resour ce admissi on resource.txt Client listening Resourc e Reservat ion AV Synchronization Session Monitor

Resource Admission Client – Available Application Bandwidth AB N – Application Frame Size, M N = ? – Application Frame Rate, R N = ? – Audio Bandwidth = 8000 * 16 – Bandwidth Active Mode: B N = (M N * R N ) + Audio Bandwidth Passive Mode: B N = (M N * R N ) resource.txt Optimistic Allocation 8000Hz Audio Signal In active mode, the video rate should be considered the Maximum between [15fps-25fps] provided that AB N >= B N Show your computation on the Screen fps: 10 MJPEG Video

Resource Admission Server – Available Application Bandwidth AB N – Required Bandwidth for Client B C – Admission is successful if B C <= AB N – To support multiple client You also need to consider bandwidth assigned to other resource.txt Client110 Client210 Resource Table

Rate Control: QoS Enforcement You can build leaky bucket You can implement token bucket You can use gstreamer rate control You can implement your own method Sleep (100) => 10 fps You an simply read using loop You have to do this for both audio and video

Synchronization B uild your own synchronization algorithm [Lecture 23] S ynchronization Skew < 80 ms D isplay synchronization skew on the screen Payload Media Type Timestamp RTP header You can additional information e.g., Decode Timestamp in the RTP header

Server-side Data Flow Architecture Video source Video filter Video encoder App Sink Audio source Audio filter Audio encoder App Sink Video Queue Audio Queue Rate Control App Sink App Sink Capture timestamp RTP header Network You can use gstreamer Pipeline

Client Side Data Flow Architecture Video Queue Audio Queue Synchronization RTP header Network App Source Audio Sink App Source Video Sink Audio Decoder Video Decoder java/src/org/gstreamer/example/AppSrcTest.java?r=509 How to use AppSrc?

Evaluations Required Points: 100, Optional Points: 30 Points will be considered based on live demo and interview performance

Group Directory A shortcut link in desktop and home directory – cs414_Gn.work [note: n= your group number] You can directly access – /team/cs414/Gn [note: n= your group number] You are allowed to use any languages (C/C++/java) You are allowed to you’re your own laptop any OS Solution for MP1 is posted in the class website