MP3: Multi-view Surveillance System Instructor: Klara Nahrstedt April 20, 2012 CS414

MP3: Multi-view Surveillance System Window for server 1 Window for server 2 Server – Capture Audio and Video at a fixed rate – Video: 30 fps, Audio: 8000Hz Client – Requests for Video and/or Audio – Works in Two modes: Active Mode and Passive Mode Active Mode: Media type: Audio, Video Video Rate: 15 to 25 fps Audio Rate: 8000Hz Video Resolution: 640X480 Passive Mode: Media type: Video Video Rate: 10 fps Video Resolution: 320X240

Client Behavior Client can request active mode from Server 1 and passive mode from Server 2 Client can request active mode from Server 2 and passive mode from Server 1 Client can request active mode from both Server 1 and Server 2 Client can request passive mode from both Server 1 and Server 2

Client User Interface: Example SWITCH Video window for Server 1 ConnectEdit Video window for Server 2 Computation window for Server 2 Computation window for Server 1

Resource Admission at Client User defines streaming MODE: active or passive Client performs resource admission Server 1 Client Control Channel MODE User

Resource Admission at Client Client – Available Application Bandwidth AB N – Application Frame Size, M N = ? – Application Frame Rate, R N = ? – Audio Bandwidth = 8000 * 16 – Request Bandwidth Active Mode: B N = (M N * R N ) * 16 Passive Mode: B N = (M N * R N ) = (M N * 10) resource.txt Optimistic Allocation 8000Hz Audio Signal How to define R N for active mode? fps: 10 [fps:15-25]

Resource Admission at Client Client – Available Application Bandwidth AB N – Application Frame Size, M N = ? – Application Frame Rate, R N = ? – Request Bandwidth Active Mode: B N = (M N * R N ) + Audio Bandwidth resource.txt Show your computation on the Screen AB N = (M N *R N ) + Audio Bandwidth if R N > 25, R N = 25 if R N < 15, REJECT request else R N

Resource Negotiation with Server Information sent to server includes – Requested Frame Rate (FPS) for video, – Video Resolution, and – Media type (audio, video) Server 1 Client FPS, Resolution Media types MODE

Resource Admission at Server Server – Available Application Bandwidth AB N – Application Frame Size, M N = ? – Application Frame Rate, R N = [received from client] – Audio Bandwidth = 8000 * 16 – Request Bandwidth Active Mode: B N = (M N * R N ) * 16 Passive Mode: B N = (M N * R N ) = (M N * 10) – Admitted if B N <= AB N – Else Rejected resource.txt Optimistic Allocation

Video Data Flow Architecture RTP Packet Rate Control Network Video SourceVideo Filter Media EncoderVideo Muxer App Sink RTP Packet App Source Decode Bin2 Video Sink Synchronization Payload Media Type Timestamp RTP header ID UDP Packet Payload Segment Merge Server-Side Client-Side

Rate Control: QoS Enforcement You can build leaky bucket You can implement token bucket You can use gstreamer rate control You can implement your own method Sleep (100) => 10 fps You can simply read using a loop While(true){ SendPacket(); Thread.sleep(10 0); }

Resource Reservation: Client – Available Application Bandwidth AB N – Used Bandwidth for Server 1 B 1 – Available Bandwidth AB N = AB N –B 1 – Admission is successful for B 2 if B 2 <= AB N resource.txt Server 160 Kbps Server 235 KBps Resource Table

Video Data Flow Architecture using RtpBin RTP Packet Rate Control Network Video SourceVideo Filter Media EncoderVideo Muxer App Sink RTP Packet App Source Decode Bin2 Video Sink Synchronization Segment Merge Server-Side Client-Side RtpBin UdpSink UdpSrc

Implementation Using Server Video SourceVideo Filter Tee Video ScaleScale Filter Video Rate Rate Filter Video Encoder RTP Payload RTP Bin UDP Sink Queue UDP Sink UDP Source RTCP RtpBin combines RtpSession RtpSsrcDemux RtpPtDesmux RtpJitterBuffer

Implementation Using Client Video SinkVideo DecoderRTP DePayload RTP Bin UDP Source UDP Sink RTCP rtpBin.connect(new Element.PAD_ADDED() { public void padAdded(Element element, Pad pad) { if (pad.getCaps().toString().contains("video")) { //LINK TO VIDEO SINK via VIDEO MONITOR } else if (pad.getCaps().toString().contains("audio")) { // LINK TO AUDIO SINK via AUDIO MONITOR } } });

Internals of RtpBin l

Evaluation Process for MP3 Case I: – Server 1 is requested to send stream in ACTIVE mode – Server 2 is requested to send stream in PASSIVE mode – Client has resource to support 25 fps in ACTIVE mode Case II: – Server 1 is requested to send stream in ACTIVE mode – Server 2 is requested to send stream in PASSIVE mode – SWITCH streaming mode of Server 1 and Server 2 – Client has resource to support 25 fps in ACTIVE mode

Evaluation Process for MP3 Case III: – Server 1 is requested to send streams in PASSIVE mode – Server 2 is requested to send streams in ACTIVE mode – Modify resource.txt so that ACTIVE mode use 15fps – Low interference in streaming Case IV: – Server 1 is requested to send streams in PASSIVE mode – Server 2 is requested to send streams in ACTIVE mode – SWITCH mode of Server 1 and Server 2 – Client has resources to support 15fps in ACTIVE mode

Evaluation Process for MP3 Case V: – Server 1 is requested to send streams in ACTIVE mode – Server 2 is requested to send streams in PASSIVE mode – Modify resource.txt so that ACTIVE mode use 25fps – Low interference in streaming Case VI: – Server 1 is requested to send streams in ACTIVE mode – Server 2 is requested to send streams in PASSIVE mode – Update resource.txt so that client has enough available bandwidth to support two ACTIVE modes – SWITCH mode of Server 2 into ACTIVE mode – By default, no audio playback from Server 2; however client should receive audio streams from Server 2

Evaluation Process for MP3 Case VII: – Server 1 is requested to send streams in ACTIVE mode – Server 2 is requested to send streams in ACTIVE mode – MUTE Server 1 audio playback and UNMUTE Server 2 audio playback – No interference in streaming – Client has enough resource to support two ACTIVE mode Case VIII: – Server 1 is requested to send streams in ACTIVE mode – Server 2 is requested to send streams in ACTIVE mode – SWITCH mode of Server 1 into PASSIVE mode

Evaluation Process for MP3 Case IX: – Server 1 is requested to send streams in PASSIVE mode – Server 2 is requested to send streams in ACTIVE mode – Modify resource.txt so that admission for ACTIVE mode is not successful – Streaming from Server 2 turns off, no interference in Server 1 streaming Case X: – Server 1 is requested to streams send in ACTIVE mode – Modify resource.txt so that Server 2 can be requested in ACTIVE mode with 15 fps – SWITCH between modes of Server 1 and Server 2

Evaluation Process of MP3 For All Cases: –Cases are designed to run continuously one after another process without restarting the program or starting from the beginning –Maintain audio-visual synchronization in ACTIVE mode –Show the computation of resource admission –Show the current frame rate, bandwidth usage, and network jitter

Demonstration and Interview You must use at least 3 PCs/ laptops for the demo If you want to use wireless/ wired with only your laptops, please setup the demo in SC3401 If you want to use wired demo with lab PCs, please setup your demo at SC0216 Demo date: April 30 th Demo time: – 5pm – 6pm

Additional Features for MP3? It is highly recommended that you build an additional feature in MP3 to SURPRISE US! Bonus Point: 20 Bonus point will be awarded based on the amount of surprise!