Πολυμεσικό Υλικό στο Internet: Συγχρονισμός, Επεξεργασία και Διακίνηση Διακίνηση Video με χρήση του HTTP 17/12/2015 Β. Μάγκλαρης Μ. Γραμματικού Δ. Καλογεράς

Outline Streaming over Internet 1 st Generation Progressive 2 nd Generation UDP based Real-Time Protocols (RTP, RTCP, RTSP) 3 rd Generation TCP based – Proprietary – Adobe – Adaptive HTTP based

Challenges  TCP/UDP/IP, best-effort, with no guarantees on expectation or variance of packet delay  Streaming applications delay of 5 to 10 seconds is typical and has been acceptable, but performance deteriorate if links are congested  Real-Time Interactive requirements on delay and its jitter have been satisfied by over- provisioning (providing plenty of bandwidth), what will happen when the load increases?...

4 On-Demand Streaming  Important and growing application due to reduction of storage costs, increase in high speed net access from homes, enhancements to caching  Interactive control by user (but often with long response time)  Ubiquitous on the web: – YouTube, Netflix, Vimeo – Television networks, Hollywood, etc. – Most local radio & TV stations – Virtually everywhere on websites

First Generation: HTTP Download  A simple architecture is to have the Browser request the object(s) and after their reception pass them to the player for display – No pipelining

Helper Application – i.e. media player  Displays content, which is typically requested via a Web browser; typical functions: – Decompression – Jitter removal – Error correction: use redundant packets to be used for reconstruction of original stream – GUI for user control  Examples: – RealPlayer – Adobe Flash Player – Windows Media Player – QuickTime – DivX Web Player

First Gen: HTTP Progressive Download (2)  Alternative: set up connection between server and player; player takes over  Web browser requests and receives a Meta File (a file describing the object) instead of receiving the file itself;  Browser launches the appropriate Player and passes it the Meta File;  Player sets up a TCP connection with Web Server and downloads or streams the file

8 Meta file requests

HTTP Progressive Download  With helper application doing the download, playback can start immediately...  Or after sufficient bytes are buffered  Sender sends at maximum possible rate under TCP; retransmit when error is encountered; Player uses a much larger buffer to smooth delivery rate of TCP

HTTP Progressive Download

Buffering Continuous Media  Jitter = variation from ideal timing  Media delivery must have very low jitter – Video frames every 30ms or so – Audio: ultimately samples need < 128ms jitter  But network packets have much more jitter that that!  Solution: buffers – Fill them with best effort – Drain them via low-latency, local access

Streaming, Buffers and Timing Time Max Buffer Duration = allowable jitter File Position Max Buffer Size Smooth Playback Time Buffer almost empty "Good" Region: smooth playback "Bad": Buffer underflows and playback stops "Bad": Buffer overrflows Buffer Duration Buffer Size

The myth of SBR Any single bitrate you choose for a video is by definition the wrong video for 99% of users. These users will rebuffer These have lower quality than they could sustain

Drawbacks of HTTP Progressive Download (2)  HTTP connection keeps data flowing as fast as possible to user's local buffer – May download lots of extra data if you do not watch the video – TCP file transfer can use more bandwidth than necessary  Mismatch between whole file transfer and stop/start/seek playback controls. – However: use file range requests to seek to video position  Cannot change video quality (bit rate) to adapt to network congestion

2nd Generation: Real-Time Streaming  This gets us around HTTP, allows a choice of UDP vs. TCP and the application layer protocol can be better tailored to Streaming; many enhancements options are possible

RTSP and RTP Legacy Video Streaming Approaches

Example: Real Time Streaming Protocol (RTSP) For user to control display: rewind, fast forward, pause, resume, etc… Out-of-band protocol (uses two connections, one for control messages (Port 554) and one for media stream) RFC 2326 permits use of either TCP or UDP for the control messages connection, sometimes called the RTSP Channel As before, meta file is communicated to web browser which then launches the Player; Player sets up an RTSP connection for control messages in addition to the connection for the streaming media

18 RTSP Operation

19 RTSP Media Stream  Stateful Server keeps track of client's state  Client issues Play, Pause,..., Close  Steady stream of packets – UDP - lower latency – TCP - may get through more firewalls, reliable Credit: some content adapted from Alex Zambelli

20 RTSP Exchange Example C: SETUP rtsp://audio.example.com/xena/audio RTSP/1.0 Transport: rtp/udp; compression; port=3056; mode=PLAY S: RTSP/ OK Session 4231 C: PLAY rtsp://audio.example.com/xena/audio.en/lofi RTSP/1.0 Session: 4231 Range: npt=0 (npt = normal play time) C: PAUSE rtsp://audio.example.com/xena/audio.en/lofi RTSP/1.0 Session: 4231 Range: npt=37 C: TEARDOWN rtsp://audio.example.com/xena/audio.en/lofi RTSP/1.0 Session: 4231 S: OK

21 Example 2: RTMP - Real-Time Messaging Protocol  Proprietary Adobe protocol  Runs over TCP  Manages audio, video, and other  Multiplex multiple streams over TCP connection

Flash Streaming Server  Flash Streaming Server communicates with its clients using the Adobe patented RTMP over TCP, which manages a two-way connection, allowing theserver to send and receive,video, audio and data between client and server  Typically used with FLV encoded files

Variations of RTMP  RTMP- Standard,unencrypted RTMP. The default port is  RTMPT- RTMP “tunneled”over HTTP Thedefault port is 80  RTMPS- RTMP sent over an SSL. SSL enables secure TCP/IP connections. The default port is 443  RTMPE- Enhanced and encrypted version of RTMP.  RTMPTE- Encrypts the communication channel, tunneling over HTTP

24 Drawbacks of RTSP, RTMP  Web downloads are typically cheaper than streaming services offered by CDNs and hosting providers  Streaming often blocked by routers  UDP itself often blocked by firewalls  HTTP delivery can use ordinary proxies and caches  Conclusion: rather than adapt Internet to streaming, adapt media delivery to the Internet

Bandwidth is never constant  Even preselecting bitrates will lead to rebuffering problems as bandwidth fluctuates over the duration of the vide, especially for long form content  Ideally we would like to switch bitrates constantly to always give the user the highest quality they can sustain at any point in time. Buffering

Bandwidth is never constant  Even preselecting bitrates will lead to rebuffering problems as bandwidth fluctuates over the duration of the vide, especially for long form content  Ideally we would like to switch bitrates constantly to always give the user the highest quality they can sustain at any point in time. Buffering 500kbps 1000kbps 1500kbps Adaptive delivery

3rd Generation: HTTP Streaming  Other terms for similar concepts: Adaptive Streaming, Smooth Streaming, HTTP Chunking  Client-centric architecture with stateful client and stateless server – Standard server: Web servers – Standard Protocol: HTTP – Session state and logic maintained at server  Video is broken into multiple chunks  Chunks begin with keyframe so independent of other chunks  A series of HTTP progressive downloads of chunks  Playing chunks in sequence gives seamless video

Adaptive Bit Rate with HTTP Streaming  Encode video at different levels of quality/bandwidth  Client can adapt by requesting different sized chunks  Chunks of different bit rates must be synchronized – All encodings have the same chunk boundaries and all chunks start with keyframes, so you can make smooth splices to chunks of higher or lower bit rates

How does segmentation work? Incoming video 1 3 into multiple short blocks. Each block holds the same section of video, encoded at a different size and bitrate. 500 kbps 2000 kbps 1000 kbps is split by an encoder 2

How does adaptive delivery work? The segmented video is stored on a server, along with a text file which describes the names of each segment. This text file is called a manifest. A player downloads the manifest and then begins requesting individual segments of video. It makes its choice based on bandwidth conditions, grabbing the best quality it can at the time. SERVERCLOUDPLAYER

Media Source Extension – the future of browser video  A W3C standard - file/tip/media-source/media-source.html file/tip/media-source/media-source.html  Supported today by Chrome, IE11 and Opera  MSE allows JavaScript to - fetch and parse a manifest - retrieve video, audio and text segments using XHR - feed them to the tag to be decoded and rendered. MSE gives developers incredible control over playback, buffering and adaption logic of in-page video.

Pareto principle in video popularity Also known as the “80-20” (or 90-10) rule [ Associated with highly skewed distributions Caused by positive-feedback effects (“the rich get richer”) Does non-UGC content (say movies from NetFlix) follow a similar pattern? A practical implication of this skewness: caching applies!

Caching works… Let’s build CDNs Take advantage of locality and relativity A practical implication of this skewness: caching works! Akamai You tube Netflix based on Amazon infrastracture

Adaptive HTTP Streaming System (Protocol)  Server – Can be standard web server – Media segment can be prepared in-line or off- line  Client – Sends series of HTTP GET segment requests and receives segments – Performs rate adaptation before sending a new GET segment request

35 Advantages of HTTP Streaming Easy to deploy: it's just HTTP, work with existing caches/proxies/CDN/Firewall Fast startup by downloading lowest quality/smallest chunk Bitrate switching is seamless Many small files – Small with respect to the movie size – Large with respect to TCP 5-10 seconds of 1Mbps – 3Mbps  0.5MB – 4MB per chunk

Which adaptive streaming formats exist today?  HTTP Live Streaming (HLS) - Controlled by Apple - Known as HLS, it is supported well by iOS, Safari and half-heartedly by Android.  HTTP Dynamic Streaming (HDS) - Controlled by Adobe - Known as HDS, it is played back only by Flash clients with custom apps.  Smooth Streaming (SMOOTH) - Controlled by Microsoft - Played back via Silverlight clients, Xbox,  MPEG-DASH (DASH) - An ISO standard - Playback via MSE in browsers, native apps on iOS, Android and WInOS.

Simple HLS example

Example of HLS Meta Data

RTMP Other Video Streaming Approaches

Several players support it Several commercial content providers use it Microsoft Smooth Streaming Player Netflix Player Adobe OSMF/Zeri Player Apple Player Adaptive video streaming over HTTP

HTTP Adaptive Video Streaming

HTTP Adaptive Video Streaming: Manifest File and Fragments

Smooth Streaming Player

Sample HTTP Request: – GET /mediadl/iisnet/smoothmedia/Experience/BigBuckBunny7 20p.ism/QualityLevels( )/Fragments(video= ) HTTP/1.1

Adobe OSMF/Zeri Player

Adobe OSMF Player Sample HTTP Request: – GET /content/inoutedit- mbr/inoutedit\_h264\_3000Seg1-Frag5 HTTP/1.1

What Makes Netflix Interesting? Commercial, feature-length movies and TV shows  and not free; subscription-based Nonetheless, Netflix is huge!  25 million subscribers and ~20,000 titles (and growing)  consumes 30% of peak-time downstream bandwidth in North America A prime example of cloud-sourced architecture  Maintains only a small “in-house” facility for key functions  e.g., subscriber management (account creation, payment, …)  Majority of functions are sourced to Amazon cloud (EC2/S3)  user authentication, video search, video storage, …  DNS service is sourced to UltraDNS  Leverage multiple CDNs (content-distribution networks) for video delivery  Akamai, Level 3 and Limelight

Netflix Architecture Netflix has its own “data center” for certain crucial operations (e.g., user registration, billing, …) Most web-based user-video interaction, computation/storage operations are cloud-sourced to Amazon AWS Video delivery is out/cloud-sourced to 3 CDNs Users need to use MS Silverlight player for video streaming

Netflix Videos and Video Chunks Netflix uses a numeric ID to identify each movie – IDs are variable length (6-8 digits): , , – video IDs do not seem to be evenly distributed in the ID space – these video IDs are not used in playback operations Each movie is encoded in multiple quality levels, each is identified by a numeric ID (9 digits) – various numeric IDs associated with the same movie appear to have no obvious relations 49

Netflix Player

Sample HTTP Request: GET /sa2/946/ wmv/range/ ?token= \_d6f f1fb6 0cc48eab59ea55\&random= HTTP/1.1

Netflix Videos and Video Chunks Videos are divided in “chunks” (of roughly 4 secs), specified using (byte) “range/xxx-xxx?” in the URL path: Limelight: ?p=58&e= &h=2caca6fb4cc2c522e657006cf69d4ace Akamai: range/ ?token= _411862e41a33dc93ee71e2e3b3fd8534 Level3: ?etime= &movieHash=094&encoded= df 0656e697cbd Netflix uses a version of (MPEG-)DASH for video streaming 52

Terms and Definitions of Adaptive HTTP Streaming  Need – Media Presentation Description (MDP) which provides metadata For requesting (GET request) media segments For rate adaptation purpose – Segment which may include media data or metadata to decode  Use DASH an example in the few slides

Example: DASH Client Thomas Stockhammer, Qualcomm, “DASH – Design Principles and Standards, Presentation at MMSys 2011

Meta Data  DASH uses MPD (Media Presentation Descriptor) and Index Information as metadata for DASH Access Client  Initialization and Media Segments for Media Engine – Reuse of existing conta iner format Source: Stockhammer, Qualcomm, “DASH – Design Principles and Standards, Presentation at MMSys 2011

Media Presentation Data Model MDP - description of accessible segments and corresponding timing Source: Stockhammer, Qualcomm, “DASH – Design Principles and Standards, Presentation at MMSys 2011