1. 11. Mai 20061INF-3190: Multimedia Protocols Multimedia Protocols Foreleser: Carsten Griwodz Email: griff@ifi.uio.no

2. 11. Mai 2006 2INF-3190: Multimedia Protocols Media Medium: "Thing in the middle“ here: means to distribute and present information Media affect human computer interaction The mantra of multimedia users Speaking is faster than writing Listening is easier than reading Showing is easier than describing

3. 11. Mai 2006 3INF-3190: Multimedia Protocols Dependence of Media Time-independent media Text Graphics Discrete media Time-dependent media Audio Video Animation Continuous media Interdependant media Multimedia "Continuous" refers to the user’s impression of the data, not necessarily to its representation Combined video and audio is multimedia - relations must be specified

4. 11. Mai 2006 4INF-3190: Multimedia Protocols High Data Volume Throughput Higher volume than for traditional data Longer transactions than for traditional data Requires Performance and bandwidth Resource management techniques Compression Typical values Uncompressed video:140 – 216 Mbit/s Uncompressed audio (CD):1.4 Mbit/s Uncompressed speech:64 Kbit/s Compressed audio & video (VoD):down to 1.2 – 4 Mbit/s Compressed audio & video (Conf.):down to 128 Kbit/s Compressed speech:down to 6.2 Kbit/s

5. 11. Mai 2006 5INF-3190: Multimedia Protocols Compression – General Requirements

6. 11. Mai 2006 6INF-3190: Multimedia Protocols Example: MPEG-1 International Standard: Moving Pictures Expert Group Compression of audio and video for playback (1.5 Mbit/s) Real-time decoding Sequence of I-, P-, and B-Frames I-Frames “intra-coded” P-Frames predictive coded B-Frames bi-directionally coded

7. 11. Mai 2006 7INF-3190: Multimedia Protocols MPEG (Moving Pictures Expert Group) Frames can be dropped In a controlled manner Frame dropping does not violate dependancies Example: B-frame dropping in MPEG-1

8. 11. Mai 2006 8INF-3190: Multimedia Protocols Multimedia Networking Internet without network QoS support Internet applications must cope with networking problems Application itself or middleware "Cope with" means either "adapt to" or "don’t care about“ "Adapt to" must deal with TCP-like service variations "Don’t care about" approach is considered "unfair“ "Don’t care about" approach cannot work with TCP Internet with network QoS support Application must specify their needs Internet infrastructure must change – negotiation of QoS parameters Routers need more features Keep QoS-related information Identify packets as QoS-worthy or not Treat packets differently keep routing consistent

9. 11. Mai 20069INF-3190: Multimedia Protocols Non-QoS Multimedia Networking Application Layer Framing & Integrated Layer Processing

10. 11. Mai 2006 10INF-3190: Multimedia Protocols Multimedia Content Processing Problem: optimize transport of multimedia content It is application dependent and specific Application-layer processing has high overhead Application processes data as it arrives from the network Impact of lost and mis-ordered data Transport layer tries to recover from error Prevents delivery of data to application Prevents immediate processing as data arrives Application must stop processing Transport layer ignores error Application experiences processing failures Application must stop processing

11. 11. Mai 2006 11INF-3190: Multimedia Protocols Application Level Framing [Clark/Tennenhouse 1990] Give application more control Application understands meaning of data Application should have the option of dealing with a lost data Reconstitute the lost data (recompute/buffer by applications) Ignore the lost data Application level framing Application breaks the data into suitable aggregates Application Data Units (ADUs) Lower layers preserve the ADU frame boundaries ADU takes place of packet as the unit of manipulation

12. 11. Mai 2006 12INF-3190: Multimedia Protocols ALF: Application Data Units ADUs become the unit of error recovery Should be upper bounded Loss of large ADUs is more difficult to fix Lower bounded Application semantics define smallest sensible unit Small ADUs mean larger protocol overhead Segmentation/reassembly Try to avoid ADU “name” Sender computes a name for each ADU Receiver uses name to understand its place in the sequence of ADUs Receiver can process ADUs out of order

13. 11. Mai 2006 13INF-3190: Multimedia Protocols Integrated Layer Processing Layered engineering is not fundamental Assignment of functions to layers in OSI is not following fundamental principles Specific application may work better with different layering of functions or no layering at all Sequential processing through each layer  Not an efficient engineering  Processing all functions at once saves computing power Integrated Layer Processing Vertical integration Performing all the manipulation steps in one or two integrated processing loops, instead of serially

14. 11. Mai 2006 14INF-3190: Multimedia Protocols Integrated Layer Processing Ordering constraint Data manipulation can only be done after specific control steps Data manipulation can only be done once the data unit is in order Layered multiplexing (extract the data before it can be demultiplexed) Minimize inter-layer ordering constraints imposed on implementors Implementors know best which data must be ordered Drawback: complex design due to fully customized implementation

15. 11. Mai 200615INF-3190: Multimedia Protocols Non-QoS Multimedia Networking RTP – Real-Time Transfer Protocol

16. 11. Mai 2006 16INF-3190: Multimedia Protocols Real-time Transport Protocol (RTP) RFC 3550 (replaces RFC 1889) Designed for requirements of real-time data transport NOT real-time NOT a transport protocol Two Components Real-Time Transfer Protocol (RTP) RTP Control Protocol (RTCP) Provides end-to-end transport functions Scalable in multicast scenarios Media independent Mixer and translator support RTCP for QoS feedback and session information

17. 11. Mai 2006 17INF-3190: Multimedia Protocols Real-time Transport Protocol (RTP) No premise on underlying resources layered above transport protocol no reservation / guarantees Integrated with applications RTP follows principles of Application Level Framing and Integrated Layer Processing

18. 11. Mai 2006 18INF-3190: Multimedia Protocols RTP RTP services are Sequencing Synchronization Payload identification QoS feedback and session information RTP supports Multicast in a scalable way Generic real-time media and changing codecs on the fly Mixers and translators to adapt to bandwidth limitations Encryption RTP is not designed for Reliable delivery QoS provision or reservation

19. 11. Mai 2006 19INF-3190: Multimedia Protocols RTP Functions RTP with RTCP provides Support for transmission of real-time data Over multicast or unicast network services Functional basis for this Loss detection – sequence numbering Determination of media encoding Synchronization – timing Framing - “guidelines” in payload format definitions Encryption Unicast and multicast support Support for stream “translation” and “mixing” (SSRC; CSRC)

20. 11. Mai 2006 20INF-3190: Multimedia Protocols RTP Packet Format Typical IETF RFC bit-exact representation a byte a longword (32 bit)

21. 11. Mai 2006 21INF-3190: Multimedia Protocols RTP Packet Format 4 bit CSRC count, indicates the number of contributing sources in the header Marker bit Meaning depends on payload profile, e.g. frame boundary Version number, Always 2 Padding indicator bit if set, number of padding bytes is in last byte of payload Header extension bit True if header extension is present 7 bit payload type Allows identification of the payload’s content type

22. 11. Mai 2006 22INF-3190: Multimedia Protocols RTP Packet Format 16 bit sequence number Several 32 bit CSRC Contribution source identifier, the number is indicated by CC A mixer copies the original sources’ SSRCs here 32 bit timestamp 32 bit SSRC Synchronization source identifier, a random number identifying the sender Header extension multiples of 32 bit

23. 11. Mai 2006 23INF-3190: Multimedia Protocols RTP Architecture Concepts Integrated Layer Processing Typical layered Data units sequentially processed by each layer Integrated layer processing Adjacent layers tightly coupled Therefore, RTP is not complete by itself: requires application-layer functionality/information in header 16 bit sequence number 32 bit timestamp 7 bit payload type Allows identification of the payload’s content type Marker bit Meaning depends on payload profile, e.g. frame boundary

24. 11. Mai 2006 24INF-3190: Multimedia Protocols RTP Packet Format Relatively long header (>40 bytes) overhead carrying possibly small payload header compression other means to reduce bandwidth (e.g. silence suppression) No length field Exactly one RTP packet carried in UDP packet Can use TCP or ATM AAL5 do-it-yourself packaging Header extensions for payload specific fields possible Specific codecs Error recovery mechanisms

25. 11. Mai 2006 25INF-3190: Multimedia Protocols RTP Profile (RFC 1890) Set of standard encodings and payload types Audio: e.g. PCM-u, GSM, G.721 Video: e.g. JPEG, H.261 Number of samples or frames in RTP packet Sample-based audio: no limit on number of samples Frame-based audio: several frames in RTP packet allowed Clock rate for timestamp Packetized audio: default packetization interval 20 ms Video: normally 90 kHz, other rates possible

26. 11. Mai 2006 26INF-3190: Multimedia Protocols RTP Profiles Payload type identification RTP provides services needed for generic A/V transport Particular codecs with additional requirements Payload formats defined for each codec: syntax and semantic of RTP payload Payload types Static: RTP AV profile document Dynamic: agreement on per-session basis Profiles and Payload Formats in RTP Framework RTP / RTCP AV Profile Additional Profiles Payload Formats Dynamic Payload Types PT mapping outside RTP (e.g. SDP)

27. 11. Mai 2006 27INF-3190: Multimedia Protocols RTP Profiles General Associated with a media type. Provides association between PT field and specific media format Defines sampling rate of timestamp May also define or recommend a definition for the “marker” bit Video Profile Marker bit recommended to mean last packet associated with a timestamp Timestamp clock: 90000 Hz Defines PT mapping for a number of different video encoding types

28. 11. Mai 2006 28INF-3190: Multimedia Protocols RTP Profiles Audio Profile Marker bit set on the first packet after a silence period where no packets sent Timestamp equals sampling rate Recommends 20ms minimum frame time Recommends that samples from multiple channels be sent together Defines PT for a number of different audio encoding types

29. 11. Mai 2006 29INF-3190: Multimedia Protocols RTP Profile for MPEG Video Payload Frame headers GOP header

30. 11. Mai 2006 30INF-3190: Multimedia Protocols RTP Profile for MPEG Video Payload Fragmentation rules Video sequence header if present, starts at the beginning of an RTP packet GOP sequence header Either at beginning of RTP packet Or following video sequence header Picture header Either at beginning of RTP packet Following GOP header No header can span packets Marker Bit Set to 1 if packet is end of picture

31. 11. Mai 2006 31INF-3190: Multimedia Protocols RTP Profile for MPEG Video Payload MPEG-1 Video specific payload header MBZ Must be zero TR Temporal reference The same number for all packets of one frame For ordering inside an MPEG GOP S 1 is sequence header is in this packet B 1 if payload starts with new slice E 1 if last byte of payload is end of slice P 3 bits that indicate picture type (I,P, or B) FBV, BFC, FFC, FFC Indicate how a P or B frame is related to other I and P frames MPEG Video Profile 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | MBZ | TR |MBZ|S|B|E| P | | BFC | | FFC | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ FBV FFV

32. 11. Mai 2006 32INF-3190: Multimedia Protocols RTP Quality Adaptation Component interoperations for control of quality Evaluation of sender and receiver reports Modification of encoding schemes and parameters Adaptation of transmission rates Hook for possible retransmissions (outside RTP)

33. 11. Mai 2006 33INF-3190: Multimedia Protocols RTP Control Protocol (RTCP) Companion protocol to RTP (tight integration with RTP) Monitoring of QoS of application performance Feedback to members of a group about delivery quality, loss, etc. Sources may adjust data rate Receivers can determine if QoS problems are local or network-wide Loose session control Convey information about participants Convey information about session relationships Automatic adjustment to overhead report frequency based on participant count Typically, “RTP does...” means “RTP with RTCP does...”

34. 11. Mai 2006 34INF-3190: Multimedia Protocols RTCP Packets Several RTCP packets carried in one compound packet RTCP Packet Structure SRSender Report (statistics from active senders: bytes sent -> estimate rate) RRReceiver Report (statistics from receivers) SDESSource Descriptions (sources as “chunks” with several items like canonical names, email, location,...) BYEexplicit leave APPextensions, application specific RSR / RRBYESDESAPP Compound (UDP) Packet

35. 11. Mai 2006 35INF-3190: Multimedia Protocols RTCP Sender / Receiver Reports Sender report Sender Information Timestamps Packet Count, Byte Count List of statistics per source Receiver report For each source Loss statistics Inter-arrival jitter Timestamp of last SR Delay between reception of last SR and sending of RR Analysis of reports Cumulative counts for short and long time measurements NTP timestamp for encoding- and profile independent monitoring Header Sender Information Reception Report Profile Specific Extensions Reception Report... Header Reception Report Profile Specific Extensions Reception Report...

36. 11. Mai 2006 36INF-3190: Multimedia Protocols RTP Mixer Mixer Reconstructs constant spacing generated by sender Translates audio encoding to a lower-bandwidth Mixes reconstructed audio streams into a single stream Resynchronizes incoming audio packets New synchronization source value (SSRC) stored in packet Incoming SSRCs are copied into the contributing synchronization source list (CSRC) Forwards the mixed packet stream Useful in conference bridges

37. 11. Mai 2006 37INF-3190: Multimedia Protocols RTP Translator Translation between protocols e.g., between IP and ST-2 Two types of translators are installed Translation between encoding of data e.g. for reduction of bandwidth without adapting sources No resynchronization in translators SSRC and CSRC remain unchanged ATM UDP Protocol Translator MPEG Source MPEG Sink H.263 Sink Profile Translator

38. 11. Mai 2006 38INF-3190: Multimedia Protocols RTP Identifiers S1S3 S2S4 M1 M2 T1 R1 S1:10 S2:1 M1:33 (10,1) S4:13 S3:19 M2:17 (19,13,33) SSRC chosen by sender S1 Translators keep SSRCs and CSRCs SSRC chosen by mixer M1 CSRCs from mixed sources S1 and S2 CSRCs contain previous SSRCs, but not previous CSRCs

39. 11. Mai 2006 39INF-3190: Multimedia Protocols Protocol Development Changes and extensions to RTP Scalability to very large multicast groups Congestion Control Algorithms to calculate RTCP packet rate Several profile and payload formats Efficient packetization of Audio / Video RTCP-based retransmission Loss / error recovery

40. 11. Mai 200640INF-3190: Multimedia Protocols Non-QoS Multimedia Networking Signalling Protocols: RTSP and SIP

41. 11. Mai 2006 41INF-3190: Multimedia Protocols Signaling Protocols Applications differ Media delivery controlled by sender or receiver Sender and receiver “meet” before media delivery Signaling should reflect different needs Media-on-demand Receiver controlled delivery of content Explicit session setup Internet telephony and conferences: Bi-directional data flow, live sources (mostly) explicit session setup, mostly persons at both ends Internet broadcast Sender announces multicast stream No explicit session setup

42. 11. Mai 2006 42INF-3190: Multimedia Protocols Real-Time Streaming Protocol (RTSP) Internet media-on-demand Select and playback streaming media from server Similar to VCR, but Potentially new functionality Integration with Web Security Varying quality Need for control protocol Start, stop, pause, … RTSP is also usable for Near video-on-demand (multicast) Live broadcasts (multicast, restricted control functionality)...

43. 11. Mai 2006 43INF-3190: Multimedia Protocols RTSP Approach In line with established Internet protocols Similar to HTTP 1.1 in style Uses URLs for addressing: rtsp://video.server.com:8765/videos/themovie.mpg Range definitions Proxy usage Expiration dates for RTSP DESCRIBE responses Other referenced protocols from Internet (RTP, SDP) Functional differences from HTTP Data transfer is separate from RTSP connection typically via RTP Server maintains state – setup and teardown messages Server as well as clients can send requests

44. 11. Mai 2006 44INF-3190: Multimedia Protocols RTSP Features Rough synchronization Media description in DESCRIBE response Timing description in SETUP response Fine-grained through RTP sender reports Aggregate and separate control of streams possible Virtual presentations Server controls timing for aggregate sessions RTSP Server may control several data (RTP) servers Load balancing through redirect at connect time Use REDIRECT at connect time Caching Only RTSP caching so far Data stream caching is under discussion

45. 11. Mai 2006 45INF-3190: Multimedia Protocols RTSP Methods OPTIONS C  S determine capabilities of server/client C  S DESCRIBE C  S get description of media stream ANNOUNCE C  S announce new session description SETUP C  S create media session RECORD C  S start media recording PLAY C  S start media delivery PAUSE C  S pause media delivery REDIRECT C  S redirection to another server TEARDOWN C  S immediate teardown SET_PARAMETER C  S change server/client parameter GET_PARAMETER C  S read server/client parameter

46. 11. Mai 2006 46INF-3190: Multimedia Protocols media server RTSP Integration HTTP server RTSP server data source web browser AV subsystem RTSP plug-in HTTP GET presentation description file RTSP SETUP RTSP OK RTSP PLAY RTSP OK RTP AUDIO RTP VIDEO RTSP TEARDOWN RTSP OK

47. 11. Mai 2006 47INF-3190: Multimedia Protocols Session Initiation Protocol (SIP) Lightweight generic signaling protocol Internet telephony and conferencing Call: association between number of participants Signaling association as signaling state at endpoints (no network resources) Several “services” needed Name translation User location Feature negotiation Call control Changing features

48. 11. Mai 2006 48INF-3190: Multimedia Protocols SIP Basics Call user Re-negotiate call parameters Forwarding (manual and automatic) Call center Supports personal mobility (change of terminal) Through proxies or redirection Terminate / transfer calls ASCII (readable) protocol – SIP vs. H.323 Similarities (request/response, proxies...) Differences (server state, server may initiate actions...) Control, location and media description (via SDP) Extensible towards Usage for IP-IP, POTS-IP, inter-gateway interaction with firewalls, billing system,... Different modes Proxy mode Redirect mode

49. 11. Mai 2006 49INF-3190: Multimedia Protocols SIP Operation – Proxy Mode Proxy forwards requests Possibly in parallel to several hosts Cannot accept or reject call Useful to hide location of callee User with “symbolic name” calls another 1. Invite u@domain 7. Ok Site A 8. ACK u@domain 11. Ok 4. Invite user@host 6. Ok 9. ACK user@host 10. Ok 5. “Ring” 2. Where? 3. user@host Proxy Mode Site B Location Server

50. 11. Mai 2006 50INF-3190: Multimedia Protocols SIP Operation – Redirect Mode User with “symbolic name” calls another 2. Where? 3. user@host Location Server Redirect Mode 1. Invite u@domain 4. Moved temporarily Location: user@domain2 5. ACK u@domain 6. Invite user@domain2 7. “Ring” 7. Ok 8. ACK user@domain2 Site A Site B

51. 11. Mai 2006 51INF-3190: Multimedia Protocols SIP – Methods Basic Methods (RFC 2543) TABLE Additional Methods (partially standardized) INFO: carry information between User Agents REFER: ask someone to send an INVITE to another participant SUBSCRIBE: request to be notified of specific event NOTIFY: notification of specific event

52. 11. Mai 2006 52INF-3190: Multimedia Protocols RTP RTP services are Sequencing Synchronization Payload identification QoS feedback and session information RTP supports Multicast in a scalable way Generic real-time media and changing codecs on the fly Mixers and translators to adapt to bandwidth limitations Encryption RTP is not designed for Reliable delivery QoS provision or reservation