1. RTP/RTCP Overview Hank Peng

2. Audio/Video Transport (avt) Chartered 1992-Mar-05 – to specify a protocol for real-time transmission of audio and video over unicast and multicast UDP/IP Concluded 2011-Mar-19 – RTP, associated profiles, extensions, and payload formats

3. Real-time Applications and Infrastructure Area – RAI: Real-time Applications and Infrastructure Real-time Applications and Infrastructure – avtcoreAudio/Video Transport Core Maintenance – avtextAudio/Video Transport Extensions – mmusicMultiparty Multimedia Session Control – p2psipPeer-to-Peer Session Initiation Protocol – payloadAudio/Video Transport Payloads – rtcwebReal-Time Communication in WEB-browsers – simpleSIP for Instant Messaging and Presence Leveraging Extensions – sipcoreSession Initiation Protocol Core – xmppExtensible Messaging and Presence Protocol

4. Current Tasks for avt - 1 1) Maintenance of the core RTP/SRTP/RTCP protocols and the AVP, SAVP, AVPF, and SAVPF profiles – maintain and enhance the SRTP Profile, with review and input from the Security Area – specify how to use Explicit Congestion Notification (ECN) with RTP/RTCP – continue to investigate the export of summarized RTP/RTCP information for operations and monitoring purposes 2) Specification and maintenance of payload formats for use with RTP – provide guidelines for payload format design – develop payload formats for new media codecs – review and revise existing payload formats to advance those which are useful to Draft Standard or Standard, and to declare others as Historic

5. Current Tasks for avt - 2 3) Specification of metric blocks for use with the RTCP Extended Report (XR) framework – provide a mechanism allowing identification data for a set of related metric reports to be carried separately and identified in those reports by reference – provide report blocks for delay, delay variation, and discard – provide report blocks for burst/gap discard and loss – provide report blocks supporting rapid synchronization of multiple RTP streams – provide report blocks for jitter buffer metrics – provide report blocks for loss concealment metrics 4) Extensions to the core protocols to facilitate joining, synchronizing, control, and monitoring of RTP multimedia sessions – develop tools to support rapid acquisition of multimedia streams – specify necessary extensions to support rapid synchronization of multiple RTP streams

6. Documents http://datatracker.ietf.org/wg/avt/ Documents

7. RFCs Related to QoS2.0 Sprint 1 rfc3550 - RTP: A Transport Protocol for Real-Time Applications rfc3551 - RTP Profile for Audio and Video Conferences with Minimal Control rfc3611 - RTP Control Protocol Extended Reports (RTCP XR) rfc4571 - Framing Real-time Transport Protocol (RTP) and RTP Control Protocol (RTCP) Packets over Connection-Oriented Transport rfc4585 - Extended RTP Profile for Real-time Transport Control Protocol (RTCP)-Based Feedback (RTP-AVPF) rfc5104 - Codec Control Messages in the RTP Audio-Visual Profile with Feedback (AVPF) rfc5117 - RTP Topologies rfc5285 - A General Mechanism for RTP Header Extensions rfc5761 - Multiplexing RTP Data and Control Packets on a Single Port rfc5968 - Guidelines for Extending the RTP Control Protocol (RTCP) rfc6035 - Session Initiation Protocol Event Package for Voice Quality Reporting rfc6184 - RTP Payload Format for H.264 Video rfc6190 - RTP Payload Format for Scalable Video Coding draft-alvestrand-rmcat-remb - RTCP message for Receiver Estimated Maximum Bitrate draft-alvestrand-rtcweb-congestion - A Google Congestion Control Algorithm for Real-Time Communication on the World Wide Web

8. Transport Functions Application Support – Reliability control: loss recover, in-sequence delivery, etc. Network Control – Congestion control, rate allocation, etc The distinction between the two is not sharp. – Rate allocation and scheduling can be viewed as part of either one above. This dual view arises when we contemplate traditional transports: TCP and UDP

9. Overview of RTP Provides end-to-end delivery services for real-time traffic: interactive audio and video – Payload identification, sequence numbering, timestamping and delivery monitoring Runs on top of UDP, and less often, TCP. – RTP does not guarantee delivery or prevent out-of-order delivery. Primarily designed to support multiparty multimedia conferences, typically assumes IP multicast.

10. Overview – Cont. The protocol has two parts. – RTP: carry real-time data – RTP control protocol (RTCP): monitor the quality of service and to convey information about the participants. Principles of application level framing and integrated layer processing. – Is malleable to provide application specific info. – Is typically integrated into the application processing. – Protocol is deliberately not complete. It only contains the common functions. – A complete specification for an application also includes a profile and a payload format document.

11. Example- Multicast audio conference Need a multicast address and a pair of ports: one for data and one for (RTCP) control. RTP header contains type of audio encoding (such as PCM). Senders can change encoding during the conference. RTP header contains timing information. Audio data can be played out as they are produced by the source. Senders and receivers multicast reports through RTCP. Packet loss ratio, delay jitter, and other status info can be monitored.

12. Example – Mixers and Translators Mixers: a RTP-level entity that receives streams of RTP data packets from one or more sources and combines them into a single stream. A translator forwards RTP packets from different sources separately. Mixer is like a new RTP-level source to the receivers. Translator is more transparent. Receivers can identify individual sources even though packets pass through the same translator and carry the translator’s network source address. Mixer can re-synchronize the incoming stream and generates its own timing info.

13. Example – Mixers and Translators Mixers: a RTP-level entity that receives streams of RTP data packets from one or more sources and combines them into a single stream. A translator forwards RTP packets from different sources separately. Mixer is like a new RTP-level source to the receivers. Translator is more transparent. Receivers can identify individual sources even though packets pass through the same translator and carry the translator’s network source address. Mixer can re-synchronize the incoming stream and generates its own timing info.

14. Example: Translator at Firewall Translator Firewall Translator On multicast Address a, port p, p+1 On multicast Address b, port q, q+1 Inside Firewall Note that UDP or TCP connections terminates at Firewall.

15. Some RTP Definitions Transport address: network address + port RTP session: communications on a pair of transport addresses (data + control) Synchronization source (SSRC): the source of a stream of RTP packets – Identified by 32-bit SSRC identifier. – All packets from the same SSRC form a single timing and sequencing space. Receivers group packets by SSRC for playback. – Not dependent on network address. – Examples: all packets from a camera; from a mixer; for layered encoding transmitted on separate RTP sessions a single SSRC is used for all layers. – A participant need not use the same SSRC for all RTP sessions in a multimedia session.

16. RTP Fixed Header P: Padding X: Header Extension CC: CSRC count M: Marker of record boundary PT: Payload type; mapping can be specified by profile of the application Sequence number: for each packet can be used by the receiver to detect loss or restore sequence.

17. RTP Fixed Header – Cont. Timestamp – Reflects sampling instant of the first byte of data – Clock frequency can be specified by profile of payload format documents for the application. – Example: for fixed-rate audio, clock may increment by one for each sampling period. SSRC: chosen randomly for each synchronization source; with the intent that no two synchronization sources in the same session have the same SSRC.

18. Profile-Specific Modifications to the RTP Header Marker bit and payload type are interpreted according to the application’s profile. Moreover, the byte containing them can be redefined by the profile. If a particular class of application needs additional functionality, the profile should define additional fixed fields following SSRC. If X bit is 1, exactly one header extension follows CSRC list (if present). – Variable length – Used to experimental purpose

19. RTCP Primary function is to provide feedback on the quality of data distribution. – Through sender and receiver reports; – For adaptive encoding (adaptive to network congestion); – Can be used to diagnose faults RTCP carries a persistent transport-level identifier for an RTP source, called canonical name, CNAME. – Receivers use CNAME to keep track of each participant – And to synchronize related media streams (with the help of NTP) Passes participant’s identification for display.

20. RTCP Packets SR: sender reports; sending and reception stat. RR: receiver reports; for reception statistics from multiple sources. SDES: source description item, include CNAME BYE: indicates end of participation APP: application specific functions

21. Compound RTCP Packets A compound RTCP packet contains multiple RTCP packets of the previous types. Example:

22. SR Packet

23. SR Packet – Cont. RC: receiver report count Length: in 32-bit words – 1 NTP ts: wallclock time, used to calculate RTT RTP ts: in unit and offset of RTP ts in data packets. Can be used with NTP ts for inter-media synchronization. Fraction lost: since the last RR or SR packet was sent. Short term loss ratio. LSR: last SRT time stamp; middle 32 bits of NTP timestamp. DLSR: delay since last SR; expressed in 1/65536 seconds between receiving the last SR packet from SSRC_n and sending this report. Source SSRC_n can compute RTT using DLSR, LSR and the reception time of the report, A. RTT = A – LSR – DLSR An application’s profile can define extensions to SR or RR packets

24. SDES Packet

25. CNAME Item in SDES Packet Mandatory Provides a persistent identifier for a source. Provides a binding across multiple media used by one participant in a set of related RTP sessions. CNAME should be fixed for that participant. SSRC is bound to CNAME Example: doe@sleepy.megacorp.com; doe@192.0.2.89, etc.doe@sleepy.megacorp.comdoe@192.0.2.89

26. Other Items in SDES Packet NAME: user name EMAIL: PHONE: LOC: location TOOL: application or tool name NOTE: notice/status

27. BYE: Goodbye RTCP Packet Mixers should forward the BYE packet with SSRC/CSRC unchanged. Reason for leaving: string field; e.g., “camera malfunction”

28. APP RTCP Packet Subtype: allows a set of APP packets to be defined under one unique name. Name: unique name in the scope of one this application.

29. Conclusions - I RTP defines transport support for common functions of real-time applications. – Timing information: sampling period and NTP – Synchronization source for playback – Payload types (encoding) – Quality reports: short-term and long-term packet loss, and jitters. – Participants indication: CNAME, NAME, EMAIL, etc. – Multicast distribution support – Conversion: mixers and translators Extensible protocol by profile payload format documents Customizable to application or application classes

30. Conclusion – II Separation of control and data stream (analogous to out-band signaling) – Data header overhead is small. – Can accomplish complex control features. – Complexity of the protocol/algorithm is not so bad, because there is little hard guarantee (It relies on TCP or application for hard guarantees).

31. Conclusions – III Congestion control is not defined in baseline document, but may be defined by application’s profile. – Leads to application-specific congestion control or adaptation RTP can be considered as user-space transport entities, but does not run as stand-alone process. Mixers and translators are stand-alone processes. They terminate TCP or UDP connections.

32. RTP Algorithms - I RTCP packets generation: need to limit the control traffic – Control traffic takes 5% of data traffic bandwidth (not defined) – ¼ of the RTCP bandwidth is used by senders – Interval between RTCP packets scales linearly with the number of members in the group. – Each compound RTCP packet must include a report packet and a SDES packet for timely feedback.

33. RTP Algorithms - II SSRCs are chosen randomly and locally and can collide. Loops introduced by mixers and translators – A translator may incorrectly forward a packet to the same multicast group from which it has received the packet. – Parallel translators. Collision avoidance of SSRC and loop detection are entangled.

34. Example of A Profile Document RTP data header: – use one marker bit – No additional fixed fields – No RTP header extensions are defined. RTCP – No additional RTCP packet types. – No SR/RR extensions are defined – SDES use: CNAME is sent every reporting interval, other items should be sent only every fifth reporting interval. rfc3551 - RTP Profile for Audio and Video Conferences with Minimal Control

35. Payload Types

36. rfc3611 - RTP Control Protocol Extended Reports (RTCP XR) Supplements the six statistics that are contained in SR and RR Support multicast inference of network characteristics (MINC) or voice over IP (VoIP) monitoring Convey information used for network management

37. Report Block categories Packet-by-packet block – Loss RLE RB – Duplicate RLE RB – Packet Receipt Times RB Reference time related block – Receiver Reference Time RB – DLRR RB Summary metric block – Statistics Summary RB – VoIP Metrics RB

38. XR Packet Format

39. Extended RB Framework

40. Loss RLE RB

41. Duplicate RLE RB

42. Packet Receipt Times RB

43. Receiver Reference Time RB

44. DLRR – Delay since last Receiver Report

45. Statistics Summary

46. VoIP Metrics RB

47. rfc4585 - Extended RTP Profile for Real-time Transport Control Protocol (RTCP)-Based Feedback (RTP-AVPF) Receivers may use the base mechanisms of RTCP to report packet reception statistics and thus allow a sender to adapt its transmission behavior in the mid-term RTP-AVPF enables receivers to provide, statistically, more immediate feedback to the senders and thus allows for short-term adaptation and efficient feedback-based repair mechanisms to be implemented – Retransmission – FEC – media-specific mechanisms - reference picture selection Modification/addition – Early RTCP messages and algorithms allowing for low-delay feedback in small multicast groups – general-purpose feedback messages

48. Definitions Regular RTCP mode – RTCP messages are sent following the rules of RFC3550, minimal interval of 5s Early RTCP mode – receiver is often (but not always) capable of reporting events of interest back to the sender close to their occurrence Immediate Feedback mode – each receiver is, statistically, capable of reporting each event of interest immediately back to sender Feedback (FB) message – convey information about events observed at a receiver Feedback (FB) threshold – indicates the transition between Immediate Feedback and Early RTCP mode

49. Modes of Operation an indication of whether or not the receiver will, on average, be able to report all events to the sender in a timely fashion Immediate Feedback mode – group size is below the FB threshold – N<=B*T/R, where N: average number of events per interval T, B: RTCP bandwidth fraction, R: average RTCP packet size Early RTCP mode – N > B*T/R Regular RTCP Mode

50. Modes of Operation

51. Format of RTCP Feedback Messages Transport layer FB – general purpose feedback information, i.e., information independent of the particular codec or the application – Generic NACK Payload-specific FB – specific to a certain payload type and will be generated and acted upon at the codec layer – Picture Loss Indication (PLI) – Slice Loss Indication (SLI) – Reference Picture Selection Indication (RPSI) Application layer FB – convey feedback from the receiver’s to the sender’s application

52. Common Packet Format for Feedback Messages

53. Generic NACK indicate the loss of one or more RTP packets Packet ID (PID) – RTP sequence number of the lost packet bitmask of following lost packets (BLP) – any of the 16 RTP packets immediately following the RTP packet indicated by the PID

54. Payload-Specific Feedback Messages Picture Loss Indication (PLI) – a decoder informs the encoder about the loss of an undefined amount of coded video data belonging to one or more pictures Slice Loss Indication (SLI) – a decoder can inform an encoder that it has detected the loss or corruption of one or several consecutive macroblock(s) in scan order Reference Picture Selection Indication (RPSI) – an encoder has learned about a loss of encoder- decoder synchronicity

55. Application Layer Feedback Messages transport application-defined data directly from the receiver’s to the sender’s application the application MUST be able to identify the message payload

56. Early Feedback and Congestion Control The way to react to the feedback received depends on the application using the feedback mechanisms Common requirement for (TCP-friendly) congestion control on the media stream RTCP feedback itself is insufficient for congestion control purposes as it is likely to operate at much slower timescales than other transport layer feedback mechanisms (that usually operate in the order of RTT) Therefore, additional mechanisms are required to perform proper congestion control

57. rfc5104 - Codec Control Messages in the RTP Audio-Visual Profile with Feedback (AVPF)