1. Transport Layer Doug Young Suh suh@khu.ac.kr Last update : August 1, 2009 Transport Layer

2. Contents Transport layer QoS UDP : User Datagram (connection-less) RTP : Realtime Transport (connection-oriented) RFC 3984 RTP Payload format for H.264 Video Transport Layer

3. Transport layer and Realtime Multimedia Transport Layer Transport layer : end-to-end Flow control : “Send as much as it receives.” Error control : packet loss Nothing to do with the routers

4. Networked Video Transport layer Call setup : path, resources QoS control : RTP/RTCP QoS enhancement (not standardized, yet) FEC with erasures Interleaving Dejittering buffer Retransmission if allowed Upper Layers Transport Layer Network Layer Data Link Layer Physical Layer Error resilience, scalable coding, rate control FEC, retransmission, flow control Upper Layers Transport Layer Network Layer Data Link Layer Physical Layer Network Layer Data Link Layer Physical Layer Network Layer Data Link Layer Physical Layer Resource allocation …… Mobile terminal Base station VOD Server Routers

5. Networked Video the avt WG? 8 Areas in IETF (http://ietf.org) Applications AreaApplications Area - General AreaGeneral Area Internet AreaInternet Area - Routing AreaRouting Area Security AreaSecurity Area - Sub-IP AreaSub-IP Area Operations and Management Area Transport Area Among 27 working groups in Transport Area (avt- request@ietf.org)

6. RTCP (RTP control) In every interval(e.g. 1 sec), exchange RTCP-SR (Sender Report) and RTCP-RR (Receiver Report). Report Packet loss ratio (PLR)  by using SN The receiver counts number of lost RTP packets. Measure RTT (Round Trip Time, but, not one way delay)  by using Time Stamp RTT = T4 – T1 – (T3 – T2) Networked Video senderreceiver RTCP-SR RTCP-RR T1T2 T4 T3

7. Networked Video Time varying end-to-end QoS (Suwon  Pusan)

8. Networked Video FEC (forward erasure correction) dataparity lossy channel recovery RFC2733 : XOR-based (one parity packet) IETF : draft-ietf-avt-uxp-02.txt

9. Networked Video FEC with erasure Erasures : RTP Sequence numbers of lost packets Recover up to N-k lost packets by using RS(N,k) code How to overcome?

10. Networked Video Interleaving (RFC 3267) to combat against bursty loss : loss rate vs. delay Example) 1) No Interleaving : 2 out of each set of 6 packets (1 2 3 4 P1 P2) (5 6 7 8 P3 P4)(9 10 11 12 P5 P6) (13 14 15 16 P7 P8)...... 2) Interleaving : up to 12 consecutive packets 1 5 9 13 17 21 2 6 10 14 18 22 3 7 11 15 19 23 4 8 12 16 20 24 P1 P3 P5 P7 P9 P11 P2 P4 P6 P8 P10 P12 How to overcome?

11. RFC 3409 : FEC with UED and UEP XOR based ULP 0 1 0 1 1 1 0 0 1 0 1 0 0 1 1 0 1 1 1 0 1 0 1 0 1 1 0 0 1 0 1 1 1 0 0  Increase in data rate P1 P2 P3  RS code with UEP Parity bytes 10 packets Packet size

12. Introduction to UDP The UDP header (connection-less). Transport Layer

13. Real-time Transport Protocol (a) The position of RTP (connection-oriented) (b) Packet nesting. Transport Layer

14. RTP (RFC 3550, 2003) RTP Use Scenarios Simple Multicast Audio Conference Audio and Video Conference Mixers and Translators Layered Encodings RTP Header, RTCP Transport Layer SSRC CSRC

15. Semantics of fields in RTP Header Marker bit (M): 1 bit Set for the very last packet of the access unit Payload type (PT): 7 bits MPEG-1 Video, MPEG-2 Video, AAC etc. Sequence number (SN): 16 bits decoding order for the NALU. DON (Decoding Order Number) Timestamp: 32 bits the sampling timestamp A 90 kHz clock rate MUST be used. Transport Layer

16. RFC3550: RTCP(RT Control Protocol, 2003) RTCP SR(Sender Report), RR(Receiver Report) The server monitors round trip time, jitter, and packet loss ratio. Exchanged periodically (<5% bitrate of total) 2015-09-09MediaLab, Kyunghee University16

17. Networked Video RFC3611 RTCP XR(Extended Report, 2003) Block types 1. Loss RLE Report Block  10. post repair RLE Block (2009) 2. Duplicate RLE Report Block 3. Packet Receipt Times Report Block 4. Receiver Reference Time Report Block 5. DLRR Report Block 6. Statistics Summary Report Block 7. VoIP Metrics Report Block Against burst loss (Internet, wireless) 255. reserved for future extensions SDP attributes Attribute name : rtcp-xr Long form: RTP Control Protocol Extended Report Parameters Type of attribute: session and media level loss Run Length Encoding

18. RFC3550 RTP Translators and Mixers Intermediate systems between “network clouds” Multicast audio and video via firewalls and low bandwidth connections Or IPv4  IPv6 The notation "M1: 48(1,17)" designates a packet originating a mixer M1, identified by M1's (random) SSRC value of 48 and two CSRC identifiers, 1 and 17, copied from the SSRC identifiers of packets from E1 and E2. 2015-09-09MediaLab, Kyunghee University18

19. Networked Video RFC 3984 : RTP Payload Format for H.264 Video RTP header + NAL header + slice header + coded slice Decoupling sampling/transmission/decoding/presentation time RTP header V(2)+P(1)+X(1)+CC(4)+PT(7)+SN(16)+TS(32)+SSRC(32)+SCRC MANE (Media Aware Network Element) with NAL header F(1) + NRI (2) + NAL Type (5) Packetization 1. A single NALU, 2. Aggregation, 3. Fragmentation Order of packets FEC with RFC 2733 (1999) An RTP Payload Format for Generic Forward Error Correction

20. RFC 3984 (Feb. 2005) RTP Payload format for H.264/AVC ITU-T Rec. H.264 ISO/IEC IS 14496-10 (MPEG-4 Part 10 AVC) Advanced Video Coding Packetization of one or more Network Abstraction Layer Units (NALUs) Applications simple low bit-rate conversational usage Internet video streaming with interleaved transmission high bit-rate video-on-demand. Transport Layer

21. RTP and H.264/AVC VCL (Video Coding Layer) NAL (Network Abstraction Layer) RTP (Realtime Transport Protocol) Transport Layer Slice data NAL header RTP header

22. NAL header F (1 bit) : forbidden_zero_bit. 1 as a syntax violation set by MANE NRI (2 bits) nal_ref_idc 00 not used to reconstruct reference Type (5 bits) : nal_unit_type SPS (7), PPS (8), slice data Transport Layer Type (5)NRI (2)F (1)

23. NAL Types Transport Layer NALU RTP NALU RTP NALU RTP NALU RTP Single NAL Unit packet STAP (Single Time Aggregation Packet) MTAP (Multi Time Aggregation Packet) FU (Fragmentation Unit) picture NALU pictures

24. NAL Types 0 undefined 1-23 NAL unit Single NAL unit packet per H.264 Single NALU per one RTP packet Low delay conversational, non-interleaved 24 STAP-A Single-time aggregation packet 25 STAP-B Single-time aggregation packet 26 MTAP16 Multi-time aggregation packet 27 MTAP24 Multi-time aggregation packet 28 FU-A Fragmentation unit 29 FU-B Fragmentation unit 30-31 undefined Transport Layer Type (5)NRI (2)F (1)

25. An example of MTAP16 packet 16 bits for time stamp offset Transport Layer RTP Header 4 bytes MTAP NAL HDRDecoding order number baseNALU 1 Size NALU 1 DOND NALU 1 TS Offset NAL 1 DATA NALU 1 HDR NALU 2 SizeNALU 2 DOND NALU 2 TS Offset NALU 2 HDR NAL 2 DATA Optional RTP padding 32 bits

26. NAL Types and NRI NAL Type Content of NALU NRI ---------------------------------- 1 non-IDR coded slice 10 2 Coded slice data partition A 10 3 Coded slice data partition B 01 4 Coded slice data partition C 01 5 Primary coded picture 11 7, 8 SPS, PPS 11 Sets of SPS and PPS are selected by slice header. 6, 9-12 00 Transport Layer

27. Packetization modes yes = allowed, no = disallowed, ig = ignore Type Packet Single NAL Non-Interleaved Interleaved Unit Mode Mode Mode --------------------------------------------- 0 undefined ig ig ig 1-23 NAL unit yes yes no 24 STAP-A no yes no 25 STAP-B no no yes 26 MTAP16 no no yes 27 MTAP24 no no yes 28 FU-A no yes yes 29 FU-B no no yes 30-31 undefined ig ig ig Transport Layer

28. Packetization modes (1) Single NAL Unit Packet Single-time aggregation packet (STAP) aggregates NAL units with identical NALU-time without DON (STAP-A),including DON (STAP-B). Multi-time aggregation packet (MTAP): aggregates NAL units with potentially differing NALU-time 16, 24 differing in the length of the NAL unit timestamp offset Transport Layer

29. Packetization modes (2) Fragmentation Units (FUs) a NAL unit into several RTP packets Big NAL units (> 64 kbytes) over an IPv4 network Because of a limit of the number of slices per picture For FEC (Forward Error Correction) Especially for SPS, PPS, or Partition A as in RFC 2733 without DON (FU-A), with DON (FU-B). Interleaving (sprop_deint_buf_req MIME parameter) Interleaved (2) : STAP-Bs, MTAPs, FU-As, FU-Bs Non-interleaved (1) : single, STAP-As, FU-As Transport Layer

30. MANE (Media Aware Network Element) Installed in application layer gateway Parsing RTP header and payload Drop packets due to congestion No duplication Transformation between Single mode and aggregation mode RTP packets An RTP translator is required to handle RTCP as per RFC 3550. Considering path MTU size, UEP Transport Layer

31. Signaling for an RTP Session MIME registration (or SDP) Media Type name: video Media subtype name: H264 Required parameters: none OPTIONAL parameters: profile-level-id max-mbps, max-fs, max-cpb, max-dpb, and max-br redundant-pic-cap, packetization-mode sprop-parameter-sets : stream properties, SPS, PPS max-rcmd-nalu-size: to signal the capabilities of a receiver Transport Layer

32. SDP in SIP Offer/Answer Offerer -> Answer SDP message: m=video 49170 RTP/AVP 100 99 98 a=rtpmap:98 H264/90000 a=fmtp:98 profile-level-id=42A01E; packetization-mode=0; sprop-parameter-sets=Z0IACpZTBYmI,aMljiA== a=rtpmap:99 H264/90000 a=fmtp:99 profile-level-id=42A01E; packetization-mode=1; sprop-parameter-sets=Z0IACpZTBYmI,aMljiA== a=rtpmap:100 H264/90000 a=fmtp:100 profile-level-id=42A01E; packetization-mode=2; sprop-parameter-sets=Z0IACpZTBYmI,aMljiA==; sprop-interleaving-depth=45; sprop-deint-buf- req=64000; sprop-init-buf-time=102478; deint-buf-cap=128000 Transport Layer

33. Transporting parameter set NALUs 3 Ways 1. Using a session control protocol (out-of-band) prior to the actual RTP session. 2. Using a session control protocol (out-of-band) during an ongoing RTP session. 3. Within the RTP stream in the payload (in-band) during an ongoing RTP session. Transport Layer

34. Security Issues Baugher, M., McGrew, D., Naslund, M., Carrara, E., and K. Norrman, "The Secure Real-time Transport Protocol (SRTP)", RFC 3711, March 2004. data origin authentication data integrity protection Denial of service H.264 is vulnerable to such attack. complex to decode cause the receiver to be overloaded Transport Layer

35. Congestion Issues In real-time encoding Rate control Pre-coded video Switching by using SI/SP slices of the Extended Profile Switching streams at IDR picture boundaries Terminate and re-start MANE removes certain unusable packets. Transport Layer

36. Application Examples Video Telephony According to ITU-T Rec. H.241 in H.323 No Slice Data Partitioning, No NALU Aggregation one-picture-per-one-packet Interleaved using NALU aggregation in H.324M/H.223 AL3 with 100 byte SDU With Data Partitioning : DP and UEP (2733) Video Telephony or Streaming With FUs and Forward Error Correction Streaming Low Bit-Rate Streaming Robust Packet Scheduling Transport Layer

37. Video Telephony or Streaming with FUs and FEC RFC 2733 repetition or XOR based FEC Residual Packet Loss Ratio 10%  (2,1)  1% Extended Golay (24, 12) 0.01% To reduce delay, FEC applied to FUs or slices of similar length Transport Layer

38. Decoding Order Number With/without interleaving 11 steps of video processing camera & capturing / pre-encoding buffer encoder / encoded picture buffer transmitter / transmission channel / receiver receiver buffer decoder / decoded picture buffer display Transport Layer

39. Interleaving and DON Early IDR pictures in sending order... I00 N58 N59 R03 N01 N02 R06 N04 N05...... 62 63 64 65 66 67 68 69 70... Receiver buffer... N58 N59 I00 R03 N01 N02 R06 N04 N05...... 63 64 65 66 67 68 69 70 71... Receiver buffer after reordering... N58 N59 I00 N01 N02 R03 N04 N05...... 64 65 66 67 68 69 70 71... I : intra, R : predictive, N : bi-directional Transport Layer

40. Networked Video RFC5109 ULP(Unequal Loss Protection, 2007) XOR based

41. Networked Video RFC4588 Feedback/Retransmission (2006) Against burst loss (Internet, wireless) Unicast or (small) multicast real-time applications with relaxed delay bounds (e.g. 2-3 sec in streaming) RTP/AVPF 0: unassigned 1: Picture Loss Indication (PLI) 2: Slice Lost Indication (SLI) 3: Reference Picture Selection Indication (RPSI) …………..

42. Abbreviations DON: Decoding Order Number DONB: Decoding Order Number Base DOND: Decoding Order Number Difference FEC: Forward Error Correction FU: Fragmentation Unit IDR: Instantaneous Decoding Refresh IEC: International Electrotechnical Commission ISO: International Organization for Standardization ITU-T: International Telecomm. Union, Telecomm. Standardization Sector MANE: Media Aware Network Element MTAP: Multi-Time Aggregation Packet MTAP16: MTAP with 16-bit timestamp offset MTAP24: MTAP with 24-bit timestamp offset NAL: Network Abstraction Layer NALU: NAL Unit SEI: Supplemental Enhancement Information Transport Layer  AU Access Unit  IDR Instantaneous Decoding Refresh  MIME Multipurpose Internet Mail Extensions  RTSP Real-Time Streaming Protocol  SAP Session Announcement Protocol  SDP Session Description Protocol  SIP Session Initiation Protocol  SPS (PPS) Sequence (Picture) Parameter Set  STAP: Single-Time Aggregation Packet  STAP-A: STAP type A  STAP-B: STAP type B  TS: Timestamp  VCL: Video Coding Layer