1. Multiplexing H.264/AVC Video with MPEG-AAC Audio Harishankar Murugan University of Texas at Arlington

2. Outline : Multiplexing: Areas of applications Why H.264 and AAC? Multiplexing De-multiplexing Synchronization and Playback Results Conclusions Future work References

3. Multiplexing : Areas of applications DVB : DVB-C, DVB-T ATSC IPTV

4. Multiplexing : Areas of applications

5. Why H.264 Video? Up to 50% in bit rate savings: Compared to H.263v2 (H.263+) or MPEG-2 Simple Profile. High quality video: H.264 offers consistently good video quality at high and low bit rates. Error resilience: H.264 provides the tools necessary to deal with packet loss in packet networks and bit errors in error-prone wireless networks. Wide areas of application streaming mobile TV, HDTV, and storage options for the home user

6. Important features of H.264 IDR (Instantaneous decoder refresh) picture: Anchor picture with only I-slices. Sequence parameter set: profile and level indicator. decoding or playback order. number of reference frames. aspect ratio or color space details. Picture parameter set: entropy coding mode used. slice data partitioning and macroblock reordering. Flags indicating the usage of weighted (bi) prediction. Quantization parameter details.

7. AAC Audio Advanced Audio Coding is a standardized, lossy compression scheme for audio. Encoder Block diagram of AAC

8. AAC Audio Profiles : Low Complexity (LC) - the simplest and most widely used; Main Profile (MAIN) - LC profile with backwards prediction; Sample-Rate Scalable (SRS) – LC profile with gain control tool ; Bit stream Formats: ADIF - Audio Data Interchange Format: Only one header in the beginning of the file followed by raw data blocks ADTS - Audio Data Transport Stream Separate header for each frame enabling decoding from any frame

9. Why AAC Audio? Supports Sample frequencies from 8 kHz to 96 kHz (official MP3: 16 kHz to 48 kHz) Higher coding efficiency and simpler filterbank (pure MDCT ) as compared to mp3 (hybrid filter bank ) Improved compression provides higher-quality audio with smaller bit rates. Superior performance at bit rates > 64 kbps and at bit rates reaching as low as 16 kbps.

10. Factors to be considered for Multiplexing and Transmission Split the video and audio coded bit streams into smaller data packets Multiplex with equal priority given to all elementary streams Detect packet losses and errors Additional information to help synchronize audio and video

11. Packetization H264 Encoder AAC Encoder Packetize r MultiplexerTransport Stream Video Source Audio Source MPEG encoded stream Data Source Packetizer PES 2 layers of packetization : PES - Packetized Elementary stream : Transport Stream :

12. Packetized Elementary stream (PES) Elementary streams (ES): Encoded video stream Encoded audio stream Data stream (Optional) PES contains access units that are sequentially separated and packetized PES headers distinguish different ES and contain timestamp information Packet size varies with the size of access units

13. Packetized Elementary stream (PES) AUDIO OR VIDEO ELEMENTARY STREAM PES HeaderPayload

14. PES Header Description 3 bytes of start code – 0x000001 1 byte of stream ID 2 bytes of packet length 2 bytes of time stamp (Frame number)

15. Frame number as time stamp Video frame rate : constant (25/30/.. fps) time = frame number/fps Audio sampling rate : constant (8 – 96 kHz) Number of samples/frame (AAC) : 1024 time = 1024*frame number/(sampling rate)

16. Advantages over the method that uses clock samples as time stamps Saves the extra header bytes used for sending program clock reference (PCR) information periodically No synchronization problem due to clock jitters No propagation of delay between audio and video Less complex and more suitable for software implementation

17. Transport Packets PES from various elementary sources are broken into smaller packets called transport packets Transport packets have a fixed length of 188 bytes Constraints Each packet can have data from only one PES PES header should be the first byte of the transport packet payload. Stuffing bytes are added if the above constraints are not met

18. Transport stream PES Payload PES Header Transport Header Transport Stream Packet Stuffing bytes

19. Packet Header SyntaxNumber of bits Sync byte8 PID10 Payload unit start indicator1 Adaptation field control1 Continuity counter4 if adaptation field control ==1 payload byte offset8 stuffing bytes or additional header payload

20. PID (Packet identifier) : Each elementary stream has a unique PID. Some are reserved for NULL packets and PSI (Program Specific Information). PSI (Program specific information) : Sequence parameter set and picture parameter set are sent as PSI at frequent intervals. Payload unit start indicator : 1 bit flag to indicate presence of PES header in the payload. Adaptation field control : 1 bit flag to indicate presence of any data other than PES data in payload. Packet Header

21. Continuity counter : 4 bit rolling counter which is incremented by 1 for each consecutive TS packet of the same PID. To detect packet loss. Payload Byte offset : If adaptation field control bit is ‘1’, byte offset value of the start of the payload or the length of adaptation field is mentioned here. Adaptation field : Stuffing bytes, if PES data < TS packet size Additional header information

22. Multiplexing method adopted Multiplexing method affects buffer fullness at the de-multiplexer and in turn playback Video and audio timing counters are used to ensure proper multiplexing Timing counters are incremented according to the playback time of each packet multiplexed PES with the least timing counter value is always given preference during packet allocation

23. Multiplexing method adopted Video PES => 1/25 = 40 ms fps = 25 4 TS packets => 40/4 = 10 ms PES length = 570 # of TS = round(570/185)

24. Multiplexed transport stream P1 V 0x2 P1 A 0x4 P1 A 0x5 P1 A 0x6 P1 V 0x3 NNNN P1 A 0x7 PID15 16 16 16 15 1024 16 Video PES Audio PES Transport stream

25. Demultiplexing

26. Buffer fullness at demultiplexer Test criteria Buffer details Video buffer 100kB Video buffer 600kB Video buffer 600kB Start TS packet number113850 End TS packet number80243417928 Video buffer fullness (kB)100600 Audio buffer fullness (kB)33142119 Number of video frames43211173 Number of audio frames82397326 Video buffer content playback time1.72 sec8.44 sec6.92 sec Audio buffer content playback time1.75 sec8.47 sec6.94 sec

27. Synchronization and playback During playback, data is loaded from the buffer IDR frame is searched from the top of the video buffer Frame number of IDR frame is extracted Corresponding audio frame number is calculated as follows Aframe number = ( Vframe number * sampling rate) / (1024*fps)

28. Synchronization and playback If a non-integer value, frame number is rounded off and the corresponding audio frame is searched. The audio and video contents from the corresponding frame numbers are decoded with PSI and played back. Then the audio and video buffers are emptied and incoming data gets buffered and the process continues. If corresponding audio frame is not found, next IDR frame is searched and same process is repeated.

29. Results Test Clip DetailsClip 1 Duration of clip (sec)125 YUV file size (kB)468,168 WAVE file size (kB)25,624 H.264 file size (kB)6,566 AAC file size (kB)2,066 Video encoder Compression ratio71.30 Audio encoder Compression ratio12.40 H.264 encoder bit rate (kBps)52.53 AAC encoder bit rate (kBps)16.53

30. Results Test Clip DetailsClip 1 H.264 encoder bit rate (kBps)52.53 AAC encoder bit rate (kBps)16.53 Number of TS packets50,577 Total transport stream size (kB)9,508 Compression ratio with header overhead51.93 Bit rate for transmission (kBps)76.06 AVI (*.avi) file size (kB)19,456 MPEG-2 movie (*.mpg) file size (kB)15,666

31. Synchronization results Start TS packet number Synchronized frame numbers chosen Video frame playback time (sec) Audio frame playback time (sec) Delay (msec) Visual delay VideoAudio 12018340.720.7255.22Not perceptible 20001021914.084.0745.97Not perceptible 600028252911.2811.2843.57Not perceptible 236891308245352.3252.3222.49Not perceptible 4525028265299113.04113.0455.33Not perceptible

32. Conclusions Synchronization of audio and video is achieved by starting de-multiplexing from any TS packet. Visually there is absolutely no lag between video and audio Bit rate can be changed by using rate control module in the H.264 encoder

33. Test Conditions Single program Transport stream is generated Input raw video : YUV format Input raw audio : WAVE format Profiles used : H.264 : Main profile AAC : Low complexity profile (ADTS format) GOP : IBBPBB (IDR forced) Video frame rate: 25fps Audio sampling frequency : 48 kHz

34. Future work Extension of the algorithm to multiplex multiple program streams Error correction method Reduce initial buffering time

35. References Books and Papers: [1]MPEG–2 advanced audio coding, AAC. International Standard IS 13818–7, ISO/IEC JTC1/SC29 WG11, 1997. [2]MPEG. Information technology — generic coding of moving pictures and associated audio information, part 3: Audio.International Standard IS 13818–3, ISO/IEC JTC1/SC29 WG11, 1994. [3]MPEG. Information technology — generic coding of moving pictures and associated audio information, part 4: Conformance testing.International Standard IS 13818–4, ISO/IEC JTC1/SC29 WG11, 1998. [4]Information technology—Generic coding of moving pictures and associated audio—Part 1: Systems, ISO/IEC 13818-1:2005, International Telecommunications Union. [5] MPEG-4: ISO/IEC JTC1/SC29 14496-10: Information technology – Coding of audio- visual objects - Part 10: Advanced Video Coding, ISO/IEC, 2005. [6] P. V. Rangan, S. S. Kumar, and S. Rajan, “Continuity and Synchronization in MPEG,” IEEE Journal on Selected Areas in Communications, Vol. 14, pp. 52-60, Jan. 1996. [7] B.J. Lechner et. al “The ATSC Transport Layer, Including Program and System Information Protocol (PSIP)”, Proc of the IEEE, vol. 94, no. 1,pp 77-101, January 2006

36. References [8] Hari Kalva et. al “Implementing Multiplexing, Streaming,and Server Interaction for MPEG-4”, IEEE transactions on circuits and systems for video technology, vol 9, No.8, pp 1299-1311,december 1999. [9] M. Bosi and M. Goldberg “Introduction to digital audio coding and standards”, Boston : Kluwer Academic Publishers, c2003. [10] D. K. Fibush, “Timing and Synchronization Using MPEG-2 Transport Streams,” SMPTE Journal, pp. 395-400,July, 1996. [11]K. Brandenburg, “MP3 and AAC Explained”, AES 17th International Conference, Florence, Italy, September 1999. [12] S-k. Kwon, A. Tamhankar and K.R. Rao ”Overview of H.264 / MPEG-4 Part 10”, J. Visual Communication and Image Representation, vol. 17, pp.183-552, April 2006. [13]A. Puri, X. Chen and A. Luthra, “Video coding using the H.264/MPEG-4 AVC compression standard”, Signal Processing: Image Communication, vol. 19, issue 9, pp. 793-849, Oct 2004. [14] T. Wiegand et. al “Overview of the H.264/AVC Video Coding Standard,” IEEE Trans. CSVT, Vol. 13, pp. 560-576, July 2003.

37. Reference [15] R. Hopkins, “United States digital advanced television broadcasting standard,” SPIE/IS & T, Photonics West, vol. CR61,pp 220-226, San Jose, CA, Feb. 1996. [16] Z. Cai et. al “A RISC Implementation of MPEG-2 TS Packetization”, in the proceedings of IEEE HPC conference, pp 688-691, May 2000. [17] M.Fieldler, “Implementation of basic H.264/AVC Decoder”, seminar paper at Chemnitz university of technology, June 2004 [18] R.Linneman, “Advanced audo coding on FPGA”, BS honours thesis, October 2002, School of Information Technology, Brisbane. [19] J. Watkinson, “The MPEG Handbook”, Second Edition, Oxford ; Burlington, MA : Elsevier/Focal Press, 2004. [20] I.E.G.Richardson, “H.264 and MPEG-4 Video Compression: Video Coding for Next Generation Multimedia”, John Wiley & Sons, 2003. [21]Proceedings of the IEEE, Special issue on Global Digital Television: Technology and Emerging Services, vol.94,pp 5-7, Jan. 2006. [22] P.D Symes “Digital video compression“, McGraw-Hill, c2004 [23] C. Wootton, “Practical guide to video and audio compression : from sprockets and rasters to macro blocks”, Oxford : Focal, 2005.

38. References [24] “FAAC and FAAD AAC software, website www.audiocoding.com [25] MPEG official website www.mpeg.org [26] Alternative AAC software from http://www.psytel-research.co.yu [27] H.264 software JM (10.2) from http://iphome.hhi.de/suehring/tml/ [28] Bauvigne G. “MPEG-2/MPEG-4 AAC”, MP3 Tech Website, www.mp3-tech.org [29] Whittle R., “Comparing AAC and MP3”, Website http://www.firstpr.com.au/audiocomp/aac- mp3-vq.html [30] Public discussion forum website for a/v containers: http://forum.doom9.org/forumdisplay.php?s=c68a3cd483892abb630cf026aa06d3c5&f [32] JVT documents website: http://www.dspr.com/www/technology/JVT-G050.pdf [33]Audio test files website http://www.rme-audio.com/english/download/audtest.htm [34]Reference for H.264 website http://www.vcodex.com/h264.html

39. Demultiplexer Video Buffer Audio buffer H.264 Decoder AAC Decoder Timestamp information Synchronized playback Transport stream