ISO Media Based File Format and its Derivatives DPIF Symposium at Dresden, Germany Wo Chang, NIST Digital Media Group, Manager wchang@nist.gov Chaired of MPEG Query Format AHG Chaired of MPEG Multimedia Application Formats AHG Chaired of MPEG-7 Profile AHG Chaired of MPEG-4 Reference Software AHG Co-Chaired of JPEG Search Subgroup DPIF Symposium at Dresden, Germany, April 21 – 23, 2010, Wo Chang, NIST <#>

ISO Base Media File Format Objectives The ISO Base Media File Format (BMFF) provides structural and media data information for timed presentations of media data such as audio, video, metadata, etc. The structure is intentionally general, so that by structuring files in different ways the same base specification can be used for files in: Capture Exchange and download, including incremental download and play Local playback Editing, composition, and lay-up Streaming from streaming servers DPIF Symposium at Dresden, Germany, April 21 – 23, 2010, Wo Chang, NIST <#>

Some History about MPEG-4 (14496) Reorganization of ISO/IEC 14496-1 3rd edition, Oct., 2002, N5293 Before 14496-1 Amd.1, Amd.2, Cor.1, Cor.2 Amd.5 (ISO format) Amd.3 (IPMP) Amd.6 (MPEG-4 file format) Amd.7 (AVC file format) After Part 1 3rd Part 11 Part 12 Part 13 Part 14 Part 15 DPIF Symposium at Dresden, Germany, April 21 – 23, 2010, Wo Chang, NIST <#>

ISO Base Media File Format (BMFF) family of standards (14496-12) 1st Edition, April, 2003 Dec., 2001 Apple’s QuickTime (QT) File Format (*.mov) 3GPP & 3GPP2 File Format (.3gp & .3g2) AVC File Format (14496-15) MPEG-4 File Format (14496-14) Higher than 8 bits per sample Higher chroma sampling (4:2:2, 4:4:4) Alpha blending on multiple video scenes others… Motion JPEG2000 File Format (15444-3) (.mj2) FRExt * AVC File Format (14496-15 Amd-1) Serve users with different displays and network by single bit stream for: Spatial scalability (suitable resolution) Temporal scalability (suitable frame rate) Quality scalability (suitable data rate) (overview: N9709) SVC AVC File Format (14496-15 Amd-2) MPEG-21 File Format (21000-9) MVC AVC File Format (14496-15 Amd-3) 3D video for 3D depth impression Free viewpoint video for selectable view (overview: N9580) *.mp4 files * http://www.fastvdo.com/spie04/spie04-h264OverviewPaper.pdf DPIF Symposium at Dresden, Germany, April 21 – 23, 2010, Wo Chang, NIST <#>

ISO Base Media File Format (BMFF) family of standards Temporal/inter-view prediction structure for MVC DPIF Symposium at Dresden, Germany, April 21 – 23, 2010, Wo Chang, NIST <#>

ISO Base Media File Format (BMFF) family of standards MPEG-4 File Format – File-based mp4 file Interleaved, time-ordered, BIFS, OD, video, and audio access units (AC) moov mdat IOD trak (BIFS) trak (OD) trak (video) … other boxes trak (audio) DPIF Symposium at Dresden, Germany, April 21 – 23, 2010, Wo Chang, NIST <#>

ISO Base Media File Format (BMFF) family of standards MPEG-4 File Format – Streaming-based mp4 file Interleaved, time-ordered, BIFS, OD, video, audio access units (AC), and “hint” instructions moov mdat IOD trak (BIFS) trak (OD) trak (video) … other boxes trak (audio) hint DPIF Symposium at Dresden, Germany, April 21 – 23, 2010, Wo Chang, NIST <#>

ISO File Format (FF) family of standards 14496-12: Base Media FF, 1st Edition, Apr. 2003, N5295 14496-12 Amd-1: add metadata + enabling AVC & MPEG-21, Mar. 2004, N6323 14496-12 Amd-2: add timed metadata + non-square pixels and clean aspect ratio in video + improve design for sample group descriptions, Mar. 2004, N6323 14496-12: Base Media FF, 2nd Edition, Apr. 2005 14496-12: 2nd Edition Amd-1: add Hint track + multiple metadata box + streaming server instructions, Nov. 2006, N8661 14496-14: MPEG-4 File Format, Apr., 2003, N5298 14496-15: Advance Video Coding File Format (AVC FF), Aug. 2003, N5780 14496-15 Amd-1: add FRExt (Fidelity Range Ext.), Apr. 2005, N7235 14496-15 Amd-2: add SVC (Scalable Video Coding), Jan. 2008, N9682 14496-15 Amd-3: add MVC (MultiView Video Coding), Oct. 2008, N10253 21000-9: MPEG-21 File Format, Jan. 2005, N6975 DPIF Symposium at Dresden, Germany, April 21 – 23, 2010, Wo Chang, NIST <#>

ISO BMFF Architecture Overview based on “box” structure Example of a simple in ISO/BM File Example of static un-timed metadata in ISO/MP4 File DPIF Symposium at Dresden, Germany, April 21 – 23, 2010, Wo Chang, NIST <#>

ISO BMFF Architecture Overview based on “box” structure ftyp * 4.3 file type and compatibility pdin 8.43 progressive download information moov 8.1 container for all the metadata mvhd 8.3 movie header, overall declarations trak 8.4 container for an individual track or stream tkhd 8.5 track header, overall information about the track tref 8.6 track reference container edts 8.25 edit list container elst 8.26 an edit list mdia 8.7 container for the media information in a track mdhd 8.8 media header, overall information about the media hdlr 8.9 handler, declares the media (handler) type minf 8.10 media information container vmhd 8.11.2 video media header, overall information (video track only) smhd 8.11.3 sound media header, overall information (sound track only) hmhd 8.11.4 hint media header, overall information (hint track only) nmhd 8.11.5 Null media header, overall information (some tracks only) dinf 8.12 data information box, container dref 8.13 data reference box, declares source(s) of media data in track stbl 8.14 sample table box, container for the time/space map stsd 8.16 sample descriptions (codec types, initialization etc.) stts 8.15.2 (decoding) time-to-sample ctts 8.15.3 (composition) time to sample stsc 8.18 sample-to-chunk, partial data-offset information stsz 8.17.2 sample sizes (framing) stz2 8.17.3 compact sample sizes (framing) stco 8.19 chunk offset, partial data-offset information co64 64-bit chunk offset stss 8.20 sync sample table (random access points) DPIF Symposium at Dresden, Germany, April 21 – 23, 2010, Wo Chang, NIST <#>

ISO BMFF Architecture Overview based on “box” structure stsh 8.21 shadow sync sample table padb 8.23 sample padding bits stdp 8.22 sample degradation priority sdtp 8.40.2 independent and disposable samples sbgp 8.40.3.2 sample-to-group sgpd 8.40.3.3 sample group description subs 8.42 sub-sample information mvex 8.29 movie extends box mehd 8.30 movie extends header box trex * 8.31 track extends defaults ipmc 8.45.4 IPMP Control Box moof 8.32 movie fragment mfhd 8.33 movie fragment header traf 8.34 track fragment tfhd 8.35 track fragment header trun 8.36 track fragment run mfra 8.37 movie fragment random access tfra 8.38 track fragment random access mfro 8.39 movie fragment random access offset mdat 8.2 media data container free 8.24 free space skip udta 8.27 user-data cprt 8.28 copyright etc. meta 8.44.1 metadata hdlr 8.9 handler, declares the metadata (handler) type dinf 8.12 data information box, container dref 8.13 data reference box, declares source(s) of metadata items iloc 8.44.3 item location ipro 8.44.5 item protection sinf 8.45.1 protection scheme information box frma 8.45.2 original format box imif 8.45.3 IPMP Information box schm 8.45.5 scheme type box schi 8.45.6 scheme information box iinf 8.44.6 item information xml 8.44.2 XML container bxml binary XML container pitm 8.44.4 primary item reference DPIF Symposium at Dresden, Germany, April 21 – 23, 2010, Wo Chang, NIST <#>

Musical Slideshow MAF FF ftyp moov meta trak iloc/iinf MP3 I tem_ID = 1 Content_type = audio/mp3 JPEG 1 Item_ID = 2 Content_t ype = image/jpeg JPEG 2 Item_ID = 3 Content_type = image/jpeg … JPEG N Item_ID = N C ontent_type = image/jpeg Text Item_ID = N+1 Content_type = text xml XML for MPEG - 4 LASeR mdia stbl for JPEG stbl for MP3 stbl for Timed Text MPEG 7 SMP mdat JPEGs 1 2 N DPIF Symposium at Dresden, Germany, April 21 – 23, 2010, Wo Chang, NIST <#>

Musical Slideshow MAF FF metadata MPEG-7 XML or MPEG-21 DID MP3 JPEG Music Slide Show DPIF Symposium at Dresden, Germany, April 21 – 23, 2010, Wo Chang, NIST <#>

Portable Video Player MAF FF DPIF Symposium at Dresden, Germany, April 21 – 23, 2010, Wo Chang, NIST <#>

Portable Video Player MAF FF File load/play Main menu LASeR handling capability? Go to submenus Show menu Play video Play main feature Play extra materials Video track 1 DPIF Symposium at Dresden, Germany, April 21 – 23, 2010, Wo Chang, NIST <#>

Stereoscopic Video MAF FF DPIF Symposium at Dresden, Germany, April 21 – 23, 2010, Wo Chang, NIST <#>

Stereoscopic Video MAF FF DPIF Symposium at Dresden, Germany, April 21 – 23, 2010, Wo Chang, NIST <#>

Digital Multimedia Broadcast MAF FF DPIF Symposium at Dresden, Germany, April 21 – 23, 2010, Wo Chang, NIST <#>

Digital Multimedia Broadcast MAF FF Audio Video Java Application Web Pages Slideshow POI *POI (point of interest) 1 4 2 Communication Networks 3 DPIF Symposium at Dresden, Germany, April 21 – 23, 2010, Wo Chang, NIST <#>

Interactive Music MAF FF DPIF Symposium at Dresden, Germany, April 21 – 23, 2010, Wo Chang, NIST <#>

Interactive Music MAF FF DPIF Symposium at Dresden, Germany, April 21 – 23, 2010, Wo Chang, NIST <#>