Acoustics Research Institute Austrian Academy of Science MPEG-7 Todays Multimedia Standard Peter Balazs Institut für Schallforschung der Österreichischen Akademie der Wissenschaften: A-1010 Wien; Liebiggasse 5. Tel / ; Fax +43 1/ ; OeAW-ISF Peter Balazs 1999 started as programmer at the ISF 2001 finshed mathematics (University of Vienna)

MPEG-7 OeAW-ISF ISO / IEC Standard Mulitmedia Content Description Interface Multimedia data / metadata description system Low Level – High Level; content based Open system Inheritance Description of methods normativ – informativ

MPEG-7 OeAW-ISF ISO / IEC Standard Mulitmedia Content Description Interface Multimedia data / metadata description system Low Level – High Level Open system Inheritance Description of methods normativ – informativ IDDogBarks IDState IDState IDState IDState IDState IDState

MPEG-7 OeAW-ISF History Call for Proposals October 1998 Evaluation February 1999 First version of Working Draft (WD) December 1999 Committee Draft (CD) October 2000 Final Committee Draft (FCD) February 2001 Final Draft International Standard (FDIS) July 2001 International Standard (IS) September 2001 Development Amendment AudioMay 2002 Call for Proposals (Systems, version 2)July 2002 MPEG 21 international standardApril 2009

XML = eXtensible Markup Language XML OeAW-ISF Metasprache Hypertext Markup markup = tag... Open Standard <!DOCTYPE document [ <!ELEMENT ADRESSE (Vorname, Nachname, Wohnort)>.... ]> <!DOCTYPE document [ <!ELEMENT ADRESSE (Vorname, Nachname, Wohnort)>.... ]> Peter Balazs Tulln <!DOCTYPE document [ <!ELEMENT ADRESSE (Vorname, Nachname, Wohnort)>.... ]> Peter Balazs Tulln

XML = eXtensible Markup Language XML OeAW-ISF Metasprache Hypertext Markup markup = tag... Open Standard <!DOCTYPE document [ <!ELEMENT ADRESSE (Vorname, Nachname, Wohnort)>.... ]> Peter Balazs Tulln CursorOpts = SignalOpts= 1 1 FrameOpts= GraphXY= 0 1e Method= Average=

MPEG-7 OeAW-ISF Descriptors Low Level Descriptor Schemes High Level, container Descriptor Definition Language (DDL) XML Schema, STX Schema System Tools ASCII Text - binary

MPEG-7 OeAW-ISF Out of [1]

OeAW-ISF MPEG-7 Audio: Low Level Descriptors Single Sample Segments DS, compare to STX Out of [1]

OeAW-ISF MPEG-7 Audio: Low Level Descriptors Scalar Vector Single Series series of vectors = table, matrix Scalable Series Out of [2]

OeAW-ISF MPEG-7 Audio: Low Level Descriptors Basic AudioWaveform, AudioPower

OeAW-ISF MPEG-7 Audio: Low Level Descriptors Basic AudioWaveform, AudioPower Basic Spectral AudioSpectrumEnvelope, AudioSpectrumCentroid, AudioSpectrumSpread, AudioSpectrumFlatness

OeAW-ISF MPEG-7 Audio: Low Level Descriptors Basic AudioWaveform, AudioPower Basic Spectral AudioSpectrumEnvelope, AudioSpectrumCentroid, AudioSpectrumSpread, AudioSpectrumFlatness Signal Parameters AudioHarmonicity, AudioFundamentalFrequency

OeAW-ISF MPEG-7 Audio: Low Level Descriptors Basic AudioWaveform, AudioPower Basic Spectral AudioSpectrumEnvelope, AudioSpectrumCentroid, AudioSpectrumSpread, AudioSpectrumFlatness Signal Parameters AudioHarmonicity, AudioFundamentalFrequency Timbral Temporal LogAttackTime, TemporalCentroid

OeAW-ISF MPEG-7 Audio: Low Level Descriptors Basic AudioWaveform, AudioPower Basic Spectral AudioSpectrumEnvelope, AudioSpectrumCentroid, AudioSpectrumSpread, AudioSpectrumFlatness Signal Parameters AudioHarmonicity, AudioFundamentalFrequency Timbral Temporal LogAttackTime, TemporalCentroid Timbral Spectral SpectralCentroid, HarmonicSpectralCentroid, HarmonicSpectralDeviation, HarmonicSpectralSpread, HarmonicSpectralVariation

OeAW-ISF MPEG-7 Audio: Low Level Descriptors Basic AudioWaveform, AudioPower Basic Spectral AudioSpectrumEnvelope, AudioSpectrumCentroid, AudioSpectrumSpread, AudioSpectrumFlatness Spectral Basis AudioSpectrumBasis, AudioSpectrumProjection Signal Parameters AudioHarmonicity, AudioFundamentalFrequency Timbral Temporal LogAttackTime, TemporalCentroid Timbral Spectral SpectralCentroid, HarmonicSpectralCentroid, HarmonicSpectralDeviation, HarmonicSpectralSpread, HarmonicSpectralVariation Out of [1]

OeAW-ISF MPEG-7 Audio: Low Level Descriptors Basic AudioWaveform, AudioPower Basic Spectral AudioSpectrumEnvelope, AudioSpectrumCentroid, AudioSpectrumSpread, AudioSpectrumFlatness Spectral Basis AudioSpectrumBasis, AudioSpectrumProjection Signal Parameters AudioHarmonicity, AudioFundamentalFrequency Timbral Temporal LogAttackTime, TemporalCentroid Timbral Spectral SpectralCentroid, HarmonicSpectralCentroid, HarmonicSpectralDeviation, HarmonicSpectralSpread, HarmonicSpectralVariation Out of [1]

OeAW-ISF MPEG-7 Audio: Low Level Descriptors Basic AudioWaveform, AudioPower Basic Spectral AudioSpectrumEnvelope, AudioSpectrumCentroid, AudioSpectrumSpread, AudioSpectrumFlatness Spectral Basis AudioSpectrumBasis, AudioSpectrumProjection Signal Parameters AudioHarmonicity, AudioFundamentalFrequency Timbral Temporal LogAttackTime, TemporalCentroid Timbral Spectral SpectralCentroid, HarmonicSpectralCentroid, HarmonicSpectralDeviation, HarmonicSpectralSpread, HarmonicSpectralVariation Out of [2] Silence Out of [1]

OeAW-ISF MPEG-7 Audio: High Level DSs AudioSignature AudioSpectrumFlatness

OeAW-ISF MPEG-7 Audio: High Level DSs AudioSignature AudioSpectrumFlatness Musical Instrument Timbre Description Tool HarmonicInstrumentTimbre (LAT + timbre spectral) PercussiveInstrumentTimbre (timbre temporal + SpectralCentroid)

OeAW-ISF MPEG-7 Audio: High Level DSs AudioSignature AudioSpectrumFlatness Musical Instrument Timbre Description Tool HarmonicInstrumentTimbre (LAT + timbre spectral) PercussiveInstrumentTimbre (timbre temporal + SpectralCentroid) Melody Description Tools MelodyContour DS, Melody Sequence DS

OeAW-ISF MPEG-7 Audio: High Level DSs AudioSignature AudioSpectrumFlatness Musical Instrument Timbre Description Tool HarmonicInstrumentTimbre (LAT + timbre spectral) PercussiveInstrumentTimbre (timbre temporal + SpectralCentroid) Melody Description Tools MelodyContour DS, Melody Sequence DS General Sound Recognition and Indexing Description Tool SpectralBasis, SoundClassificationModel : SoundModels, classification scheme; SoundModelStatePath, SoundModelStateHistogram

OeAW-ISF MPEG-7 Audio: High Level DSs AudioSignature AudioSpectrumFlatness Musical Instrument Timbre Description Tool HarmonicInstrumentTimbre (LAT + timbre spectral) PercussiveInstrumentTimbre (timbre temporal + SpectralCentroid) Melody Description Tools MelodyContour DS, Melody Sequence DS General Sound Recognition and Indexing Description Tool SpectralBasis, SoundClassificationModel : SoundModels, classification scheme; SoundModelStatePath, SoundModelStateHistogram SpokenContentDescription Tools SpokenContentHeader : WordLexicon, PhonLexicon; SpokenContentLattice: WordLinks, PhonLinks.

OeAW-ISF MPEG-7 Audio: Amendment New Base types optional attribute for channel Modification of Spoken Content Description Tools acoustics only score possible for speech recognition; prosody, syllabels Audio Signal Quality DS BackgroundNoiseLevel, BalanceType, DCoffsetType, BandwidthType. TransmissionTechnologyType: shellac, vinyl,.... Additional Tools: tempo description, compact variable precision representation (BAM) Liguistic Description Tools: semantic structure of liguistic data

OeAW-ISF MPEG-7 Literatur: [1] José M. Martínez, MPEG-7 Overview (version 8) ISO/IEC JTC1/SC29/WG11N4980, Klagenfurt, July 2002, [2] ISO / IEC, Information Technology – Multimedia Content Description Interface – Part 4: Audio, Geneva, July 2001 [3] Oliver Pott, Günter Wielange, XML Praxis und Referenz, München 2001 [4] J. Bitzer, J. H. Martínez, Information Technology Multimedia Content Description Interface Part 4: Audio Proposed Draft Amendment, Fairfax, May 2002 Links: [4] MPEG Home Page, [5] Extensible Markup Language, [6] STX,