An XML Multi-layer Framework for Music Information Description Luca A. Ludovico LIM (Laboratorio di Informatica Musicale) DICo (Dipartimento di Informatica e Comunicazione) Università degli Studi di Milano Via Comelico, 39 - 20135 Milano, Italia

Motivations and State of the Art Part I Motivations and State of the Art Part I: Motivations and State of the Art Part II: Methodology of Work Part III: The MX Format Part IV: Scientific Advancement Part V: Conclusions

An XML Multi-layer Framework for Music Information Description Music contents Music contents are heterogeneous: Symbolic (scores from a logic standpoint) General (catalog metadata) Structural (relationships among music objects) Notational (graphical instances) Computer-driven performances (computer generated music) Audio/video (recordings) Other aspects (sketches, fashion plates, on-stage photos, playbills, etc.) 15/01/2019 An XML Multi-layer Framework for Music Information Description

Existing formats Music contents and corresponding formats Symbolic DARMS, NIFF, MusicXML, … General TXT, XML, database, … Structural [?] Notational EPS, JPEG, TIFF, … Computer-driven performances Csound, MIDI, SASL/SAOL, … Audio/video AIFF, MP3, WAV; AVI, MPEG, … Other EPS, JPEG, TIFF, … Possibility to describe only particular aspects of music information 15/01/2019 An XML Multi-layer Framework for Music Information Description

An XML Multi-layer Framework for Music Information Description State of the art Close, proprietary and non-standard formats Often binary (non-readable) formats Many formats to catch different aspects of music information No way to synchronize heterogeneous music information described in different formats 15/01/2019 An XML Multi-layer Framework for Music Information Description

Part II Methodology of Work Part I: Motivations and State of the Art Part III: The MX Format Part IV: Scientific Advancement Part V: Conclusions

An XML Multi-layer Framework for Music Information Description Steps Analysis of the problem from the point of view of Music Information Technology Communication Interaction with experts and other workgroups Choice of a technological framework Definition of a suitable format Validation 15/01/2019 An XML Multi-layer Framework for Music Information Description

Part III The MX Format Part I: Motivations and State of the Art Part II: Methodology of Work Part III: The MX Format Part IV: Scientific Advancement Part V: Conclusions

An XML Multi-layer Framework for Music Information Description The MX format MX is an XML-based format to achieve a comprehensive description of music information Key requirements Multimedia support Support for in-use formats Distinguishing Features Multi-layer structuring of (heterogeneous) music information Full synchronization among contents 15/01/2019 An XML Multi-layer Framework for Music Information Description

Multi-layer structuring Music information is heterogeneous Consequences on the format: From a qualitative standpoint, heterogeneous information should be supported From a quantitative standpoint, more objects of the same type should be supported Common structure referenced by all the layers: spine 15/01/2019 An XML Multi-layer Framework for Music Information Description

An XML Multi-layer Framework for Music Information Description MX layers General Music Logic Layers Structural Notational Performance Audio 15/01/2019 An XML Multi-layer Framework for Music Information Description

An XML Multi-layer Framework for Music Information Description MX layers in detail General - catalog metadata Logic Spine Logically Organized Symbols (LOS) Layout Structural - musicological analyses, relationships among music objects Notational - graphical aspect of score Performance - sub-symbolic formats for computer-driven performances Audio - audio formats General Logic Structural Notational Performance Audio 15/01/2019 An XML Multi-layer Framework for Music Information Description

An XML Multi-layer Framework for Music Information Description Spine Definition: a space-time construct In practice: music event identification Spacing and timing: virtual and relative Purposes: Glue among layers Synchronization 15/01/2019 An XML Multi-layer Framework for Music Information Description

An XML Multi-layer Framework for Music Information Description 15/01/2019 An XML Multi-layer Framework for Music Information Description

Intra-layer synchronization It takes place among objects belonging to the same layer: homogeneous information 15/01/2019 An XML Multi-layer Framework for Music Information Description

Inter-layer synchronization It takes place among different layers: heterogeneous information 15/01/2019 An XML Multi-layer Framework for Music Information Description

How synchronization is achieved 15/01/2019 An XML Multi-layer Framework for Music Information Description

Part IV: Scientific Advancement Part I: Motivations and State of the Art Part II: Methodology of Work Part III: The MX Format Part IV: Scientific Advancement Part V: Conclusions

An XML Multi-layer Framework for Music Information Description Thesis contributions Definition of a reference unified framework for XML-based music specification: integrated with pre-existing partial formats multi-layered around the spine core element incorporating synchronization capabilities 15/01/2019 An XML Multi-layer Framework for Music Information Description

An XML Multi-layer Framework for Music Information Description Synchronization Analysis of different facets of music information. Design and implementation of XML descriptors for the synchronization of heterogeneous contents. Design and implementation of algorithms to allow advanced music enjoyment. 15/01/2019 An XML Multi-layer Framework for Music Information Description

An XML Multi-layer Framework for Music Information Description Contribution to IEEE P1599 Institute of Electrical and Electronics Engineers (IEEE) Computer Society 1992 - Task Force on Computer Generated Music 1994 - Technical Committee on Computer Generated Music 2001 - Project Authorization Request 1599 (IEEE P1599): “Definition of a Commonly Acceptable Musical Application Using the XML Language” 2002 - “MAX2002 – First International Conference on Musical Application using XML” 2006 – PhD thesis: “An XML Multi-layer Framework For Music Information Description” March 2007 – Balloting 15/01/2019 An XML Multi-layer Framework for Music Information Description

MX-based Applications Three ways to enjoy music contents Simple synchronized fruition Real-time switch among multimedia contents User-interaction with immediate synchronization An example: MX Navigator 15/01/2019 An XML Multi-layer Framework for Music Information Description

List of MX-related papers HAUS, Goffredo, e LUDOVICO, Luca A., Music Segmentation: an XML-Oriented Approach, CMMR 2004 Post-Symposium Proceedings, Lecture Notes in Computer Science (LNCS), Springer Verlag, 2004 CASTANO, Silvana, FERRARA, Alfio, HAUS, Goffredo, LUDOVICO, Luca Andrea, MONTANELLI, Stefano, RACCA, Gianpaolo, e VERCELLESI, Giancarlo, The MX Formalism for Semantic Web-Compatible Representation of Music Metadata, International Workshop on Multimedia Information Systems (MIS) 2005 Proceedings, Lecture Notes in Computer Science (LNCS), Springer Verlag, 2005 BARATE', Adriano, HAUS, Goffredo, LUDOVICO, Luca Andrea,  e VERCELLESI, Giancarlo, MXDemo: a Case Study about Audio, Video, and Score Synchronization, AXMEDIS 2005 Proceedings, Firenze, 2005 BAGGI, Denis, BARATE', Adriano, HAUS, Goffredo, e LUDOVICO, Luca A., A Computer Tool to Enjoy and Understand Music, EWIMT 2005 Workshop Proceedings, London, 2005 CASTANO, Silvana, FERRARA, Alfio, HAUS, Goffredo, LUDOVICO, Luca Andrea, MONTANELLI, Stefano, A Semantic Web Ontology for Context-based Classification and Retrieval of Music Resources, ACM Transactions on Multimedia Computing, Communications, and Applications. BARATE', Adriano, HAUS, Goffredo, e LUDOVICO, Luca A., An XML-Based Format for Advanced Music Fruition, Sound and Music Computing Conference 2006 (SMC 2006), Marseille. BAGGI, Denis, BARATE', Adriano, HAUS, Goffredo, e LUDOVICO, Luca A., Developing Intuition in Engineering Education: New Technology To Capture Structures in Music, 35th International IGIP Symposium, Tallinn, 2006. BARATE', Adriano, HAUS, Goffredo, e LUDOVICO, Luca A., MX Navigator: An Application for Advanced Music Fruition, AXMEDIS 2006, Leeds, 2006. BARATE', Adriano, HAUS, Goffredo, e LUDOVICO, Luca A., Music representation of score, sound, MIDI, structure and metadata all integrated in a single multilayer environment based on XML, Intelligent Music Information Systems, IRM Press, 2006 BARATE', Adriano, e LUDOVICO, Luca A., An XML-based Synchronization of Audio and Graphical Representations of Music Scores, Proceedings of the 8th International Workshop on Image Analysis for Multimedia Interactive Services (WIAMIS 2007), Santorini, Greece, 2007 15/01/2019 An XML Multi-layer Framework for Music Information Description

Part IV: Conclusions Part I: Motivations and State of the Art Part II: Methodology of Work Part III: The MX Format Part IV: Scientific Advancement Part V: Conclusions

An XML Multi-layer Framework for Music Information Description Recent work New elements and attributes to catch other kinds of description, models, forms of analysis, etc. Implementation of an SDK in C# aimed at MX encoding Implementation of an MX integrated editor/viewer IEEE Balloting 15/01/2019 An XML Multi-layer Framework for Music Information Description

Further research issues Integration of ontologies for music information retrieval inside General layer [Ferrara et al. 2005] Automatic translation among layers and automatic generation of synchronization information Automatic segmentation Automatic extraction of information 15/01/2019 An XML Multi-layer Framework for Music Information Description

An XML Multi-layer Framework for Music Information Description Questions Thank you for your attention! 15/01/2019 An XML Multi-layer Framework for Music Information Description