15 Nov Visualization of Music Donald Byrd School of Informatics & Jacobs School of Music Indiana University Updated 15 Nov Copyright © , Donald Byrd

15 Nov Basic Representations of Music & Audio (1) Audio (e.g., CD, MP3): like speech Time-stamped Events (e.g., MIDI file): like unformatted text Music Notation: like text with complex formatting

15 Nov Basic Representations of Music & Audio (2) Notation: very complex structure Audio: no (explicit) structure Events/MIDI: simple structure

15 Nov Basic and Specific Representations vs. Encodings Basic and Specific Representations (above the line) Encodings (below the line)

15 Nov A Music Collection: Visualization via “Islands of Music”, etc. Cooper, Matthew; Foote, Jonathan; Pampalk, Elias; & Tzanetakis, George (2006). Visualization in Audio-Based Music Information Retrieval. Computer Music Journal 30(2), pp. 42—62.

15 Nov Chopin Preludes in R: All at once; the C major alone

15 Nov Types of Music Visualizations What music is visualized? –Abstractions of music, no particular works –Collections of pieces –Individual pieces How is visualization organized? By time (horizontal structure) Other Performances or works? How dynamic? Dimensionality –2D (on paper or screen) –3D (simulated on screen, virtual reality, etc.)

15 Nov Types of Visualizations: Static or Changing? Changing (“animation”) resulting from scrolling is superficial –Effectively “semi-dynamic” Truly dynamic visualizations are less common

15 Nov Visualization Pitfalls Disorientation (navigation confusion) –Common with… Simulated 3D: most people aren’t good at Multiple visualizations –…but can occur w/ any poorly-designed interface Distracting glitziness Unresponsive Misleading –Cf. Edward Tufte’s books The more parameters, the harder to make everything clear –Music has loads of parameters!

15 Nov The Obvious Static Visualization The first few measures of Variation 8 of the “Twinkle” Variations

15 Nov Static Multiple Visualizations of Music via Text: Standard MIDI File (file dump) 0: 4D E0 4D54 MThd ‡MT 16: 726B FF B70 C000 FF58 rk......Q..p¿..X 32: FF 2F00 4D54 726B ñ4./.MTrk.. 48: FF E 6F U....Piano.êH8Å 64: C90 4A A 400C 904B 80: B40 0C90 4D D 400C 96: 904F F F : F D : 804D 400D FF2F 004D B : FF E6F 8F B Pianoè.êA+ÅdÄ 160: C C : C FF2F 192: 00.

15 Nov The Obvious Static Visualization: A Complex Case Complications on staff 2: Editorial additions (small notes) Instruments sharing notes only some of the time Mixed durations in double stops Multiple voices (divisi notation) Rapidly gets worse with more than 2!

15 Nov Static Visualizations Aligned by Time: Hendrix Experience, L.L.C.

15 Nov Static Visualizations Aligned by Time: Audacity & Don’s Audio Segmentation demo

15 Nov Static Visualizations Aligned by Time: Sonic Visualiser

15 Nov Static Visualization over Time: Variations2 Timeliner

15 Nov Static Visualization over Time: Chopin Mazurkas Project (Harmony) (from

15 Nov Semi-Dynamic Visualization over Time: Malinowski’s Music Animation Machine (from

15 Nov Dynamic Visualization over Time: Langner & Goebl’s “Performance worm”

15 Nov Why is Visualization Relevant to Music Informatics? Affects music representation Brings up HCI (Human/Computer Interaction) issues A powerful tool for finding patterns in information