1. 15 Nov. 07 1 Visualization of Music Donald Byrd School of Informatics & Jacobs School of Music Indiana University Updated 15 Nov. 2007 Copyright © 2003-07, Donald Byrd

2. 15 Nov. 07 2 Basic Representations of Music & Audio (1) Audio (e.g., CD, MP3): like speech Time-stamped Events (e.g., MIDI file): like unformatted text Music Notation: like text with complex formatting

3. 15 Nov. 07 3 Basic Representations of Music & Audio (2) Notation: very complex structure Audio: no (explicit) structure Events/MIDI: simple structure

4. 15 Nov. 07 4 Basic and Specific Representations vs. Encodings Basic and Specific Representations (above the line) Encodings (below the line)

5. 15 Nov. 07 5 A Music Collection: Visualization via “Islands of Music”, etc. Cooper, Matthew; Foote, Jonathan; Pampalk, Elias; & Tzanetakis, George (2006). Visualization in Audio-Based Music Information Retrieval. Computer Music Journal 30(2), pp. 42—62.

6. 15 Nov. 07 6 Chopin Preludes in R: All at once; the C major alone

7. 15 Nov. 07 7 Types of Music Visualizations What music is visualized? –Abstractions of music, no particular works –Collections of pieces –Individual pieces How is visualization organized? By time (horizontal structure) Other Performances or works? How dynamic? Dimensionality –2D (on paper or screen) –3D (simulated on screen, virtual reality, etc.)

8. 15 Nov. 07 8 Types of Visualizations: Static or Changing? Changing (“animation”) resulting from scrolling is superficial –Effectively “semi-dynamic” Truly dynamic visualizations are less common

9. 15 Nov. 07 9 Visualization Pitfalls Disorientation (navigation confusion) –Common with… Simulated 3D: most people aren’t good at Multiple visualizations –…but can occur w/ any poorly-designed interface Distracting glitziness Unresponsive Misleading –Cf. Edward Tufte’s books The more parameters, the harder to make everything clear –Music has loads of parameters!

10. 15 Nov. 07 10 The Obvious Static Visualization The first few measures of Variation 8 of the “Twinkle” Variations

11. 15 Nov. 07 11 Static Multiple Visualizations of Music via Text: Standard MIDI File (file dump) 0: 4D54 6864 0000 0006 0001 0003 01E0 4D54 MThd.........‡MT 16: 726B 0000 0014 00FF 5103 0B70 C000 FF58 rk......Q..p¿..X 32: 0402 0218 0896 34FF 2F00 4D54 726B 0000.....ñ4./.MTrk.. 48: 0055 00FF 0305 5069 616E 6F00 9048 3881.U....Piano.êH8Å 64: 6480 4840 0C90 4A38 8164 804A 400C 904B dÄH@.êJ8ÅdÄJ@.êK 80: 3881 6480 4B40 0C90 4D38 8164 804D 400C 8ÅdÄK@.êM8ÅdÄM@. 96: 904F 3883 4880 4F40 1890 4F38 8360 9050 êO8ÉHÄO@.êO8É`êP 112: 3883 4880 4F40 1890 4D38 8330 8050 4018 8ÉHÄO@.êM8É0ÄP@. 128: 804D 400D FF2F 004D 5472 6B00 0000 3200 ÄM@../.MTrk...2. 144: FF03 0550 6961 6E6F 8F00 9041 2B81 6480...Pianoè.êA+ÅdÄ 160: 4140 0C90 4330 8164 8043 400C 9044 3181 A@.êC0ÅdÄC@.êD1Å 176: 6480 4440 0C90 4647 8164 8046 4001 FF2F dÄD@.êFGÅdÄF@../ 192: 00.

12. 15 Nov. 07 12 The Obvious Static Visualization: A Complex Case Complications on staff 2: Editorial additions (small notes) Instruments sharing notes only some of the time Mixed durations in double stops Multiple voices (divisi notation) Rapidly gets worse with more than 2!

13. 15 Nov. 07 13 Static Visualizations Aligned by Time: Hendrix Experience, L.L.C.

14. 15 Nov. 07 14 Static Visualizations Aligned by Time: Audacity & Don’s Audio Segmentation demo

15. 15 Nov. 07 15 Static Visualizations Aligned by Time: Sonic Visualiser

16. 15 Nov. 07 16 Static Visualization over Time: Variations2 Timeliner

17. 15 Nov. 07 17 Static Visualization over Time: Chopin Mazurkas Project (Harmony) (from http://www.mazurka.org.uk/info/keyscape/)

18. 15 Nov. 07 18 Semi-Dynamic Visualization over Time: Malinowski’s Music Animation Machine (from http://www.musanim.com/)

19. 15 Nov. 07 19 Dynamic Visualization over Time: Langner & Goebl’s “Performance worm”

20. 15 Nov. 07 20 Why is Visualization Relevant to Music Informatics? Affects music representation Brings up HCI (Human/Computer Interaction) issues A powerful tool for finding patterns in information