A language for auralizing data I’m Peyton Sherwood, the Project manager of the Upbeat language team, and I’m here to give you an introduction to Upbeat

Motivation Visual input overload! Automatic musical scoring Do you have too much information clouding your visual field? Would you like to consume that information in a less confrontational way?

Upbeat A flexible language to auralize input data aur al iz a tion (n): 1. the act or process of interpreting in aural or musical terms 2. the process of converting data into music Previous attempts haven’t sounded great Upbeat is a flexible language to auralize input data. Auralization is a word that we invented which simply means to turn data into music. Nyquist, Max/MSP, Supercollider

Easy Coding, Better Music Easy to use Familiar syntax for data processing Declarative musical syntax Portable: based on Java Sounds great! Goal for today: react to stock ticker Musician "StockMusician" {     phrase tune = "tune.mid";     "ChangePitch" {         Change Pitch by 3_semitones;     }     "Play" {         ||: tune :||;     } }

Coming up… Intro to Upbeat Upbeat Code Compiler Structure & Tools Peyton Sherwood (Project Manager) Upbeat Code Adrian Weller (Language Guru) Compiler Structure & Tools Fan Lin 林凡 (System Integrator) Runtime Framework Matt D’Zmura (System Architect) Testing & What We’ve Learned Miles Ulrich (Verification and Validation)

Sample Code Conductor Cues Musician “Keyboard” Musician “FunkyDrummer” TIME_SIGNATURE 4/4; TEMPO 140;   Conductor {     cue up = "GoingUp";     cue down = "GoingDown";     cue background = "GoingNowhere";     float param = 0.95;     float threshold = 5.0;     float Avg = 70.0;            // weighted average     Conduct(time currentMeasure) {         Avg = param*Avg + (1-param)*input.currentValue;         Case {          (currentMeasure==1)              Add background currentMeasure;             (input.currentValue > Avg + threshold)                 Add up currentMeasure;             (input.currentValue < Avg - threshold)                 Add down currentMeasure;         }     } } Musician "Keyboard" { phrase intro = "intro.mid";     phrase bridge = "bridge.mid";     phrase UpbeatRules = "upbeatrules.mid";               phrase SongSnippet = "goingdown.mid";                     "GoingNowhere" {      ||: intro | bridge :||;     }     "GoingUp" {         Change Volume to 100; // volume is always in [0,100]         ||: UpbeatRules :||2;     "GoingDown" {         Change Volume to 70;         ||: SongSnippet :||2 ;     }              Musician "FunkyDrummer" {     phrase introdrums = "introdrum.mid";     phrase bridgedrums = "bridgedrum.mid";                phrase funkydrums = "drumpatch.mid";     phrase cymbal = "cymbal.mid";     Change Instrument to 32;            // drums happen to be on MIDI channel 32         "GoingNowhere" {      ||: introdrums | bridgedrums :||;     "GoingUp" {         || funkydrums | cymbal ||; Conductor Cues Musician “Keyboard” CODE HERE Adrian talks about Conductor vs. musician Musician “FunkyDrummer”

Conductor cue up = "GoingUp"; cue down = "GoingDown"; TIME_SIGNATURE 4/4; TEMPO 140;   Conductor {     cue up = "GoingUp";     cue down = "GoingDown";     cue background = "GoingNowhere";     float param = 0.95;     float threshold = 5.0;     float Avg = 70.0;            // weighted average     Conduct(time currentMeasure) {         Avg = param*Avg + (1-param)*input.currentValue;         Case {          (currentMeasure==1)              Add background currentMeasure;             (input.currentValue > Avg + threshold)                 Add up currentMeasure;             (input.currentValue < Avg - threshold)                 Add down currentMeasure;         }     } } Musician "Keyboard" { phrase intro = "intro.mid";     phrase bridge = "bridge.mid";     phrase UpbeatRules = "upbeatrules.mid";               phrase SongSnippet = "goingdown.mid";                     "GoingNowhere" {      ||: intro | bridge :||;     }     "GoingUp" {         Change Volume to 100; // volume is always in [0,100]         ||: UpbeatRules :||2;     "GoingDown" {         Change Volume to 70;         ||: SongSnippet :||2 ;     }              Musician "FunkyDrummer" {     phrase introdrums = "introdrum.mid";     phrase bridgedrums = "bridgedrum.mid";                phrase funkydrums = "drumpatch.mid";     phrase cymbal = "cymbal.mid";     Change Instrument to 32;            // drums happen to be on MIDI channel 32         "GoingNowhere" {      ||: introdrums | bridgedrums :||;     "GoingUp" {         || funkydrums | cymbal ||; cue up = "GoingUp"; cue down = "GoingDown"; cue background = "GoingNowhere"; Avg = param*Avg + (1-param)*input.currentValue; Case { (currentMeasure==1)    Add background currentMeasure;     (input.currentValue > Avg + threshold)         Add up currentMeasure;     (input.currentValue < Avg - threshold)         Add down currentMeasure; } Conductor CODE HERE Adrian talks about Conductor vs. musician

Musician: “Keyboard” "GoingNowhere" { ||: intro | bridge :||; } TIME_SIGNATURE 4/4; TEMPO 140;   Conductor {     cue up = "GoingUp";     cue down = "GoingDown";     cue background = "GoingNowhere";     float param = 0.95;     float threshold = 5.0;     float Avg = 70.0;            // weighted average     Conduct(time currentMeasure) {         Avg = param*Avg + (1-param)*input.currentValue;         Case {          (currentMeasure==1)              Add background currentMeasure;             (input.currentValue > Avg + threshold)                 Add up currentMeasure;             (input.currentValue < Avg - threshold)                 Add down currentMeasure;         }     } } Musician "Keyboard" { phrase intro = "intro.mid";     phrase bridge = "bridge.mid";     phrase UpbeatRules = "upbeatrules.mid";               phrase SongSnippet = "goingdown.mid";                     "GoingNowhere" {      ||: intro | bridge :||;     }     "GoingUp" {         Change Volume to 100; // volume is always in [0,100]         ||: UpbeatRules :||2;     "GoingDown" {         Change Volume to 70;         ||: SongSnippet :||2 ;     }              Musician "FunkyDrummer" {     phrase introdrums = "introdrum.mid";     phrase bridgedrums = "bridgedrum.mid";                phrase funkydrums = "drumpatch.mid";     phrase cymbal = "cymbal.mid";     Change Instrument to 32;            // drums happen to be on MIDI channel 32         "GoingNowhere" {      ||: introdrums | bridgedrums :||;     "GoingUp" {         || funkydrums | cymbal ||; "GoingNowhere" { ||: intro | bridge :||; } "GoingUp" {     Change Volume to 100;     ||: UpbeatRules :||2; }       "GoingDown" {     Change Volume to 70;     ||: SongSnippet :||2 ; }              CODE HERE Adrian talks about Conductor vs. musician Musician: “Keyboard”

Coming up… Intro to Upbeat Upbeat Code Compiler Structure & Tools Peyton Sherwood (Project Manager) Upbeat Code Adrian Weller (Language Guru) Compiler Structure & Tools Fan Lin 林凡 (System Integrator) Runtime Framework Matt D’Zmura (System Architect) Testing & What We’ve Learned Miles Ulrich (Verification and Validation)

Compiler Structure Runtime Framework Music API Front End Lexer Parser jMusic API Wrapper Front End Lexer Parser .up main Java Compiler Our compiler could be divided into three parts: The front end, parsing the upbeat source code and produce source code in target language, which is java. The runtime framework, which will be compiled into a java library and will be used in the compiled java code. The music API, which actually is a part of the runtime framework, dealing with music generation. [Click] When we get a upbeat source code, the Lexer will tokenize the source code and give it to the parser to generate the java code. We put the generated java code and the runtime framework library together. And the main class will also be generated at this time. Then, using java compiler, we compile the generated code together with runtime library. Using JVM to run the target program, and generate music. JVM

System Integration Runtime Framework Music API Front End Lexer Parser jMusic API Wrapper main Front End Lexer Parser .up main main As a system integrator, here I briefly explain how we develop different components at the same time and integrate them together. Basically, we divide our team into 3 groups, they are Front End, Runtime framework and Music API. [Click] To decouple the connection between Front end and the Runtime framework. We created some Sample code which will be generated by the compiler. With these sample code, the runtime framework could be developed and even tested without any front end program. Then, when the front end is developing, they keep a copy of those sample code, and just trying to generate code looks like the sample code. To decouple runtime framework and Music API, we used some intermediate data structure to represent how different pieces of music are assembled. And the music API just read those data structures and generate music. By defining clear interfaces at the very begging, we could develop different components together and integrate them easily.

Tools Runtime Framework Music API Front End Lexer Parser API Wrapper jMusic API Wrapper main Front End Lexer Parser .up Since the compiler is developed using java, we choose those java-friendly tools. We use JFlex as our lexer generator Byacc for java as the parser generator And we found a excellent java music library jMusic to manipulate midi files and generate music Finally, eclipse is our main IDE, and we use google code to host our project and use subversion, a svn plug-in for eclipse to do the version control.

Coming up… Intro to Upbeat Upbeat Code Compiler Structure & Tools Peyton Sherwood (Project Manager) Upbeat Code Adrian Weller (Language Guru) Compiler Structure & Tools Fan Lin 林凡 (System Integrator) Runtime Framework Matt D’Zmura (System Architect) Testing & What We’ve Learned Miles Ulrich (Verification and Validation)

Compiled Upbeat Application Data File Compiled Upbeat Application Drums.mid Strings.mid Horn.mid Piano.mid Input Midi Files

Run-time Process Run-time Framework Measures Cues Input Music API M1: “Drums.mid” M3: “Strings.mid” Measure 1 M2: “Drums.mid” M4: “Strings.mid” Measure 2 Cues M1: GoingNowhere M9: GoingUp M13: GoingUp Data File Input Music API jMusic API Wrapper Keyboard Drummer Conductor Drums.mid Strings.mid Horn.mid Piano.mid Input Midi Files

Coming up… Intro to Upbeat Upbeat Code Compiler Structure & Tools Peyton Sherwood (Project Manager) Upbeat Code Adrian Weller (Language Guru) Compiler Structure & Tools Fan Lin 林凡 (System Integrator) Runtime Framework Matt D’Zmura (System Architect) Testing & What We’ve Learned Miles Ulrich (Verification and Validation)

Testing Setup Unit Tests Upbeat Source Files Lexer + Parser Target Java Code One source file per feature Validate against sample code End-to-End tests Upbeat Source Files Compiler Toolchain Target MIDI Files Collection of source, input Music verified by ear

What we’ve learned Agree on a metaphor early K.I.S.S. Project planning Schedule Weekly check-ins

UPBT: Stock Ticker