Music Processing Roger B. Dannenberg

Overview  Music Representation  MIDI and Synthesizers  Synthesis Techniques  Music Understanding

Music Representation  Acoustic Level: sound, samples, spectra  Performance Information: timing, parameters  Notation Information: parts, clefs, stem direction  Compositional Structure: notes, chords, symbolic structure

Performance Information  MIDI bandwidth is 3KB/s, or 180KB/min  More typical: 3KB/minute, 180KB/hour Complete Scott Joplin: 1MB Output of 50 Composers (400 days of music): 500MB (1 CD-ROM)  Synthesis of acoustic instruments is a problem

Music Notation  Compact, symbolic representation  Does not capture performance information  Expressive “performance” not fully automated

Compositional Structure  Example: Nyquist (free software!) (defun melody1 () (seq (stretch q (note a4) (note b4) (note cs5) (note d5)))) (defun counterpoint () …) (defun composition () (sim (melody1) (counterpoint))) (play (transpose 4 (composition)))

MIDI: Musical Instrument Digital Interface  Musical Performance Information: Piano Keyboard key presses and releases “instrument” selection (by number) sustain pedal, switches continuous controls: volume pedal, pitch bend, aftertouch very compact (human gesture < 100Hz bandwidth)

MIDI (cont’d)  Point-to-point connections: MIDI IN, OUT, THRU Channels  No time stamps (almost) everything happens in real time  Asynchronous serial, 8-bit bytes+start+stop bits, 31.25K baud = 1MHz/32

MIDI Message Formats 8 chkey#vel Key Up 9 chkey#vel Key Down Program Change Polyphonic Aftertouch System Exclusive A chpresskey# C chindex# B chctrl#value Control Change Channel Aftertouch D chpress E chlo 7hi 7 Pitch Bend F 0 F E … DATA …

Standard MIDI Files  Key point: Must encode timing information  =1 or more, =, = midi data or, = FF =1 or more, =, = midi data or, = FF Delta times use variable length encoding, omit for zero. Interleave time differences with MIDI data...

Music Synthesis Introduction  Primary issue is control No control  Digital Audio (start, stop,...) Complete control  Digital Audio (S[0], S[1], S[2],... ) Parametric control  Synthesis

Music Synthesis Introduction (cont’d)  What parameters? pitch loudness timbre (e.g. which instrument) articulation, expression, vibrato, etc. spatial effects (e.g. reverberation)  Why synthesize? high-level representation provides precision of specification and supports interactivity

Additive Synthesis  amplitude A[i] and frequency  [i] specified for each partial (sinusoidal component)  potentially 2n more control samples than signal samples!

Additive Synthesis (cont’d)  often use piece-wise linear control envelopes to save space  still difficult to control because of so many parameters  and parameters do not match perceptual attributes

Table-Lookup Oscillators  If signal is periodic, store one period  Control parameters: pitch, amplitude, waveform Phase + Frequency Amplitude x n Efficient, but... n Spectrum is static n Efficient, but... n Spectrum is static (Note that phase and frequency are fixed point or floating point numbers)

FM Synthesis  Usually use sinusoids  “carrier” and “modulator” are both at audio frequencies  If frequencies are simple ratio ( R ), output spectrum is periodic  Output varies from sinusoid to complex signal as MOD increases A F AMPL out = AMPL· sin(2  ·FREQ· t + MOD sin(2  R ·FREQ· t )) + FREQMOD

FM Synthesis (cont’d)  Interesting sounds,  Time-varying spectra, and...  Low computation requirements  Often uses more than 2 oscillators … but …  Hard to recreate a specific waveform  No successful analysis procedure

 Samplers store waveforms for playback  Sounds are “looped” to extend duration  Spectrum is static (as in table- lookup), so: different samples are used for different pitches simple effects are added: filter, vibrato, amplitude envelope attack portion, where spectrum changes fastest, added to front Sample-based Synthesis AttackLoopLoop again...

Physical Models  Additive, FM, and sampling: more-or-less perception-based.  Physical Modeling is source-based: compute the wave equation, simulate attached reeds, bows, etc.  Example: ReedBoreBell

Physical Models (cont’d)  Difficult to control, and...  Can be very computationally intensive … but...  Produce “characteristic” acoustic sounds.

Music Understanding  Introduction  Score Following, Computer Accompaniment  Interactive Performance  Style Recognition  Conclusions