Songsmith: Dan Morris, Ian Simon, Sumit Basu, and the MSR Advanced Development Team Microsoft Research Using Machine Learning to Help People Make Music

Songsmith: Dan Morris, Ian Simon, Sumit Basu, and the MSR Advanced Development Team Microsoft Research Using Machine Learning to Help People Make Music

Computational User Experiences (CUE) group In general: HCI + (sensors, devices, machine learning, health, physiology)

Computational User Experiences (CUE) group  Using physiological signals for input

Computational User Experiences (CUE) group  Using physiological signals for input  Health and wellness

Computational User Experiences (CUE) group  Using physiological signals for input  Health and wellness  Creativity support tools

Songsmith: Dan Morris, Ian Simon, Sumit Basu, and the MSR Advanced Development Team Microsoft Research Using Machine Learning to Help People Make Music

What is Songsmith? Songsmith Your music You, singing... “Automatic accompaniment generation for vocal melodies”

High-Risk Live Demo (What other kind of live demo is there?)

Who is Songsmith for?

Today’s Talk  Overview and demo  How Songsmith works  Exposing machine learning parameters  What are people doing with Songsmith?  Creativity support Microsoft Research

Songsmith: 5000’ Overview G Amin C Daug

Chords from Melody  Songsmith’s core: Hidden Markov Model Song start Chord 1 Chord 2 Chord 3 Chord 4 Song end Hidden states (chords) Observations (note vectors)

States Observations What does an HMM do for me? What does an HMM want from me? HMMs in 5 Minutes or Less

States Observations HMMs in 5 Minutes or Less Possible states: C major C minor C diminished … … Transition probabilities: P(C Major | A Minor)? P(C Major | D Major)? P(F# Major | F# Major)? Observation probabilities: P( | A Minor)? P( | D Major)?

Building our HMM  Things my HMM wants from me: Possible states Transition probabilities Observation probabilities Observations

 Data-driven  Not heuristic-driven Finding the Probabilities

Training Data ~300 lead sheets (vocal melodies with chords)

C MajorC MinorC# MajorC# MinorD MajorD Minor… C Major C Minor C# Major……………… C# Minor……………… D Major……………… D Minor……………… …………………  Convert all chords to five basic triads  Transpose every song into the same key  Count the number of transitions from every chord to every other chord (transition probabilities):  Count the total duration of each melody note occurring while each chord is playing (observation probabilities): Processing the Database Notes played over C Major:

Building our HMM  Things my HMM wants from me: Possible states Transition probabilities Observation probabilities Observations

Observations: what did the user sing? Input: Hard!

Building a Pitch Histogram FFT

Building our HMM  Things my HMM wants from me: Possible states Transition probabilities Observation probabilities Observations  Run Viterbi algorithm, get “best” sequence of chords… thank you, HMM!  One hitch: key determination…

Today’s Talk  Overview and demo  How Songsmith works  Exposing machine learning parameters  What are people doing with Songsmith?  Creativity support Microsoft Research

So we can choose optimal chords... so what?  My demo took 10 seconds… is that fun?  What would a songwriter do next?  How can we build creative exploration into a learning-driven system?

 There are always hard-coded “magic numbers” in machine learning  Machine learning also use lots of learned parameters  Can we let users control those numbers? UI: Exposing Learning Parameters

A Bad User Interface Songsmith: A fun way to make music (if you have a PhD in math and/or computer science) Observation weight: Transition matrix (edit me!) CC#DD#E C C# D D# E Expected pitch histograms (edit me!) C C# D D# E F Frequency smoothing: Conjugate prior:

 Can we let users control those numbers? UI: Exposing Learning Parameters  Can we let users intuitively control those numbers?

Exposing Model Parameters in Songsmith

The “Happy Factor”

 Partition database into two databases (major and minor songs) using clustering  Build separate transition probability matrix for each database  When we actually run our HMM, blend the two transition matrices together according to user input… Happy Factor: Implementation

The “Happy Factor” Bonus question: what’s wrong with this equation?

Another “Happy Factor” Example Happy Sad

The “Jazz Factor”

 When running our HMM, we need to make chords match the voice and each other  Computing how well each chord fits at a given position: k log( P(this chord | what the user sang) ) + (1-k) log( P(this chord | the previous chord) )  Just put k on a slider! Jazz Factor: Implementation

Chord Locking  Global sliders are very coarse  Chords can be “locked” by the user

“Suggested Chords”  Songwriters will often explore “chord substitutions”…  …but we’re assuming our audience doesn’t know that much music theory…  Expose suboptimal marginal probabilities at each node as “suggestions”

Interactive Machine Learning  Songsmith is one example of IML Roughly: moving what used to be in the domain of ML experts into users’ hands  Related work: image classification Fogarty et al, CHI 2008: CueFlik Fails and Olsen, IUI 2003: Crayons  Why? Harness end-user knowledge Use ML as a tool for data exploration Use ML as a tool for creative expression

Today’s Talk  Overview and demo  How Songsmith works  Exposing machine learning parameters  What are people doing with Songsmith?  Creativity support Microsoft Research

Today’s Talk  Overview and demo  How Songsmith works  Exposing machine learning parameters  What are people doing with Songsmith?  Creativity support Microsoft Research

Dynamic Mapping of Physical Controls for Tabletop Groupware (Fiebrink, CHI 2009) One project, two problems: 1.Direct-touch, tabletop input is great for collaboration… …but suffers from serious precision issues. 2.Working on music alone is boring.

 Incorporate high-precision controllers into a tabletop environment  Evaluate in a collaborative audio-editing app Dynamic Mapping of Physical Controls for Tabletop Groupware (Fiebrink, CHI 2009)

Data-Driven Exploration of Musical Chord Sequences (Nichols, IUI 2009)  Problem: let people create and explore music by moving around in a reduced- dimensionality space  Genres, artists make for intuitive labels

 Why isn’t this easy?  Solution(s): Divergence-maximizing clustering, PCA Data-Driven Exploration of Musical Chord Sequences (Nichols, IUI 2009)

Future work  Make Songsmith freakin’ amazing, and port it to a bajillion platforms, and make an amazing community Web site, and build it into audio hosts… etc…  Other applications of machine learning in creativity support tools… Writing? Painting? Web Design? CAD? Music? Future work?

Songsmith Dan Morris, Ian Simon, Sumit Basu, and the MSR Advanced Development Team Microsoft Research research.microsoft.com/songsmith Live/Google: songsmith