DeepMIDI: Music Generation Arindam Bhattacharya, Jonathan Burge, Bryce Codell | MSiA 490-30 Deep Learning | Spring 2017 | Northwestern University Problem Statement Technical Approach Results Our most successful attempt at generating a jazzy bass line resulted from seeding with “Autumn Leaves,” a classic 1940’s piece that strongly exhibits typical characteristics of jazz harmony and music. We have visualized how strongly our generated music exhibits these characteristics below: There is constant demand for new musical content for a multitude of uses, ranging from artistic expression, to jingles for new TV shows, to elevator music. Objective: to generate original music content using deep learning. Audio files were converted from MIDI to text and fed into a text generation model, the output of which was converted back to MIDI. The primary metric of success was whether or not the music demonstrated the typical characteristics of jazz music. Pre- Process Fit Model 2 Post- Convert midi files to csv Compute derived features Subset features Reformat data into midi-csv structure Provide proper midi metadata Convert from csv back to midi format Vectorize text using “musical” vocabulary Train models LSTM, GRU Provide new ‘seeds’ Save generated data to csv 1 3 Most significant challenge: deriving the appropriate features and vocabulary necessary to generate meaningful output Derived features: note duration and note delay (start time of current note - start time of previous note) Alleviated several otherwise difficult-to-address constraints, e.g. generated start times needing to be in strictly ascending order Vocabulary: instead of generating all four note components with the standard text generation vocabulary (e.g. ‘1’, ‘2’, etc.), we made four distinct vocabularies, each corresponding the to unique numeric values of the four note components We fit four models in parallel, each corresponding to the four components of the notes being generated Experimented with a variety of diversity, dropout, and memory/window sizes to avoid common traps (e.g. getting stuck in a loop) Legendary jazz bassist Charles Mingus Dataset Our dataset was obtained from freemidi.org We converted these MIDI files to csv format prior to model fitting Below are samples of what our data looks like prior and after pre- processing Data cleaning involved several steps: Removing files of text that did not directly pertain to the notes being played Standardization of note format Subsetting specific channels from individual MIDI files We faced several data processing challenges: identifying a method for batch conversion of MIDI files into csv format, understanding the details of how MIDI files are converted into audio, generalizing our data processing to account for the significant variance in syntax and structure among MIDI files, and enforcing proper output format after converting model output back to MIDI format. Our training set consisted of bass lines from 8 jazz classics, all converted to MIDI by Mel Webb. Conclusion Overall, DeepMIDI showed promising potential for music generation. It demonstrated the ability to identify and integrate key features of jazz bass into its generated music (particularly those which are rhythmic in nature) However, it struggled a bit more with tonality, perhaps due to the small training data set and/or the inclusion of multiple key signatures Moderate diversity (0.6 - 0.8) produced the best sounding results One-hot encoding numeric values (effectively transforming them into categorical variables) has several major advantages, but also imposes limitations on the range of output prediction possibilities Additional limitations are inherent to the parallelized computation undertaken here - models ignore relationships between the four note components (e.g. shorter delays tend to be associated with shorter durations) Model Training Music Generation References and Related Work Freemidi.org - Free Midi Music Songs Download. Retrieved from https://freemidi.org. Keras/Theano Jazz Generation - https://github.com/jisungk/deepjazz LSTM Text Generation - https://github.com/fchollet/keras/blob/ master/examples/lstm_text_generation.py MidiCSV - Convert midi file to and from csv. Retrieved from http://www.fourmilab.ch/webtools/midicsv/ Our model consisted of 4 LSTM neural networks which each handled a single feature that together compose a note. The “note” and “velocity” models had fewer hidden nodes because their vocabularies were significantly smaller than those of “duration” and “delay.”