2. SYNTHESIS BY EXAMPLE Creation of sound synthesis algorithms using evolutionary methods By: Ricardo A. García rago@media.mit.edu MIT - Media Lab Machine Listening Group © 2001

3. MIT - Media Lab Machine Listening Group Synthesis by Example Ricardo A. Garcia © 2001 INTRODUCTION zHow is that sound coming out of your soundcard being generated? zIs that a real violin? zWhat kind of controls do we have over it? pitch, length, other performance parameters sound attributes: brightness, harmonic content, etc The questions: zHow are those sound synthesis algorithms conceived? zCan this design process be automated?

4. MIT - Media Lab Machine Listening Group Synthesis by Example Ricardo A. Garcia © 2001 In a nutshell... zUser supplies a set of examples (inputs/target output) zThe system ”reverse engineers" the sound and returns a sound synthesis technique that is capable of rendering a sound "similar" to the target sound! ? Inputs Example Output TARGET

5. MIT - Media Lab Machine Listening Group Synthesis by Example Ricardo A. Garcia © 2001 Talk outline zIntroduction zSound Synthesizers zDesign of Sound Synthesis Techniques (SST) zProposed approach for design zGenetic Programming zAutomatic Generation of Sound Synthesizers (AGeSS) system zExamples

6. MIT - Media Lab Machine Listening Group Synthesis by Example Ricardo A. Garcia © 2001 What is a sound synthesizer? zAny method or device used to produce sound zSound Synthesis Technique (SST): Algorithm that produces sound samples, usually implemented in a digital computer xInputs: –internal parameters (init time) –control signals (time varying) xOutput: –sound samples (Waveform) zStructured audio, symbolic representation Model analysis

7. MIT - Media Lab Machine Listening Group Synthesis by Example Ricardo A. Garcia © 2001 Inside a SST yFunctional form of the algorithm Hard to do! Prefabricated formulas and algorithms xY=A*sin(x+B)+C yInternal parameters “Easy to do” Parameter estimation Analysis tools xA=0.4, B=0.1, C=-0.3

8. MIT - Media Lab Machine Listening Group Synthesis by Example Ricardo A. Garcia © 2001 Traditional design zPick a Functional Form zDo parameter estimation to fit your target … LPC FM Additive Parameter Estimation TARGETInputs USE IT Human selects Functional Form

9. MIT - Media Lab Machine Listening Group Synthesis by Example Ricardo A. Garcia © 2001 Proposed approach z“Suggest” functional form (suggest “new” SST) zDo parameter estimation zCompare with target zSuggest new SST Parameter Estimation TARGETInputs Suggest SST functional form + Initial Parameters Functional Elements Compare with Target

10. MIT - Media Lab Machine Listening Group Synthesis by Example Ricardo A. Garcia © 2001 Representation of SSTs zPseudocode zFormula zTopology graph assign 3 to x sum x and 0.4 return x

11. MIT - Media Lab Machine Listening Group Synthesis by Example Ricardo A. Garcia © 2001 SST space zSST space: all the possible valid combinations of a set of functional elements and their connections! HUGE! zHypothesis: ”given a set of inputs, target and fitness metrics, it is possible to find the functional-form and internal parameters of a SST capable of synthesizing a sound “close” to the target”.

12. MIT - Media Lab Machine Listening Group Synthesis by Example Ricardo A. Garcia © 2001 Design of SST as a search zSpecs: xInputs xoutput (TARGET) xFitness metric zSearch: find a point (SST, Functional form + internal parameters) that fulfills the specs EXAMPLE

13. MIT - Media Lab Machine Listening Group Synthesis by Example Ricardo A. Garcia © 2001 The Genetic Programming Loop HUMAN Vs MACHINE

14. MIT - Media Lab Machine Listening Group Synthesis by Example Ricardo A. Garcia © 2001 Comparing sounds --fitness function-- Analytical distance metrics: MSE of complex spectrogram (magnitude and phase)

15. MIT - Media Lab Machine Listening Group Synthesis by Example Ricardo A. Garcia © 2001 Perceptual Fitness Function zA perceptual distance metrics xPerceptually relevant components: full error xAbove masking threshold: error up to M.T. xBelow masking threshold: no error

16. MIT - Media Lab Machine Listening Group Synthesis by Example Ricardo A. Garcia © 2001 Development mapping from expression tree to topology graph

17. MIT - Media Lab Machine Listening Group Synthesis by Example Ricardo A. Garcia © 2001 AGeSS system zAutomatic Generation of Sound Synthesizers zMatlab scripts and C++ programs

18. MIT - Media Lab Machine Listening Group Synthesis by Example Ricardo A. Garcia © 2001 Examples zA synthetic tone (des35comp2.avi) (des35comp2.avi) zA piano note (des44comp2.avi) (des44comp2.avi) zFM synthesizer (chow727.avi) (chow727.avi)

19. MIT - Media Lab Machine Listening Group Synthesis by Example Ricardo A. Garcia © 2001 FM synthesis example zTarget formula zOutput (generation 220+)

20. MIT - Media Lab Machine Listening Group Synthesis by Example Ricardo A. Garcia © 2001 FM synthesis example (scale) Training example (A880, B988) Output (best of generation 850) Scale Scale with best of generation 850

21. MIT - Media Lab Machine Listening Group Synthesis by Example Ricardo A. Garcia © 2001 For more information… Ricardo A. Garcia Email: rago@media.mit.edu Web: http://www.media.mit.edu/~rago