2. Nov 30, Fall 2006IAT 4101 Audio Sound Audio synthesis

3. Nov 30, Fall 2006IAT 4102 Audio Perception  Sound: Pressure waves in frequencies between 50Herz - 22,000Herz  Lower frequencies more felt by the whole body than heard  Sounds can be perceived as coming from a location –Not terribly accurate –Cone of confusion

4. Nov 30, Fall 2006IAT 4103  Cone of confusion –Cone-shaped zones in front of and behind head  3D Audio cues: –Interaural Time Difference –Interaural Intensity Difference –Pinnae filtering –Body filtering 3D Audio Perception

5. Nov 30, Fall 2006IAT 4104 3D Audio Perception  Goal for 3D sound is “Spatialization”  The sense that the –Sound originates outside your head –Sound has a direction  Interaural Time Difference –The more extremely left or right, the greater the difference –Time difference < 5ms

6. Nov 30, Fall 2006IAT 4105 3D Audio Perception  Interaural Intensity Difference –Head absorbs and reflects sound energy –The first ear to get sound gets loudest sound –“Head Shadow”  Cone of confusion: –Time difference too small to detect –Intensity is similar in both ears

7. Nov 30, Fall 2006IAT 4106 Pinnae Filtering  Outer ear (Pinna) shape filters sound based on its direction  Childhood learning trains brain to associate filtering effects with direction  Unique per person  Record directional white noise –Microphone in ear canal –Sounds from speakers located about head

8. Nov 30, Fall 2006IAT 4107 Pinnae Filtering  A “Generic” Pinna can be simulated  Record directional white noise received by dummy head  Body filtering –Reflection and absorption –Included in Pinna model

9. Nov 30, Fall 2006IAT 4108 Head-Related Transfer Function  HRTF is the general term  Transformation of “real” sound to spatialized sound  Best delivered by earphones

10. Nov 30, Fall 2006IAT 4109 Environmental Effects  Sound exists in an environment –Bounces off objects –Is absorbed by objects  Simple effects –Reverb: Simulate the environmental echo Echo is the attenuated signal Gives a richer room-like feeling Larger room has longer time delay

11. Nov 30, Fall 2006IAT 41010 Audio signals  Nyquist limit: –Must sample signal at least twice as frequently as highest reproducible frequency –Audio: 44.1KHz (CD) –22KHz –11KHz (Analog AM Radio) –8KHz (Telephone)

12. Nov 30, Fall 2006IAT 41011 Audio - Digital Implications  44,100 Hz –44,100 Samples/sec –16-bit samples –Stereo –172KBytes/sec  Specialized hardware - Sound card

13. Nov 30, Fall 2006IAT 41012 Reproduction  Sampling –Record sounds by whatever means  Synthesis –Analog Synthesis –FM Synthesis –Wavetable Synthesis

14. Nov 30, Fall 2006IAT 41013 Control  MIDI - Musical Instrument Digital Interface  Developed to control music synthesizers –Details of synthesis are controlled by synthesizer  MIDI data –Sets synthesis parameters –Sets music sequence

15. Nov 30, Fall 2006IAT 41014 Synthesis  Analog Synthesis –Simple sum of frequencies –Select from a palette of source frequencies –Sum of frequencies is filtered  FM –One frequency is controlled by another  Wavetable –Digitize audio signals

16. Nov 30, Fall 2006IAT 41015 Analog Synthesis  Fourier’s observation –Any signal can be created as the sum of sine waves –Square wave: Infinite sum –f + 2f + 4f…  Synthesizer: –Collection of oscillators

17. Nov 30, Fall 2006IAT 41016 Frequency Domain  A perfectly periodic signal plotted in the frequency domain (Time domain plot)

18. Nov 30, Fall 2006IAT 41017 Spectrum  Spectrum represents the set of frequencies present in the signal

19. Nov 30, Fall 2006IAT 41018 Filters  Eliminate part of the signal by removing certain frequencies  Analog filters don’t have these “square” response shapes  Band pass –Bandwidth

20. Nov 30, Fall 2006IAT 41019 FM Synthesis  Modulate the frequency of a wave  Carrier frequency is modulated by Modulator signal

21. Nov 30, Fall 2006IAT 41020 FM Synthesis for Synthesizers  The greater the Modulator amplitude, the greater the Carrier frequency variation –Higher Carrier bandwidth  DX: Carrier and Modulator are “musically- tuned frequency” –Depends on the note you are playing –Controls the harmonic content of a note

22. Nov 30, Fall 2006IAT 41021 Wavetable Synthesis  Collect a sample of the real sound  Issues: –Reduce memory load by looping sample –Shift pitch instead of sampling each individual note –Apply interpolation techniques to make pitch shifting work right

23. Nov 30, Fall 2006IAT 41022 Audio Path Raw Sound (Sample, FM, etc) Tuned MIDI Note Sequence Resampling Envelope Loudness Control Mixing/ Combination Reverb, Environment. Spatialization

24. Nov 30, Fall 2006IAT 41023 Wavetable Synthesis Example  Leyanda (Guitar)  Leyanda (CDShaw)

25. Nov 30, Fall 2006IAT 41024 Interactive Sound  Goal –Want to enhance the interactive experience –Give the user a sense of presence –Add to the emotional content of the game –Make it more fun

26. Nov 30, Fall 2006IAT 41025 Interactive Sound  Music  Sound effects –Noises –Commentary - Sports  Narrative  Conversations

27. Nov 30, Fall 2006IAT 41026 Interactive Problems  Regular music composition has –Beginning –Middle –End  Interactive user control makes this difficult –Some genres have this structure

28. Nov 30, Fall 2006IAT 41027 Interactive Music  Game genres with order –Sports –Racing –Fighting  Semi-Ordered –Puzzle –Adventure

29. Nov 30, Fall 2006IAT 41028 Music Genres  The Infinite Loop –Theme and variation plays forever –Pomp & Circumstance –Diablo  Problems: –30 second piece repeated over 6 hours! –720 repetitions! –Diablo example: 12 Repetitions/hour

30. Nov 30, Fall 2006IAT 41029 Repetition Solutions  Make the Dominant theme hard to find –No catchy theme! –Create a variety of textures –Make only transitions stand out  Where repetition is small –Don’t repeat musical phrases

31. Nov 30, Fall 2006IAT 41030 Music Strategies  Play Win or Lose music –Music must be long enough to be meaningful –Music may be so long that the game situation changes before completion –Very short music makes little sense  Interrupt current music –Sounds jarring

32. Nov 30, Fall 2006IAT 41031 Modules  Modular chunks –Each segment of the game plays independently of others –Some thematic relation –Disjointed

33. Nov 30, Fall 2006IAT 41032 Music Strategies  Compose many themes in parallel  Switch between themes  Connect modular components together

34. Nov 30, Fall 2006IAT 41033 Analogy: Parallel Trains  N trains of music running in parallel  Each train serves an emotional purpose –Train A: Calm –Train B: Rising Excitement –Train C: Climactic moments –Train D: Falling Excitement  Generally, Train A would be most commonly played

35. Nov 30, Fall 2006IAT 41034 Within Each Musical Train  Each “Car” contains a few bars of music  Switch between trains when a “Car” is complete –Don’t switch in the middle of a “Car”  Simple version: –Each musical phrase ends on last bar of “Car”  Complex: –Notes at end are carried over to next

36. Nov 30, Fall 2006IAT 41035 Parallel Trains Bars 1-8Bars 9-16Bars 17-24Bars 25-32Bars 1-8Bars 9-16Bars 17-24Bars 25-32Bars 1-8Bars 9-16Bars 17-24Bars 25-32Bars 1-8Bars 9-16Bars 17-24Bars 25-32

37. Nov 30, Fall 2006IAT 41036 Parallel Trains: Shuffle Cars  Shuffle cars –Instead of playing cars in order  Problem: Random cars sound like random radio tuning  Must determine –Appropriate car pairings –Reasonable paths

38. Nov 30, Fall 2006IAT 41037 Repetition In Trains  Use repeated phrases carefully  Maybe use a statistical tool to analyze paths –Bayesian nets  Endings: –Use transitions as an opportunity to “End” –Use next Car to “Begin” new series

39. Nov 30, Fall 2006IAT 41038 Composing a Train  Create a piece with all layers –Piece can probably survive a layer or two removed –Variation = piece with layer removed  Be careful with prominent instruments –Fallback: Use instruments with similar acoustical properties Piano, Organ, Woodwinds No Trumpets, Drums or Screaming guitar!