1. Copyright 2004 Ken Greenebaum Introduction to Interactive Sound Synthesis Lecture 7: Latency + Additive Synth Ken Greenebaum

2. Copyright 2004 Ken Greenebaum Assignments Picking up complexities/subtleties Picking up complexities/subtleties Math Math Data structures Data structures Algorithms Algorithms Aesthetics Aesthetics Opportunities for creativity Opportunities for creativity

3. Copyright 2004 Ken Greenebaum Assignment 3a: Partials synthesizer Sum of exponentially decaying sinusoids Sum of exponentially decaying sinusoids G(t) = Σ n Φ n e -δ n t cos(ω n t) G(t) = Σ n Φ n e -δ n t cos(ω n t) G(t) waveform over time G(t) waveform over time Φ n (phi) Initial Amplitude Φ n (phi) Initial Amplitude δ n (delta) Damping Constant δ n (delta) Damping Constant ω n (omega) Frequency of partial n ω n (omega) Frequency of partial n

4. Copyright 2004 Ken Greenebaum Assignment 3a: Partials synthesizer Name partials.c Name partials.c Input (stdin) Input (stdin) List of floating point, ASCII, parameters List of floating point, ASCII, parameters Lines of the form Lines of the form Φ 1 δ 1 ω 1 Φ 1 δ 1 ω 1 Φ 2 δ 2 ω 2 Φ 2 δ 2 ω 2 EOF EOF

5. Copyright 2004 Ken Greenebaum Assignment 3a: Partials synthesizer Output (PABLIO) Output (PABLIO) Sum of partials Sum of partials 48000Hz Frame Rate 48000Hz Frame Rate

6. Copyright 2004 Ken Greenebaum Assignment 3a: Partials synthesizer Challenges Challenges Synthesizing variable number of partials Synthesizing variable number of partials Adding partials with minimal clipping Adding partials with minimal clipping Halting program after energy dies out Halting program after energy dies out

7. Copyright 2004 Ken Greenebaum Assignment 3a: Partials synthesizer Hint: Synth variable number of partials Hint: Synth variable number of partials Parse input into data structure Parse input into data structure Traverse data structure for each sample Traverse data structure for each sample Hint: Adding partials with minimal clipping Hint: Adding partials with minimal clipping Halting program after energy dies out Halting program after energy dies out

8. Copyright 2004 Ken Greenebaum Assignment 3a: Partials synthesizer Hint: Adding partials with minimal clipping Hint: Adding partials with minimal clipping Scale each accordingly Scale each accordingly Beware: no partial is normalized Beware: no partial is normalized Partials loose energy quickly but at diff rates Partials loose energy quickly but at diff rates Most partials will start with < max power Most partials will start with < max power

9. Copyright 2004 Ken Greenebaum Assignment 3a: Partials synthesizer Hint: Halting program after energy dies out Hint: Halting program after energy dies out Partials will trend to zero over time Partials will trend to zero over time Program may be halted when output Program may be halted when output Is within epsilon of zero Is within epsilon of zero Ultimately < JND of silence! Ultimately < JND of silence! For an extended number of samples For an extended number of samples

10. Copyright 2004 Ken Greenebaum Assignment 3a: Partials synthesizer Extra Hint: GNUPlot is your friend Extra Hint: GNUPlot is your friend You may use GNUPlot as a CAD You may use GNUPlot as a CAD Will help you understand the dynamics Will help you understand the dynamics

11. Copyright 2004 Ken Greenebaum Assignment 3b: Collision Named collision.c Named collision.c Takes 4 fp command line arguments Takes 4 fp command line arguments Length (1-1000 units) Length (1-1000 units) Solid/Hollowness (0-1) (1=Solid) Solid/Hollowness (0-1) (1=Solid) Metal/Woodenness (0-1) (1=Pine) Metal/Woodenness (0-1) (1=Pine) How hard struck (0-1) (0=missed, 1=max) How hard struck (0-1) (0=missed, 1=max)

12. Copyright 2004 Ken Greenebaum Assignment 3b: collision Output Output A list of partials for 3a! A list of partials for 3a!

13. Copyright 2004 Ken Greenebaum Assignment 3b: Collision Challenges Challenges No single correct solution No single correct solution

14. Copyright 2004 Ken Greenebaum Assignment 3 Collision and Partial must work together Collision and Partial must work together Example: Example: collision 800 0.5 0.2 0.7 | partial collision 800 0.5 0.2 0.7 | partial

15. Copyright 2004 Ken Greenebaum Assignment 3 Deliverables Makefile that builds collision.exe, partial.exe Makefile that builds collision.exe, partial.exe Should pickup PABLIO.h’s,.lib from c:\cs245 Should pickup PABLIO.h’s,.lib from c:\cs245 collision.c collision.c partial.c partial.c Data files Data files Collision data:.col Collision data:.col Partial data:.par Partial data:.par

16. Copyright 2004 Ken Greenebaum Assignment 3 Deliverables Makefile that builds collision.exe, partial.exe Makefile that builds collision.exe, partial.exe Should pickup PABLIO.h’s,.lib from c:\cs245 Should pickup PABLIO.h’s,.lib from c:\cs245 collision.c collision.c partial.c partial.c Partial, collision data files to generate Partial, collision data files to generate Chime.XXX Middle C wind chime Chime.XXX Middle C wind chime Liberty.XXX Liberty Bell Liberty.XXX Liberty Bell Bass.XXX Bass drum Bass.XXX Bass drum SolidDesk.XXX Fist on solid desk SolidDesk.XXX Fist on solid desk HollowDesk.XXX First on hollow desk HollowDesk.XXX First on hollow desk

17. Copyright 2004 Ken Greenebaum Collisions Bang Bang Why does it make a sound? Why does it make a sound? Why does it make this sound? Why does it make this sound?

18. Copyright 2004 Ken Greenebaum Collisions Bang2 Bang2 Why does it make a different sound? Why does it make a different sound?

19. Copyright 2004 Ken Greenebaum Collisions Bang (excite) physical object Bang (excite) physical object It resonates (vibrates) It resonates (vibrates) Re-radiates energy Re-radiates energy As sound (by moving air) As sound (by moving air) As heat (by internal friction) As heat (by internal friction)

20. Copyright 2004 Ken Greenebaum Collisions Different objects sound different when struck Different objects sound different when struck Due to different characteristics Due to different characteristics Physical Physical Geometry Geometry Termination Termination Composition Composition Internal friction Internal friction Rigidity Rigidity

21. Copyright 2004 Ken Greenebaum Collisions Same object will sound different when struck with different intensity Same object will sound different when struck with different intensity Different vibration modes excited Different vibration modes excited

22. Copyright 2004 Ken Greenebaum Collisions Vibration modes understood for Vibration modes understood for Strings Strings Bars Bars Round membranes Round membranes Square membranes (arrows pt to nodal lines) Square membranes (arrows pt to nodal lines)

23. Copyright 2004 Ken Greenebaum Collisions Vibration modes Vibration modes 1 st 12 modes of an ideal membrane 1 st 12 modes of an ideal membrane

24. Copyright 2004 Ken Greenebaum Collisions Physics of some idealized objects understood Physics of some idealized objects understood String String Bar Bar Membrane Membrane Square Square Round Round Boils down to an equation/algorithm Boils down to an equation/algorithm

25. Copyright 2004 Ken Greenebaum Collisions Can model non-idealized objects Can model non-idealized objects Using FEA Using FEA (Finite Element Analysis) (Finite Element Analysis) Can be very accurate Can be very accurate Used in civil engineering, etc. Used in civil engineering, etc. Computationally expensive Computationally expensive

26. Copyright 2004 Ken Greenebaum Collisions Can model non-ideal objects Can model non-ideal objects Real objects are non-ideal Real objects are non-ideal Most sound apps don’t require accuracy Most sound apps don’t require accuracy Less expensive/accurate methods appropriate Less expensive/accurate methods appropriate But some do when modeling: But some do when modeling: Symphony hall Symphony hall Supersonic jet Supersonic jet Ramjet Ramjet

27. Copyright 2004 Ken Greenebaum Collision Synthesiser Synth techniques good for different sounds Synth techniques good for different sounds Exponentially damped sinusoids Exponentially damped sinusoids Model strikes/collisions Model strikes/collisions

28. Copyright 2004 Ken Greenebaum Collision Synthesiser Parameters of sine terms Parameters of sine terms Damping Damping controls ringing controls ringing Frequency Frequency correspond to modes/harmonics correspond to modes/harmonics Amplitude Amplitude Corresponds to relative strength of harmonics Corresponds to relative strength of harmonics

29. Copyright 2004 Ken Greenebaum Collision Synthesiser Parameters parameters to collision synth Parameters parameters to collision synth Length of bar Length of bar Wood-Metal rank Wood-Metal rank Solid-Hollow rank Solid-Hollow rank HARD to generalize! HARD to generalize!

30. Copyright 2004 Ken Greenebaum Collision Synthesiser Parameters parameters to collision synth Parameters parameters to collision synth Length of bar Length of bar Controls the freq of the fundamental Controls the freq of the fundamental Wood-Metal rank Wood-Metal rank Influences the damping factor Influences the damping factor Meta/Glass rings Meta/Glass rings Wood is highly damped Wood is highly damped Influences the harmonics Influences the harmonics Solid-Hollow rank Solid-Hollow rank Influences the damping Influences the damping Influences the harmonics Influences the harmonics

31. Copyright 2004 Ken Greenebaum Collision Synthesizer Sum of exponentially decaying sinusoids Sum of exponentially decaying sinusoids G(t) = Σ n Φ n e -δ n t cos(ω n t) G(t) = Σ n Φ n e -δ n t cos(ω n t) G(t) Collision waveform over time G(t) Collision waveform over time Φ n (phi) Initial Amplitude Φ n (phi) Initial Amplitude Mallet hardness/Impact strength Mallet hardness/Impact strength δ n (delta) Damping Constant δ n (delta) Damping Constant Material Material ω n (omega) Frequency of partial n ω n (omega) Frequency of partial n Size/Configuration Size/Configuration

32. Copyright 2004 Ken Greenebaum Collision Synthesizer ω n partials based on material struck ω n partials based on material struck Strings are harmonic Strings are harmonic ω n = n ω 1 ω n = n ω 1 Solid plates are inharmonic Solid plates are inharmonic Random frequency shifts in harmonic pattern Random frequency shifts in harmonic pattern Solid bars Solid bars ω n = (2n + 1) 2/9 ω n = (2n + 1) 2/9 Rectangular resonators Rectangular resonators ω (p.q.r) = c/2 sqrt(p 2 /l 2 + q 2 /w 2 + r 2 /h 2 ) ω (p.q.r) = c/2 sqrt(p 2 /l 2 + q 2 /w 2 + r 2 /h 2 ) c = velocity of sound c = velocity of sound l,w,h = length, width, height of box l,w,h = length, width, height of box

33. Copyright 2004 Ken Greenebaum Perceivable Auditory Latencies Latency: Latency: Time between human input and resulting computer output Time between human input and resulting computer output

34. Copyright 2004 Ken Greenebaum Perceivable Auditory Latencies Latency is a fact of life/physics Latency is a fact of life/physics Need to commute to work Need to commute to work Can’t just jump to the weekend Can’t just jump to the weekend Have to wait through this class to get to lunch Have to wait through this class to get to lunch

35. Copyright 2004 Ken Greenebaum Perceivable Auditory Latencies Commute Latency Strategies Commute Latency Strategies Leave for work earlier Leave for work earlier Work from home Work from home Listen to Books on tape in car Listen to Books on tape in car Might not eliminate latency but Might not eliminate latency but Make it useful Make it useful Tolerable Tolerable Not noticeable Not noticeable

36. Copyright 2004 Ken Greenebaum Perceivable Auditory Latencies Some latency can be beneficial Some latency can be beneficial Work may be performed while waiting Work may be performed while waiting Can get ready for the next activity Can get ready for the next activity The delay might be relaxing The delay might be relaxing

37. Copyright 2004 Ken Greenebaum Perceivable Auditory Latencies Keys: Keys: Understanding perception Understanding perception Designing systems to: Designing systems to: Make latency undetectable Make latency undetectable Mask latency Mask latency

38. Copyright 2004 Ken Greenebaum Perceivable Auditory Latencies Design tradeoffs (typically) Design tradeoffs (typically) More latency More latency More robustness More robustness Higher quality rendering Higher quality rendering Less latency Less latency More interactive More interactive

39. Copyright 2004 Ken Greenebaum Perceivable Auditory Latencies Human perception of Audio latency Human perception of Audio latency Measured long ago Measured long ago Research not commonly known Research not commonly known Figures difficult to come by Figures difficult to come by

40. Copyright 2004 Ken Greenebaum Perceivable Auditory Latencies Ken’s hypothesis: Ken’s hypothesis: Humans are edge detectors Humans are edge detectors

41. Copyright 2004 Ken Greenebaum Perceivable Auditory Latencies Interaural time difference Interaural time difference ≈10 μS (microseconds) ≈10 μS (microseconds) (CD sample time 22.7 μS) (CD sample time 22.7 μS) dt

42. Copyright 2004 Ken Greenebaum Perceivable Auditory Latencies Three edge phenomena: Three edge phenomena: Separation between clicks Separation between clicks Gap in continuous signal Gap in continuous signal Variation in duration of signal Variation in duration of signal Similar levels of acuity ≈ 2ms (milli seconds) Similar levels of acuity ≈ 2ms (milli seconds)

43. Copyright 2004 Ken Greenebaum Perceivable Auditory Latencies Separation between clicks Separation between clicks dt Clicks

44. Copyright 2004 Ken Greenebaum Perceivable Auditory Latencies Gap in continuous signal Gap in continuous signal Gaps dt

45. Copyright 2004 Ken Greenebaum Perceivable Auditory Latencies Variation in duration of signal Variation in duration of signal Duration dt

46. Copyright 2004 Ken Greenebaum Perceivable Auditory Latencies Precedence effect Precedence effect Two separate events perceived as one Two separate events perceived as one 5-50ms depending on stimulus (click easiest) 5-50ms depending on stimulus (click easiest) dt

47. Copyright 2004 Ken Greenebaum Optional Readings: Perry Cook : Real Sound Synthesis Perry Cook : Real Sound Synthesis Pgs 131-136 Pgs 131-136 Rossing & Fletcher : Principles of Vibration and Sound Rossing & Fletcher : Principles of Vibration and Sound Berg & Stork: The Physics of Sound Berg & Stork: The Physics of Sound

48. Copyright 2004 Ken Greenebaum Next class: Additive Synthesis Additive Synthesis Physical materials Physical materials Next assignment: Collision Synth Next assignment: Collision Synth