2. BPC: Art and Computation – Fall 2006 Digital media I: Audio Glenn Bresnahan Robert Putnam glenn@bu.eduglenn@bu.edu putnam@bu.edu

3. BPC: Art and Computation – Fall 20062 Outline Part I (Glenn) –What is sound? –How do we hear? Part II (Robert) –Qualities of sound –Sound reproduction analog v. digital –Sound in VR

4. BPC: Art and Computation – Fall 20063 Waves revisited

5. BPC: Art and Computation – Fall 20064 Waves – (non)artistic rendering

6. BPC: Art and Computation – Fall 20065 Wave properties How might we describe waves?

7. BPC: Art and Computation – Fall 20066 Wave properties How might we describe waves? –Height –Time between waves –Speed of the wave

8. BPC: Art and Computation – Fall 20067 Shoals and tides

9. BPC: Art and Computation – Fall 20068 Tide tables

10. BPC: Art and Computation – Fall 20069 Cause of tides Gravity from moon and sun 365 days 27.3 days (29.5 days) New moon Full moon 1 day

11. BPC: Art and Computation – Fall 200610 Phases of the moon Moon phases: New moon Waxing crescent First quarter Waxing gibbous Full moon Waning gibbous Last quarter New moon

12. BPC: Art and Computation – Fall 200611 Moon phases

13. BPC: Art and Computation – Fall 200612 Sunrise and sunset

14. BPC: Art and Computation – Fall 200613 Sunrise and sunset solstice

15. BPC: Art and Computation – Fall 200614 Waves – sine waves Sine wave is the fundamental wave

16. BPC: Art and Computation – Fall 200615 Waves – properties Amplitude Wavelength (distance)

17. BPC: Art and Computation – Fall 200616 Waves in motion – properties Period (time for one cycle) Time 1 2 Frequency cycles per time interval

18. BPC: Art and Computation – Fall 200617 What is sound? Examples

19. BPC: Art and Computation – Fall 200618 What is sound – vibration Striking an object will cause it to vibrate The vibration is a sine wave Objects have a natural vibration frequency –Resonance frequency –Frequency depends on type of material, thickness, length/size, tension –May have multiple vibrating frequencies The pitch depends on the frequency Loudness (amplitude) depends on size of the object

20. BPC: Art and Computation – Fall 200619 What is sound – vibrations moves air string Energy (pluck) vibration Air pressure level

21. BPC: Art and Computation – Fall 200620 Properties of sound Pitch is perception of frequency Frequency is measured in cycles per second (cps) –Hertz (Hz) = cycles per second –The A above middle C is 440 Hz. –Humans hear appox. 20-20,000 Hz Sound travels at approx. 1100 feet/second –Speed depends on pressure and temperature –Approx. 750 miles/hour –Approx. 1 mile every 4.8 seconds Perceived loudness depends on pressure level –Sound pressure is measured in (micro)pascals (20uPa) –Loudness is usually expressed in decibels (dB)

22. BPC: Art and Computation – Fall 200621 Real Waves

23. BPC: Art and Computation – Fall 200622 Properties of sound Real sounds are far more complex than simple sine waves –Objects produce vibrations at multiple frequencies –Sound waves interact with other objects Waves bounce (reflect) off surface –Reverberation/echo Wave are absorbed by materials –Sound waves interact with each other

24. BPC: Art and Computation – Fall 200623 Combinations of waves

25. BPC: Art and Computation – Fall 200624 Properties of sound – real sounds

26. BPC: Art and Computation – Fall 200625 Electrification of sound Microphones –Convert pressure levels into electrical signals (voltages) Guitar pickups –Converts string vibration to voltages The pickup contains a magnet and a coil The vibrating metal strings alter the magnetic field and induce a voltage in the coil Loud speakers convert an electrical signal back into air pressure

27. BPC: Art and Computation – Fall 200626 How do we hear? Sound waves move through the air from the sound source to the ear

28. BPC: Art and Computation – Fall 200627 Anatomy of the ear

29. BPC: Art and Computation – Fall 200628 Anatomy of the ear - outer Divided into three principal sections –Outer ear –Middle ear –Inner ear Outer ear –External ear, aka pinna –Ear canal –Outer ear funnels the ear have to the eardrum

30. BPC: Art and Computation – Fall 200629 Anatomy of the ear - middle Middle ear –Eardrum –Set of 3 ear bones the 3 bones are rigid –Act as a mechanical amplifier –The 3 rd bone, stapes, induces a vibration into the inner ear, i.e. the cochlea

31. BPC: Art and Computation – Fall 200630 Anatomy of the ear - inner Inner ear / cochlea Where the real work is done Cochlea is a spiral tube and filled with fluid Stapes causes a wave to pass through the fluid

32. BPC: Art and Computation – Fall 200631 Anatomy of the ear - inner Cochlea is a spiral tube lined with hair cells on a membrane (~15K HCs) Hairs vary in length and thickness along the tube Hairs resonate at different frequencies –High freq on near end, low at far end

33. BPC: Art and Computation – Fall 200632 Anatomy of the ear - inner

34. BPC: Art and Computation – Fall 200633 Anatomy of the ear - inner Hair cells are connected to the auditory nerve cells The vibrations excite the nerve cells and cause them to fire (electrical signal) A series of nerve cells pass the signal to brain

35. BPC: Art and Computation – Fall 200634 Binaural hearing - why two ears? Two ears, so we can identify locations of sounds –Time difference –Intensity difference –Sound color difference (caused by movement of sound around head and shoulders)

36. BPC: Art and Computation – Fall 200635 Sound localization – pinna front back Sound waves interact with the asymmetric Pinna The effect on the sound varies with the direction Up/down, back/front waves result in different sounds entering ear canal

37. BPC: Art and Computation – Fall 200636 Digital media I: part II Other qualities of sound: pitch, timbre, “noise”, envelope Sound reproduction: analog v. digital Sound in VR

38. BPC: Art and Computation – Fall 200637 What is pitch? Our perception of the highness or lowness of a tone. Closely related to frequency When frequency doubles, pitch rises by an “octave” Examples But, what happens when there’s more than one frequency in a sound?

39. BPC: Art and Computation – Fall 200638 Review: modes of vibration of a string Fundamental [e.g., 110 Hz] 2 nd harmonic [e.g., 220 Hz] 3 rd harmonic [e.g., 330 Hz] Examples

40. BPC: Art and Computation – Fall 200639 Timbre Sound color, or “timbre” is a quality of sound that derives from the particular combination of frequencies (a.k.a., “harmonics” or “partials”) in a tone. Two sounds can contain the same harmonics but sound very different because their individual harmonics are of different amplitudes. Examples

41. BPC: Art and Computation – Fall 200640 Timbre, continued Easy to demonstrate timbre with human voice Hum. Slowly open mouth. Hear how the sound color changes from “dark” to “bright” Example

42. BPC: Art and Computation – Fall 200641 Timbre, continued. Timbre changes as a wind instrument is played louder or softer. Example

43. BPC: Art and Computation – Fall 200642 Unpitched sounds Can use human voice to demonstrate another distinction: pitched versus unpitched sounds Make “s” sound No identifiable “pitch” Related to concept of “noise” Examples

44. BPC: Art and Computation – Fall 200643 Examples Pitched sounds –Birdsong –Flutes –Stringed instruments –Etc. Examples of unpitched sounds –Certain percussion instruments (cymbals, ratchets, etc.) –Wind, rain, footsteps in snow Listen now. What do you hear? Frequencies, amplitudes. Pitched, unpitched. External versus internal sources.

45. BPC: Art and Computation – Fall 200644 Time variation of sounds Most naturally occurring sounds are not static; i.e., they vary over time –Amplitude –Pitch –Timbre Examples

46. BPC: Art and Computation – Fall 200645 Sound recording technologies Analog Digital

47. BPC: Art and Computation – Fall 200646 Analog recording Analog: “device or system that represents changing values as continuously variable physical quantities.” Example: clock with hour, minute and second hands Question: what values are changing when we hear sound?

48. BPC: Art and Computation – Fall 200647 Analog recording technologies Phonautograph

49. BPC: Art and Computation – Fall 200648 Analog recording technologies Mechanical: Gramophone, LP record, etc.

50. BPC: Art and Computation – Fall 200649 Analog recording technologies Magnetic: Wire, tape recorder.

51. BPC: Art and Computation – Fall 200650 Analog recording technologies Optical: movie soundtrack.

52. BPC: Art and Computation – Fall 200651 Analog sound reproduction Amplification Loudspeaker demo

53. BPC: Art and Computation – Fall 200652 Digital recording Digital: “device or system that represents changing values as discontinuous, or ‘discrete,’ values.” Example: clock with number readout.

54. BPC: Art and Computation – Fall 200653 Digital recording With digital recording, we do not store a continuous record of the rise and fall of air pressure. We make measurements of the air pressure (or the voltage produced by a microphone) thousands of times per second and store these measurements as numbers.

55. BPC: Art and Computation – Fall 200654 Digital recording Analog: continuous waveform Digital: discrete values

56. BPC: Art and Computation – Fall 200655 Some buzzwords ADC: analog-to-digital converter DAC: digital-to-analog converter Sampling rate: samples/second Word size: how much storage for each sample Quantization: [see next slide]

57. BPC: Art and Computation – Fall 200656 Quantization Selecting sample value from finite set of numbers. 16 bits = 65,536 choices 20 bits = 1,048,576 choices Source: http://advisor.matrasi-tls.fr/digital_sampling_index.html

58. BPC: Art and Computation – Fall 200657 CD audio 44,100 samples per second. 16-bit samples (65536 different possible values) Frequency range: 0-22KHz.

59. BPC: Art and Computation – Fall 200658 Some benefits of digital audio Easy to edit (visual interface) No noise with additional generations Flexible signal processing (no special hardware) Examples (reverb, pitch shift, noise reduction, etc.)

60. BPC: Art and Computation – Fall 200659 An aside: MP3 audio CD takes up a lot of space: 3 minute song = 44100*2*2*3*60 = 31752000 B. MP3 compression results in a factor of 5-10 savings in storage, but lower fidelity (e.g., noisier).

61. BPC: Art and Computation – Fall 200660 Another aside: MIDI Musical Instrument Digital Interface A communications scheme for computers, synthesizers, sequencers, etc. Suited for popular music Stores Note-On, Note-Off, Velocity, etc. (i.e., not waveforms) Example

62. BPC: Art and Computation – Fall 200661 DAFFIE audio Soundserver –Plays sound files associated with objects –Mixes many simultaneous sounds –Internet telephony support

63. BPC: Art and Computation – Fall 200662 DAFFIE localization All sounds are assigned a location in virtual space. Typically, sounds are associated with visible objects, but “ambient” sound (e.g., wind, nature) is supported too. Direction and distance are indicated by variations in loudness among the loudspeakers – done automatically by soundserver.

64. BPC: Art and Computation – Fall 200663 Uses of sound in VR Communication (via telephony) Sound effects Music selections Ambient audio Live audio (via telephony) Previous projects have involved controlling synthesizers or musical instruments remotely.

65. BPC: Art and Computation – Fall 200664 Recording sounds for DAFFIE Field recording versus studio recording Record, record, record Remember that sounds are combined “in real time” by the soundserver (so no need to put everything in a single file). Stereo is good.

66. BPC: Art and Computation – Fall 200665 Recording sounds for DAFFIE, part II. Remember that you can change sounds in various ways –Change pitch/tempo –Use filters to change spectrum –Cut and paste

67. BPC: Art and Computation – Fall 200666 DAFFIE demo Demonstrate –Proximity triggering –(Variable) Distance attenuation –Sound localization