BPC: Art and Computation – Fall 2006 Digital media I: Audio Glenn Bresnahan Robert Putnam

BPC: Art and Computation – Fall Outline Part I (Glenn) –What is sound? –How do we hear? Part II (Robert) –Qualities of sound –Sound reproduction analog v. digital –Sound in VR

BPC: Art and Computation – Fall Waves revisited

BPC: Art and Computation – Fall Waves – (non)artistic rendering

BPC: Art and Computation – Fall Wave properties How might we describe waves?

BPC: Art and Computation – Fall Wave properties How might we describe waves? –Height –Time between waves –Speed of the wave

BPC: Art and Computation – Fall Shoals and tides

BPC: Art and Computation – Fall Tide tables

BPC: Art and Computation – Fall Cause of tides Gravity from moon and sun 365 days 27.3 days (29.5 days) New moon Full moon 1 day

BPC: Art and Computation – Fall Phases of the moon Moon phases: New moon Waxing crescent First quarter Waxing gibbous Full moon Waning gibbous Last quarter New moon

BPC: Art and Computation – Fall Moon phases

BPC: Art and Computation – Fall Sunrise and sunset

BPC: Art and Computation – Fall Sunrise and sunset solstice

BPC: Art and Computation – Fall Waves – sine waves Sine wave is the fundamental wave

BPC: Art and Computation – Fall Waves – properties Amplitude Wavelength (distance)

BPC: Art and Computation – Fall Waves in motion – properties Period (time for one cycle) Time 1 2 Frequency cycles per time interval

BPC: Art and Computation – Fall What is sound? Examples

BPC: Art and Computation – Fall What is sound – vibration Striking an object will cause it to vibrate The vibration is a sine wave Objects have a natural vibration frequency –Resonance frequency –Frequency depends on type of material, thickness, length/size, tension –May have multiple vibrating frequencies The pitch depends on the frequency Loudness (amplitude) depends on size of the object

BPC: Art and Computation – Fall What is sound – vibrations moves air string Energy (pluck) vibration Air pressure level

BPC: Art and Computation – Fall Properties of sound Pitch is perception of frequency Frequency is measured in cycles per second (cps) –Hertz (Hz) = cycles per second –The A above middle C is 440 Hz. –Humans hear appox ,000 Hz Sound travels at approx feet/second –Speed depends on pressure and temperature –Approx. 750 miles/hour –Approx. 1 mile every 4.8 seconds Perceived loudness depends on pressure level –Sound pressure is measured in (micro)pascals (20uPa) –Loudness is usually expressed in decibels (dB)

BPC: Art and Computation – Fall Real Waves

BPC: Art and Computation – Fall Properties of sound Real sounds are far more complex than simple sine waves –Objects produce vibrations at multiple frequencies –Sound waves interact with other objects Waves bounce (reflect) off surface –Reverberation/echo Wave are absorbed by materials –Sound waves interact with each other

BPC: Art and Computation – Fall Combinations of waves

BPC: Art and Computation – Fall Properties of sound – real sounds

BPC: Art and Computation – Fall Electrification of sound Microphones –Convert pressure levels into electrical signals (voltages) Guitar pickups –Converts string vibration to voltages The pickup contains a magnet and a coil The vibrating metal strings alter the magnetic field and induce a voltage in the coil Loud speakers convert an electrical signal back into air pressure

BPC: Art and Computation – Fall How do we hear? Sound waves move through the air from the sound source to the ear

BPC: Art and Computation – Fall Anatomy of the ear

BPC: Art and Computation – Fall Anatomy of the ear - outer Divided into three principal sections –Outer ear –Middle ear –Inner ear Outer ear –External ear, aka pinna –Ear canal –Outer ear funnels the ear have to the eardrum

BPC: Art and Computation – Fall Anatomy of the ear - middle Middle ear –Eardrum –Set of 3 ear bones the 3 bones are rigid –Act as a mechanical amplifier –The 3 rd bone, stapes, induces a vibration into the inner ear, i.e. the cochlea

BPC: Art and Computation – Fall Anatomy of the ear - inner Inner ear / cochlea Where the real work is done Cochlea is a spiral tube and filled with fluid Stapes causes a wave to pass through the fluid

BPC: Art and Computation – Fall Anatomy of the ear - inner Cochlea is a spiral tube lined with hair cells on a membrane (~15K HCs) Hairs vary in length and thickness along the tube Hairs resonate at different frequencies –High freq on near end, low at far end

BPC: Art and Computation – Fall Anatomy of the ear - inner

BPC: Art and Computation – Fall Anatomy of the ear - inner Hair cells are connected to the auditory nerve cells The vibrations excite the nerve cells and cause them to fire (electrical signal) A series of nerve cells pass the signal to brain

BPC: Art and Computation – Fall Binaural hearing - why two ears? Two ears, so we can identify locations of sounds –Time difference –Intensity difference –Sound color difference (caused by movement of sound around head and shoulders)

BPC: Art and Computation – Fall Sound localization – pinna front back Sound waves interact with the asymmetric Pinna The effect on the sound varies with the direction Up/down, back/front waves result in different sounds entering ear canal

BPC: Art and Computation – Fall Digital media I: part II Other qualities of sound: pitch, timbre, “noise”, envelope Sound reproduction: analog v. digital Sound in VR

BPC: Art and Computation – Fall What is pitch? Our perception of the highness or lowness of a tone. Closely related to frequency When frequency doubles, pitch rises by an “octave” Examples But, what happens when there’s more than one frequency in a sound?

BPC: Art and Computation – Fall Review: modes of vibration of a string Fundamental [e.g., 110 Hz] 2 nd harmonic [e.g., 220 Hz] 3 rd harmonic [e.g., 330 Hz] Examples

BPC: Art and Computation – Fall Timbre Sound color, or “timbre” is a quality of sound that derives from the particular combination of frequencies (a.k.a., “harmonics” or “partials”) in a tone. Two sounds can contain the same harmonics but sound very different because their individual harmonics are of different amplitudes. Examples

BPC: Art and Computation – Fall Timbre, continued Easy to demonstrate timbre with human voice Hum. Slowly open mouth. Hear how the sound color changes from “dark” to “bright” Example

BPC: Art and Computation – Fall Timbre, continued. Timbre changes as a wind instrument is played louder or softer. Example

BPC: Art and Computation – Fall Unpitched sounds Can use human voice to demonstrate another distinction: pitched versus unpitched sounds Make “s” sound No identifiable “pitch” Related to concept of “noise” Examples

BPC: Art and Computation – Fall Examples Pitched sounds –Birdsong –Flutes –Stringed instruments –Etc. Examples of unpitched sounds –Certain percussion instruments (cymbals, ratchets, etc.) –Wind, rain, footsteps in snow Listen now. What do you hear? Frequencies, amplitudes. Pitched, unpitched. External versus internal sources.

BPC: Art and Computation – Fall Time variation of sounds Most naturally occurring sounds are not static; i.e., they vary over time –Amplitude –Pitch –Timbre Examples

BPC: Art and Computation – Fall Sound recording technologies Analog Digital

BPC: Art and Computation – Fall Analog recording Analog: “device or system that represents changing values as continuously variable physical quantities.” Example: clock with hour, minute and second hands Question: what values are changing when we hear sound?

BPC: Art and Computation – Fall Analog recording technologies Phonautograph

BPC: Art and Computation – Fall Analog recording technologies Mechanical: Gramophone, LP record, etc.

BPC: Art and Computation – Fall Analog recording technologies Magnetic: Wire, tape recorder.

BPC: Art and Computation – Fall Analog recording technologies Optical: movie soundtrack.

BPC: Art and Computation – Fall Analog sound reproduction Amplification Loudspeaker demo

BPC: Art and Computation – Fall Digital recording Digital: “device or system that represents changing values as discontinuous, or ‘discrete,’ values.” Example: clock with number readout.

BPC: Art and Computation – Fall Digital recording With digital recording, we do not store a continuous record of the rise and fall of air pressure. We make measurements of the air pressure (or the voltage produced by a microphone) thousands of times per second and store these measurements as numbers.

BPC: Art and Computation – Fall Digital recording Analog: continuous waveform Digital: discrete values

BPC: Art and Computation – Fall Some buzzwords ADC: analog-to-digital converter DAC: digital-to-analog converter Sampling rate: samples/second Word size: how much storage for each sample Quantization: [see next slide]

BPC: Art and Computation – Fall Quantization Selecting sample value from finite set of numbers. 16 bits = 65,536 choices 20 bits = 1,048,576 choices Source:

BPC: Art and Computation – Fall CD audio 44,100 samples per second. 16-bit samples (65536 different possible values) Frequency range: 0-22KHz.

BPC: Art and Computation – Fall Some benefits of digital audio Easy to edit (visual interface) No noise with additional generations Flexible signal processing (no special hardware) Examples (reverb, pitch shift, noise reduction, etc.)

BPC: Art and Computation – Fall An aside: MP3 audio CD takes up a lot of space: 3 minute song = 44100*2*2*3*60 = B. MP3 compression results in a factor of 5-10 savings in storage, but lower fidelity (e.g., noisier).

BPC: Art and Computation – Fall Another aside: MIDI Musical Instrument Digital Interface A communications scheme for computers, synthesizers, sequencers, etc. Suited for popular music Stores Note-On, Note-Off, Velocity, etc. (i.e., not waveforms) Example

BPC: Art and Computation – Fall DAFFIE audio Soundserver –Plays sound files associated with objects –Mixes many simultaneous sounds –Internet telephony support

BPC: Art and Computation – Fall DAFFIE localization All sounds are assigned a location in virtual space. Typically, sounds are associated with visible objects, but “ambient” sound (e.g., wind, nature) is supported too. Direction and distance are indicated by variations in loudness among the loudspeakers – done automatically by soundserver.

BPC: Art and Computation – Fall Uses of sound in VR Communication (via telephony) Sound effects Music selections Ambient audio Live audio (via telephony) Previous projects have involved controlling synthesizers or musical instruments remotely.

BPC: Art and Computation – Fall Recording sounds for DAFFIE Field recording versus studio recording Record, record, record Remember that sounds are combined “in real time” by the soundserver (so no need to put everything in a single file). Stereo is good.

BPC: Art and Computation – Fall Recording sounds for DAFFIE, part II. Remember that you can change sounds in various ways –Change pitch/tempo –Use filters to change spectrum –Cut and paste

BPC: Art and Computation – Fall DAFFIE demo Demonstrate –Proximity triggering –(Variable) Distance attenuation –Sound localization