Business and Computing Deanery Multimedia Week 9 Sound

Multimedia 2 Last week Lecture Media dictionary Bitmaps Your interface Supported session Using imported artwork Look at navigation

Multimedia 3 This week Lecture Your interface and Flash navigation Sound Chapter 9 Digital Multimedia, 2nd edition The nature of sound Digitization Compression File formats Working with sound Supported session Working with sound Look at navigation

Multimedia 4 Interactivity and Flash An event triggers every action In Flash no action occurs unless something triggers it There are two types of events in Flash Movie events User events All actions in Flash are a form of ActionScript

Multimedia 5 Flash and interaction Behaviors Ready made scripts You can use behaviors to control movie clips, video, and sound files. ActionScript A script can consist of a single command, such as instructing a SWF file to stop playing, Or a series of commands and statements, such as first evaluating a condition and then performing an action

Multimedia 6 Flash and navigation Frame to frame navigation Behaviors on (release) { //Movieclip GotoAndStop Behavior this.gotoAndStop("50"); //End Behavior Load another movie ActionScript

Multimedia 7 Events, handlers and actions Whenever a user clicks the mouse or presses a key, an event is generated. In a SWF file, frames, buttons, movie clips, and text fields all generate events to which your user can respond Actions this.gotoAndStop("50"); this.gotoAndPlay("50");

Multimedia 8 Sound in multimedia Consider why you would use sound, will it add to your multimedia production? List how it could be used in your multimedia production What controls would you use? Why?

The Nature of Sound Conversion of energy into vibrations in the air (or some other elastic medium) Most sound sources vibrate in complex ways leading to sounds with components at several different frequencies Frequency spectrum – relative amplitudes of the frequency components Range of human hearing: roughly 20Hz– 20kHz, falling off with age 275–276 Chapman N and Chapman J (2004).

Waveforms Sounds change over time e.g. musical note has attack and decay, speech changes constantly Frequency spectrum alters as sound changes Waveform is a plot of amplitude against time Provides a graphical view of characteristics of a changing sound Can identify syllables of speech, rhythm of music, quiet and loud passages, etc 276–280 Chapman N and Chapman J (2004).

Multimedia 11 Amplitude Amplitude is perceived as loudness High amplitudeLower amplitude Click on the amplitudes to hear the difference.

Multimedia 12 Frequency Frequency is perceived as pitch. High frequencyLow frequency Click on the frequencies to hear the difference.

Multimedia 13 Sampling and Quantization Why is the sampling rate important? Why is the quantisation level important? 36 Sampling Quantisation Chapman N and Chapman J (2004).

Multimedia 14 Sampling Theorem If the highest frequency component of a signal is at fh the signal can be properly reconstructed if it has been sampled at a frequency > 2fh Nyquist rate Undersamping leads to aliasing Sound distortion, image 'jaggies' or Moiré patterns, jerky or retrograde motion 40–41 Chapman N and Chapman J (2004).

Digitization – Sampling Captures amplitude values at regular intervals Sampling rate is expressed in kHz (1000 per second) Sampling Theorem implies minimum rate of 40kHz to reproduce sound up to limit of hearing CD: 44.1kHz Sub-multiples often used for low bandwidth – e.g kHz for Internet audio DAT: 48kHz (Hence mixing sounds from CD and DAT will require some resampling, best avoided) 281–282 Chapman N and Chapman J (2004).

Multimedia 16 Sampling standards Telephone Quality kHz, 8-bit depth, Mono (10kb/sec) Radio Quality , 8-bit depth, Mono (21 kb/sec) CD Quality kHz, 16-bit depth, Stereo (172 kb/sec)

Digitization – Quantization 16 bits, quantization levels, CD quality 8 bits: audible quantization noise, can only use if some distortion is acceptable, e.g. voice communication Dithering – introduce small amount of random noise before sampling Noise causes samples to alternate rapidly between quantization levels, effectively smoothing sharp transitions 283–285 Chapman N and Chapman J (2004).

Undersampling & Dithering 283–284 Chapman N and Chapman J (2004).

Multimedia 19 Stereo Stereo sound is composed of TWO very similar sound waves. Subtle differences in location and movement Must have a separate sound wave per channel (twice the size file)

Multimedia 20 Mono Uses only one channel Good way to keep file sizes small As a general rule, use mono where possible.

Data Size Sampling rate r is the number of samples per second Sample size s bits Each second of digitized audio requires rs/8 bytes CD quality: r = 44100, s = 16, hence each second requires just over 86 kbytes (k=1024), each minute roughly 5Mbytes (mono) 287 Chapman N and Chapman J (2004).

Clipping If recording level is set too high, signal amplitude will exceed maximum that can be recorded, leading to unpleasant distortion But if level is set too low, dynamic range will be restricted 287–288 Chapman N and Chapman J (2004).

Sound Editing Timeline divided into tracks Sound on each track displayed as a waveform 'Scrub' over part of a track e.g. to find pauses Cut and paste, drag and drop May combine many tracks from different recordings (mix-down) 289 Chapman N and Chapman J (2004).

Effects and Filters Noise gate Low pass and high pass filters Notch filter De-esser Click repairer Reverb Graphic equalizer Envelope Shaping Pitch alteration and time stretching etc 290–295 Chapman N and Chapman J (2004).

Compression In general, lossy methods required because of complex and unpredictable nature of audio data CD quality, stereo, 3-minute song requires over 25 Mbytes Data rate exceeds bandwidth of dial-up Internet connection Difference in the way we perceive sound and image means different approach from image compression is needed 295 Chapman N and Chapman J (2004).

Perceptually-Based Compression Identify and discard data that doesn't affect the perception of the signal Needs a psycho-acoustical model, since ear and brain do not respond to sound waves in a simple way Threshold of hearing – sounds too quiet to hear Masking – sound obscured by some other sound 298–299 Chapman N and Chapman J (2004).

MP3 MPEG Audio, Layer 3 Three layers of audio compression in MPEG-1 (MPEG-2 essentially identical) Layer 1...Layer 3, encoding proces increases in complexity, data rate for same quality decreases e.g. Same quality 192kbps at Layer 1, 128kbps at Layer 2, 64kbps at Layer 3 10:1 compression ratio at high quality Variable bit rate coding (VBR) 300–301 Chapman N and Chapman J (2004).

AAC Advanced Audio Coding Defined in MPEG-2 standard, extended and incorporated into MPEG-4 Not backward compatible with earlier standards Higher compression ratios and lower bit rates than MP3 Subjectively better quality than MP3 at the same bit rate 301 Chapman N and Chapman J (2004).

Audio Formats Platform-specific file formats AIFF, WAV, AU Multimedia formats used as 'container formats' for sound compressed with different codecs QuickTime, Windows Media, RealAudio MP3 has its own file format, but MP3 data can be included as audio tracks in QuickTime movies and SWFs 302 Chapman N and Chapman J (2004).

MIDI Musical Instruments Digital Interface Instructions about how to produce music, which can be interpreted by suitable hardware and/or software cf. vector graphics as drawing instructions Standard protocol for communicating between electronic instruments (synthesizers, samplers, drum machines) Allows instruments to be controlled by hardware or software sequencers 303–304 Chapman N and Chapman J (2004).

MIDI and Computers MIDI interface allows computer to send MIDI data to instruments Store MIDI sequences in files, exchange them between computers, incorporate into multimedia Computer can synthesize sounds on a sound card, or play back samples from disk in response to MIDI instructions Computer becomes primitive musical instrument (quality of sound inferior to dedicated instruments) 304 Chapman N and Chapman J (2004).

Multimedia 32 Editing sound Use a piece of Sound Editing Software Cool Edit 2000 Sound manipulation program open, create or record new sound files Edit existing sounds Mix audio or musical parts, touch them up, and add effects Adobe Audition

Multimedia 33 Sound and Flash Sounds that continuously, independent of the Timeline Or you can synchronize animation to a sound track. You can add sounds to buttons to make them more interactive Macromedia, Flash MX 2004

Multimedia 34 Event sounds and Stream sounds An Event sound must download completely before it begins playing, and it continues playing until explicitly stopped Stream sounds begin playing as soon as enough data for the first few frames has been downloaded; stream sounds are synchronized to the Timeline for playing on a website. Macromedia, Flash MX 2004

Multimedia 35 Importing sound You can import the following sound file formats into Flash: WAV (Windows only) AIFF (Macintosh only) MP3 (Windows or Macintosh) Select File > Import > Import to Library To add a sound to a document from the library, you assign the sound to a layer and set options in the Sound controls in the Property inspector It is recommended that you place each sound on a separate layer. To define the starting point of a sound or to control the volume of the sound as it plays, you use the sound- editing controls in the Property inspector Macromedia, Flash MX 2004

Multimedia 36 Controlling sound You can control sound playback using sound Behaviors or ActionScript The most common sound-related task in Flash is starting and stopping sounds at keyframes in synchronization with animation You can use Flash to compress sound when it is published Macromedia, Flash MX 2004

Multimedia 37 Sync options Event The entire sound must download first Can be triggered more than once Will play all the way through Start Cannot have multiple instances Stop Stops Event or Start sound Stream Breaks the sound up into frame-sized pieces which means it does not all need to be downloaded before the Movie can play

Multimedia 38 Sound strategy Re-use sound from the library It is only stored once Loop shorter sounds in place of importing longer sounds Compress as much as possible without loss of quality Use separate sound-editing program to simplify and remove extraneous noise Kay M. (2003)

Multimedia 39 A question Scenario: You have been asked by Hope library to add a sound track to a flash movie on using the library Discuss and critically compare two methods of incorporating sound within your Flash movie The MP3 sound track that the library have given you is poor quality, discuss the reasons that may have caused this during the digitisation process

Multimedia 40 Activity Play a CD Copy a portion of a track Experiment with sound effects Add fades Export 30 secs as an mp3 Make a media dictionary entry Add to a Flash movie Insert controls using Behaviours

Multimedia 41 For Next Week Accessibility Publishing and distribution Independent study Attend the supported session to get to know Flash Create accessible Flash Content Publishing

Multimedia 42 References Chapman N and Chapman J (2004). Digital Multimedia, Second Edition. London. Wiley Flash Help and How Do I Kay M. (2003) The Web Wizard's Guide to Flash, Pearson Education, USA