Multimedia – Digital Audio & Video

CSC1720 – Introduction to Internet2 Outline Audio / Video on the Web Audio / Video on the Web Basic Digital Audio Concepts Basic Digital Audio Concepts –Streaming Audio –Web Audio Formats Basic Digital Video Concepts Basic Digital Video Concepts –Video compression/decompression methods. –Video File Formats Other Multi-media Formats Other Multi-media Formats Summary Summary

CSC1720 – Introduction to Internet3 AV on the Web? It is part of the Internet. It is part of the Internet. –Radio Stations –Music, sound clips –Streaming Audio / Video –Video conferencing –Digital Cameras –Animation on the Web Shockwave, Flash Shockwave, Flash

CSC1720 – Introduction to Internet4 Basic Terms TermDefinition Bit Binary value 0 or 1 Kilobit (Kb) 1,000 bits (approx.) Byte 8 bits Kilobyte (KB) 1,000 bytes Megabyte (MB) 1,000,000 bytes Gigabyte (GB) 1,000,000,000 bytes Kbps Kilobits per second (1,000 bits in a second) KB/sec Kilobytes per second (1,000 bytes in a second)

CSC1720 – Introduction to Internet5 Analog vs. Digital Analog: A signal that has a continuous nature rather than a pulsed or discrete nature. Analog: A signal that has a continuous nature rather than a pulsed or discrete nature.signal –Note: Electrical or physical analogies, such as continuously varying voltages, frequencies, or phases, may be used as analog signals. Digital: A signal in which discrete steps are used to represent information. Digital: A signal in which discrete steps are used to represent information.signal

CSC1720 – Introduction to Internet6 Digital Signal Use Pulse Code Modulation (PCM) to represent an audio signal by digital data. Use Pulse Code Modulation (PCM) to represent an audio signal by digital data.

CSC1720 – Introduction to Internet7 ADC & DAC Figure 4.3 Conversion from Analog to Digital requires an Analog-to-Digital Converter Figure 4.4 Conversion from Digital to Analog requires a Digital-to-Analog Converter

CSC1720 – Introduction to Internet8 Basic Digital Audio Concepts Sampling rate Sampling rate –Number of sample taken of a signal in a given time (usually one second) Bit depth Bit depth –Describes the accuracy of the audio data Channels Channels –E.g. Stereo has two channels Bit rate Bit rate –Measured in kilobits per second (Kbps) is a function of the bit depth and sampling rate

CSC1720 – Introduction to Internet9 Sampling rate The more sample taken per second, the higher the accuracy. The more sample taken per second, the higher the accuracy. Typically measured in kilohertz (KHz). Typically measured in kilohertz (KHz). CD audio has 44,100 samples per second (44.1KHz). CD audio has 44,100 samples per second (44.1KHz). 8 KHz produces lower quality radio sound. 8 KHz produces lower quality radio sound. Standard sampling rates include “ 8KHz ”, “ KHz ”, … Standard sampling rates include “ 8KHz ”, “ KHz ”, … The high-end 96K is used in DVD, but is not applicable to the Web. The high-end 96K is used in DVD, but is not applicable to the Web.

CSC1720 – Introduction to Internet10 Sampling Rate demo

CSC1720 – Introduction to Internet11 Bit depth Also called “ sampling resolution ” or “ word length ”. Also called “ sampling resolution ” or “ word length ”. The more bits, the better is the quality of the audio (and a larger file of course). The more bits, the better is the quality of the audio (and a larger file of course). Common bit depths are 8-bit (telephone like), 16-bit (CD quality), and 20, 24, 32, 48-bit depths. Common bit depths are 8-bit (telephone like), 16-bit (CD quality), and 20, 24, 32, 48-bit depths. How many signal can a 8-bit and a 16-bit data represent? How many signal can a 8-bit and a 16-bit data represent? –  – 

CSC1720 – Introduction to Internet12 Quantization demo

CSC1720 – Introduction to Internet13 Channels Audio file can support one to six channels of audio formats. Audio file can support one to six channels of audio formats. Mono – one channel Mono – one channel Stereo – two channels Stereo – two channels Some others – three, four channels. Some others – three, four channels. Six channels – 5.1-channel surround sound. Six channels – 5.1-channel surround sound. More multi-channel formats announce in the coming years. More multi-channel formats announce in the coming years.

CSC1720 – Introduction to Internet14 Channel Examples

CSC1720 – Introduction to Internet15 Bit rate Audio files are measured in terms of bit rate which is measured in kilobits per second (Kbps). Audio files are measured in terms of bit rate which is measured in kilobits per second (Kbps). It can be calculated by dividing the file size by the time (in second) to play the audio clip. It can be calculated by dividing the file size by the time (in second) to play the audio clip. –E.g. 3Mb file play in 30 seconds –3000k / 30 = 100kbps. Quality at different compression rates Quality at different compression rates

CSC1720 – Introduction to Internet16 Optimizing for the Web Length of the audio clip Length of the audio clip –Keep the audio clip as short as possible. –E.g. just keep the most sweetest part of your greeting. Number of channels Number of channels –A mono audio file is halved the space of a stereo file. –Depends on your actual needs.

CSC1720 – Introduction to Internet17 Optimizing for the Web Bit depth Bit depth –Audio file on the Web are usually 8-bits. –Half the size of a 16-bit file. Sampling rate Sampling rate –Half the sampling rate will also halve the space needed. –Voice only audio file can be reduced to 8KHz. –22 KHz music clips are acceptable. Putting all things together: Mono, 8-bit, 22KHz, MP3 compression. Putting all things together: Mono, 8-bit, 22KHz, MP3 compression.

CSC1720 – Introduction to Internet18 Calculate audio size 8-bit mono: seconds x KHz 8-bit mono: seconds x KHz 16-bit mono: (seconds x KHz) x 2 16-bit mono: (seconds x KHz) x 2 8-bit stereo: (seconds x KHz) x 2 8-bit stereo: (seconds x KHz) x 2 16-bit stereo: ([seconds x KHz] x 2)x2 16-bit stereo: ([seconds x KHz] x 2)x2 E.g. the file size of 30 seconds of 16- bit, 11KHz mono audio: E.g. the file size of 30 seconds of 16- bit, 11KHz mono audio: –(30 x 11) x 2 = 660K.

CSC1720 – Introduction to Internet19 Streaming Audio What is it? What is it? –Play almost immediately after the request, continues playing the transferring data. Advantages: Advantages: –Address the problem of long download time. –Control distribution and protect copyright, because the user cannot get a copy of the file. Disadvantages: Disadvantages: –Sound quality may be be affected by low speed or unstable Internet connection. Reference Reference Reference

CSC1720 – Introduction to Internet20 What is Streaming?

CSC1720 – Introduction to Internet21 How does it work? Streaming audio Web browser Web server RealAudio Server Web browser request a RealAudio from the web server Packet are sent to a buffer on the receiving Computer, the RealPlayer will play the sound File when buffer full

CSC1720 – Introduction to Internet22 Quality Comparison

CSC1720 – Introduction to Internet23 Web Audio Formats WAV/AIFF (.wav/.aif/.aiff) WAV/AIFF (.wav/.aif/.aiff) –The Waveform Audio File format (.wav) was developed by Microsoft, supports arbitrary sampling rates and bit depths. –The Audio Interchange File format (.aif,.aiff) was developed for Macintosh platform. –They are less likely used on the Web, because people use “ mp3 ” or “ streaming ”.

CSC1720 – Introduction to Internet24 WAV/AIFF Try the Bell-lab synthesis link. Try the Bell-lab synthesis link.Bell-lab synthesisBell-lab synthesis Select the Audio format

CSC1720 – Introduction to Internet25 MP3 (.mp3) Able to maintain excellent sound quality at very small file sizes. Able to maintain excellent sound quality at very small file sizes. The compression reduces an audio file to one-tenth of its original size. The compression reduces an audio file to one-tenth of its original size. –E.g. 40MB file  3.5MB MP3 is actually MPEG-1 Layer-III MP3 is actually MPEG-1 Layer-III Good for distribution of HQ audio. Good for distribution of HQ audio. Demo: Demo:

CSC1720 – Introduction to Internet26 What is MP3 digital encoding? MP3 is actually the file extension for MPEG, audio layer 3. Layer 3 is one of three coding schemes (layer 1, layer 2, and layer 3) for the compression of audio signals. Layer 3 uses perceptual audio coding and psychoacoustic compression to remove all superfluous information. (More specifically, it removes the redundant and irrelevant parts of a sound signal--the stuff the human ear doesn't hear anyway). It also adds a MDCT (Modified Discrete Cosine Transform) that implements a filter bank, increasing the frequency resolution 18 times higher than that of layer 2. MP3 is actually the file extension for MPEG, audio layer 3. Layer 3 is one of three coding schemes (layer 1, layer 2, and layer 3) for the compression of audio signals. Layer 3 uses perceptual audio coding and psychoacoustic compression to remove all superfluous information. (More specifically, it removes the redundant and irrelevant parts of a sound signal--the stuff the human ear doesn't hear anyway). It also adds a MDCT (Modified Discrete Cosine Transform) that implements a filter bank, increasing the frequency resolution 18 times higher than that of layer 2.

CSC1720 – Introduction to Internet27 MP3 Players

CSC1720 – Introduction to Internet28 Apple QuickTime Audio (.mov) QuickTime is a well-known video format, but it can create audio-only movies. QuickTime is a well-known video format, but it can create audio-only movies. QuickTime is a container format, which is able to store still images, movie formats, … QuickTime is a container format, which is able to store still images, movie formats, … –Excellent compression, true streaming Netscape and IE have Plug-in now. Netscape and IE have Plug-in now. Quicktime : demo Quicktime : demodemo

CSC1720 – Introduction to Internet29 MIDI (.mid/.midi) MIDI stands for “ Musical Instrument Digital Interface ” which is developed for electronic musical instruments. MIDI stands for “ Musical Instrument Digital Interface ” which is developed for electronic musical instruments. MIDI files are very compact and very good for low-bandwidth delivery. MIDI files are very compact and very good for low-bandwidth delivery. Instruments are “ piano, drums, bass, orchestral strings, …” Instruments are “ piano, drums, bass, orchestral strings, …” It is very attractive for adding MIDI file to your website with very little download time. It is very attractive for adding MIDI file to your website with very little download time. Demo: Demo:

CSC1720 – Introduction to Internet30 MIDI

CSC1720 – Introduction to Internet31 RealMedia/RealAudio (.rm/.ra) RealAudio is a server-based streaming audio format. RealAudio is a server-based streaming audio format. The RealServer responses to the requests and delivers the streaming packets, including the bandwidth negotiation. The RealServer responses to the requests and delivers the streaming packets, including the bandwidth negotiation. A robust RealServer can support thousands of simultaneous listeners. A robust RealServer can support thousands of simultaneous listeners. Good for continuous-playing audio and live broadcasts to a large group of people. Good for continuous-playing audio and live broadcasts to a large group of people. Example: RTHK Radio Example: RTHK RadioRTHK RadioRTHK Radio

CSC1720 – Introduction to Internet32 RealMedia/RealAudio

CSC1720 – Introduction to Internet33 Windows Media (.wma/.asf) Windows Media is a streaming system. Windows Media is a streaming system. It wraps all media elements into a Active Streaming File (.asf). It wraps all media elements into a Active Streaming File (.asf). Audio may be saved as non-streaming Windows Media Audio format (.wma). Audio may be saved as non-streaming Windows Media Audio format (.wma). Good for continuous-play audio. Good for continuous-play audio. The encoder and player is Free, shipped with Windows OS. The encoder and player is Free, shipped with Windows OS.

CSC1720 – Introduction to Internet34 Windows Media

CSC1720 – Introduction to Internet35 Choosing an audio format Audio Needs Suggested formats Short voice greetings WAV, AIFF, MP3 News broadcasts Streaming solutions (RealAudio, Windows Media) Background music MIDI, WAV Music samples for some audience MP3 or QuickTime Radio-style or Live broadcasting RealMedia System Musical E-greeting card MIDI, WAV

CSC1720 – Introduction to Internet36 Add Audio to your Web Page A simple link A simple link – Play the song – Play the song – – Background Sound Background Sound – – Link to RealMedia Link to RealMedia – Link to the song – Link to the song –pnm://domainname.com/song.rm

CSC1720 – Introduction to Internet37 Break Time – 15 minutes

CSC1720 – Introduction to Internet38 Basic Digital Video Concepts Movie length Movie length Frame size Frame size Frame rate Frame rate Quality Quality Color bit depth Color bit depth Data rate (bit rate) Data rate (bit rate)

CSC1720 – Introduction to Internet39 Movie length File size is proportional to the movie length. File size is proportional to the movie length. Videos longer than 1 or 2 minutes cause long download times. Videos longer than 1 or 2 minutes cause long download times. If it is a long video, consider to use streaming video. If it is a long video, consider to use streaming video.

CSC1720 – Introduction to Internet40 Frame size “ Full-screen ” video is 640x480 pixels. “ Full-screen ” video is 640x480 pixels. The most common frame size for web video is 160x120 pixels. The most common frame size for web video is 160x120 pixels. Not recommend to use a frame size larger than 320x240. Not recommend to use a frame size larger than 320x240. The size depends on the CPU power and the Internet connection bandwidth. The size depends on the CPU power and the Internet connection bandwidth.

CSC1720 – Introduction to Internet41 Image and Video?

CSC1720 – Introduction to Internet42 Frame rate Frame rate is measured in number of frames per second (fps). Frame rate is measured in number of frames per second (fps). Standard TV-quality video uses 30 fps. Standard TV-quality video uses 30 fps. For the web, 15 or even 10 fps is more appropriate and produces fair smooth quality for the user. For the web, 15 or even 10 fps is more appropriate and produces fair smooth quality for the user. Commercial Internet Broadcasts are using 0.5, 0.25 frames per second. Commercial Internet Broadcasts are using 0.5, 0.25 frames per second.

CSC1720 – Introduction to Internet43 Quality Many video-editors allow you to set the overall quality of the video. Many video-editors allow you to set the overall quality of the video. The degree of compression controls the target quality. The degree of compression controls the target quality. The low or medium setting results a fairly high compression which is appropriate for web delivery. The low or medium setting results a fairly high compression which is appropriate for web delivery. Frame rate and quality are usually tradeoff in different applications. Frame rate and quality are usually tradeoff in different applications.

CSC1720 – Introduction to Internet44 Color bit depth The number of pixel colors in each frame affects the size of the video. The number of pixel colors in each frame affects the size of the video. The file size of the video will be greatly reduced by changing the number of colors from 24-bit to 8-bit. The file size of the video will be greatly reduced by changing the number of colors from 24-bit to 8-bit. It sacrifices the image quality of the video. It sacrifices the image quality of the video.

CSC1720 – Introduction to Internet45 Data rate (bit rate) This is the rate that the data must be transferred in order to ensure the video can play smoothly without interruption. This is the rate that the data must be transferred in order to ensure the video can play smoothly without interruption. It is measured in kilobytes per second (K/sec or Kbps). It is measured in kilobytes per second (K/sec or Kbps). It can be calculated by dividing the size of the file (in K) by the movie length (in seconds). It can be calculated by dividing the size of the file (in K) by the movie length (in seconds). –E.g. the video file size is 1.9MB  1900K –Play 40 seconds long, Data rate = 47.5K/sec Consider the Internet bandwidth! Consider the Internet bandwidth!

CSC1720 – Introduction to Internet46 Calculate space requirements of Video NTSC video (640 x 480 and fps) NTSC video (640 x 480 and fps) –Frame size = ([Pixel width x pixel height x bit depth]/8)/1024 –E.g. 200KB/Frame : 6.0 MB/sec –200KB x 30 fps = 6000KB/s, 6 MB/sec PAL video (768 x 576 and 25 fps) PAL video (768 x 576 and 25 fps) –E.g. 200KB/Frame : 5.0 MB/sec –200KB x 25 fps = 5000KB/s, 5 MB/sec

CSC1720 – Introduction to Internet47 Video CODECs CODEC is “ Compression/Decompression ” algorithms. CODEC is “ Compression/Decompression ” algorithms. The sound and frame images of a digital video must be compressed. The sound and frame images of a digital video must be compressed. –The vast amount of data Compressed in a number of ways Compressed in a number of ways –Lossless and Lossy compression –Spatial and Temporal compression

CSC1720 – Introduction to Internet48 Video Clip Demo reference reference reference

CSC1720 – Introduction to Internet49 Lossless and Lossy compression Lossless compression means no information is lost and the final file is the same as the original. Lossless compression means no information is lost and the final file is the same as the original. Most compression methods are lossy. Most compression methods are lossy. –Sacrifices some data from the file in order to achieve higher compression rates. –Use complicated algorithm to toss out some data that is not discernible to the human eye or ear.

CSC1720 – Introduction to Internet50 Spatial and Temporal compression Spatial (Intraframe) compression takes place on each individual frame of the video. Spatial (Intraframe) compression takes place on each individual frame of the video. Temporal (Interframe) compression applies on a series of frames, it takes advantages of areas of the image remain unchanged from frame to frame. Temporal (Interframe) compression applies on a series of frames, it takes advantages of areas of the image remain unchanged from frame to frame. –Relies on the key frames and delta frames. –A key frame is placed once every second. –E.g. 15 fps, a key frame once 15 frames.

CSC1720 – Introduction to Internet51 Delta frame

CSC1720 – Introduction to Internet52 Video File Formats QuickTime Movie (.mov) QuickTime Movie (.mov) –Introduced by Apple Computer in –First developed for Macintosh, now also supports the PCs. –Also supports streaming. How to create? How to create? –Most video editor, QuickTime Pro. How to play? How to play? –QuickTime plug-in or QuickTime player.

CSC1720 – Introduction to Internet53 Quicktime

CSC1720 – Introduction to Internet54 RealMedia (.rm) Industry standard streaming format. Industry standard streaming format. RealPlayer for playback. RealPlayer for playback. RealServer for serving streams. RealServer for serving streams. RealProducer for creating.rm files. RealProducer for creating.rm files. Good for Good for –Long-playing video or broadcast to many people. How to create? How to create? –RealSystem Producer How to play? How to play? –RealPlayer (Free), RealPlayer Plus (Commerical)

CSC1720 – Introduction to Internet55 Windows Media (.wmv/.asf) Created by Microsoft, closely integrated with Windows OS. Created by Microsoft, closely integrated with Windows OS. Support Windows Media Video (.wmv) and Advanced Streaming Format (.asf) and other formats (.avi,.mpeg, … ) Support Windows Media Video (.wmv) and Advanced Streaming Format (.asf) and other formats (.avi,.mpeg, … ) Also support streaming. Also support streaming. How to create? How to create? –Windows Media Encoder, Windows Media Author How to play? How to play? –Media Player in Windows OS

CSC1720 – Introduction to Internet56 AVI (.avi) Stands for Audio/Video Interleaved. Stands for Audio/Video Interleaved. Introduced by Microsoft in Introduced by Microsoft in In a AVI file, the audio and video information are interleaved every frame. In a AVI file, the audio and video information are interleaved every frame. Good for Good for –Short web clips, high-quality video How to create? How to create? –Most video editing tools. How to play? How to play? –Windows Media, QuickTime, etc.

CSC1720 – Introduction to Internet57 Video Clip Demo reference reference reference

CSC1720 – Introduction to Internet58 MPEG (.mpg/.mpeg) Created by Moving Picture Experts Group (MPEG). Created by Moving Picture Experts Group (MPEG). Supports 1) Video, 2) Audio, 3) Streaming. Supports 1) Video, 2) Audio, 3) Streaming. Extremely high compression rates with small quality degradation (lossy). Extremely high compression rates with small quality degradation (lossy). MPEG-1 : VHS quality MPEG-1 : VHS quality MPEG-2 : HQ standard for TV broadcast MPEG-2 : HQ standard for TV broadcast MPEG-4 : Very HQ for AV compression MPEG-4 : Very HQ for AV compression MPEG can be compressed by using three schemes: Layer-I, Layer-II, Layer-III. MPEG can be compressed by using three schemes: Layer-I, Layer-II, Layer-III.

CSC1720 – Introduction to Internet59 Official MPEG page reference reference reference

CSC1720 – Introduction to Internet60 MPEG Comparison MPEG-1MPEG-2MPEG-4 Standard since Default Video resolution (NTSC) 352 x x 480 Max. Audio Frequency range 48 KHz 96 KHz Max. audio Channel 288 Regular data rate 1380 kbit/s 6500 kbit/s 880 kbit/s Frames per sec (NTSC) Video Quality Satisfactory Very good Very Good Hardware requirement for encoding/decoding LowMediumHigh

CSC1720 – Introduction to Internet61 Flash & Shockwave Advantages: Advantages: –File sizes are small –Image quality is high –It uses streaming technology –It uses high-quality streaming audio –It is scriptable Disadvantages: Disadvantages: –A plug-in player is required –Expensive authoring software –Problems on printing their content

CSC1720 – Introduction to Internet62 Summary Discuss the basic digital audio/video terminology. Discuss the basic digital audio/video terminology. Introduction to different formats: WAV, MP3, QuickTime, RealMedia, Windows Media, AVI, MPEG. Introduction to different formats: WAV, MP3, QuickTime, RealMedia, Windows Media, AVI, MPEG. To deliver long-playing audio/video or live broadcasts, you should choose one of the streaming media. To deliver long-playing audio/video or live broadcasts, you should choose one of the streaming media. Flash and shockwave are popular and appropriate format for the Web. Flash and shockwave are popular and appropriate format for the Web.

CSC1720 – Introduction to Internet63 References Design Web Audio (J. Beggs, D. Thede), Oreilly. Design Web Audio (J. Beggs, D. Thede), Oreilly. E-Video (H. Peter Alesso), Addison-Wesley. E-Video (H. Peter Alesso), Addison-Wesley. Audio Video Knowledge Center Audio Video Knowledge Center Audio Video Knowledge Center Audio Video Knowledge Center Bible in MP3 format Bible in MP3 format Bible in MP3 format Bible in MP3 format MPEG MPEG MPEG Musiq Musiq Musiq Luke Video Luke Video Luke Video Luke Video The End. The End. Thank you for your patience! Thank you for your patience!