Live Music Performances over High-Speed IP Networks Stefan Karapetkov Director, Emerging Technologies TERENA Networking Conference Bruges, Belgium, May 20, 2008

Agenda Manhattan School of Music Audio-Video Networks Audio Technology Voice-specific Codec Functions Adjustments for Live Music Mode Video Technology Transmission Technology Live Music Mode Demo 2

MSM Testimonial 3

Audio-Video Networks Today Video Endpoints Conference Servers Call Control, Management & Scheduling Video Recording, Streaming & Content Management Security & NAT/FW Traversal IM/Presence and IP-PBX Integration

User Database Gatekeeper Terminal A Terminal B 1) H.225 SETUP 2) H.225 SETUP 6) H.245 CAPS, MS 8) H.245 CAPS, MS 5) H.245 CAPS, MS 4) H.225 CONNECT 3) H.225 CONNECT 7) H.245 CAPS, MS 9) H.245 OLC 10) H.245 OLC IP Network RTP/RTCP Stream H.323 Architecture

2) 302 Moved Temporarily 7) ACK 8) ACK 6) 200 OK 5) 200 OK 4) INVITE 1) INVITE 3) INVITE SIP Redirect Server User Database SIP Proxy Registrar IP Network SIP User Agent A SIP User Agent B RTP/RTCP Stream SIP Architecture

Audio and Video Compression 7 Concert SiteRemote Site

Advanced Audio Compression Technology Data Bit Rate Audio Fidelity G G.722 AMR-NB G.711 G Wideband Narrowband 4 kbps64 kbps128 kbps Siren 14 stereo Siren 22 stereo G.729A G.728 G.722.1C Super Wide

Siren TM 22 Stereo Codec Highlights 9 Siren TM 22 MP3 Optimized for low latency - 40ms Frequency band 22kHz Stereo High latency – 54-81ms Stereo Frequency band 18kHz Low complexity 15MIPSHigh complexity 100MIPS Low bit rate – max. 128kbps Optimized for storage - bit rates > 128kbps

Siren TM 22 on the Road to Standardization ITU-T G.719 full-band codec approved in May 2008 Based on Polycom Siren™22 and Ericsson’s advanced audio G.719 number for higher visibility ITU-T cited the strong and increasing demand for audio coding providing the full human auditory bandwidth Conferencing systems are increasingly used for more elaborate presentations, often including music and sound effects In today’s multimedia presentations, playback of audio and video from DVDs and PCs is becoming a common practice New Telepresence systems provide High Definition video and audio quality to the user, and require high-quality media delivery to create the immersive experience Extending the quality of remote meetings helps reduce travel which in turn reduces greenhouse gas emission and limits climate change. 10

Automatic Gain Control (AGC) Signal strength AGC adds 0dB AGC adds 3dB AGC adds 6dB Max. 12 feet from microphone Nominal is 2 feet from microphone

Automatic Gain Control (AGC) Activated by speech and music Ignores white noise, e.g. if a fan is working close, AGC will not ramp up the gain based on fan noise AGC destroys the natural dynamic range If the music is loud, AGC decreases the volume If the music is quiet, AGC increases the volume Therefore, AGC must be completely disabled in a codec 12

Automatic Noise Suppression (ANS) & Noise Fill 13 Signal White noise Signal Comfort Noise ANS Noise Fill

Acoustic Echo Cancellation (AEC) 14 Acoustic Coupling Hears echo AEC

Stereo Acoustic Echo Cancellation (AEC) 50-22,000 Hz operating range Adaptive filter length of 260ms This number is the max delay of the echo that we can compensate This is the room response – it includes many audio wave reflections No learning sequence needed Algorithm trains quickly on speech No need to send out white noise to train it Stereo echo canceller identifies multiple paths of the stereo loudspeakers Quickly adapts to microphones that are moved within two words of speech Moving the mike changes the echo path and the adaptive filter has to learn the new path. Echo comes back for short time (1-2 words); then canceller adjusts. 15

Stereo AEC in Live Music Mode (LMM) Standard AEC leads to audio artifacts, low notes can be cut Main complain from MSM is that sustained note (e.g. press sustain pedal on piano) cannot be heard all the way even if they are just 1dB over the noise floor AEC settings in LMM prevent very quiet musical sounds from being cut out Assumption that LMM is set in a quiet environment without background noise We changed the thresholds for signal detection to be more aggressive (low) 16

Installed Audio 17 Definition: rack-mounted systems that process all the audio in a conference room or large meeting room Microphones Speakers Video System DVD Telephony SoundStructure

Interworking: Installed Audio & Video Endpoints SoundStructure adds 8/12/16 additional inputs/outputs Digital connectivity with Polycom Video Endpoints Fully digital audio for better quality Bi-directional stereo between SoundStructure and HDX Full 22kHz stereo AEC compatible with Siren 22 audio codec Shared mute and volume control Auto-discovery between the devices – automatic configuration 18 SoundStructure HDX

Advanced Video Technology: High Definition 19 Quality Bandwidth 384kbps 512kbps1Mbps 6Mbps 480p 720p CIF SD HD 352x x x720

Advanced Video Technology: Camera Control Res 1280x720p 50/60FPS Aspect ratio 16:9 Pan +/- 100° Tilt +20° to -30° 12x optical zoom 20 FECC

Advanced Video Technology: Far End Camera Control (FECC) 21 FECC In H.323, FECC uses H.281 (binary data) over H.224 (frames) RFC 4573, MIME Type Registration for RTP Payload Format for H.224

Advanced Video Technology: Multiple Streams 22 ‘Live’ Stream ‘Presentation’ Stream ITU-T Recommendation H.239 RFC 4796, SDP Content Attribute RFC 4574, The SDP Label Attribute RFC 3388, Grouping of Media Lines in SDP RFC 4582, Binary Flow Control Protocol (BFCP) RFC 4583, SDP Format for BFCP Streams draft-even-xcon-pnc-01, Role Mgmt & Multiple Streams

Transmission Technology 23 SIP Domain H.323 Domain

Audio Precedence in Codec Negotiation 24 Audio Video High priority BandwidthStandard SettingLMM Setting > 1024Siren22 Stereo Siren22 Stereo 96Siren22 Stereo Siren22 Stereo 96Siren22 Stereo Siren22 Stereo 96Siren22 Stereo Siren14 Stereo 48

Keeping Quality Up in Transmission 25 Video Error Concealment (PVEC) IP Network Lost Packet Recovery (LPR) Video AudioVideo AudioVideo

LPR Definitions LPR is a new method of error concealment for packet based networks that is based upon Forward Error Correction (FEC) LPR constantly adjust the video bit rate to reduce the amount of loss in a packet based network 26

Lost Packet Recovery (LPR) 27 Video Encoder Encryption RTP Sender LPR Packetizer LPR Recovery Packet Generator LPR DBA Mode Decision RTCP LPR Recovery RTP Reordering Buffer LPR Regeneration Decryption Video Decoder

LPR DBA Example % 74% 58% 72% 64% 58% 70% 77% Down SpeedingUp Speeding X ms Full Bandwidth Packet loss 25%, FEC on No packet loss, FEC off … Y ms Bit rate drop 26% Packet loss 15% Bit rate drop 16% Packet loss 4%, FEC on Bit rate drop 5% Bit rate increase e.g. 10% Down Speeding No packet loss, FEC off … 58% X ms

Technology Summary Flexible Networking – H.323 and SIP Advanced Audio Technologies Audio Compression Automatic Gain Control (AGC) Automatic Noise Suppression (ANS) and Noise Fill Stereo Acoustic Echo Cancellation (AEC) Advanced Video Technology High Definition Camera Control Multiple Streams Advanced Transmission Technology Lost Packet Recovery (LPR) 29

Live Music Mode Demo