Media Services in CINEMA Presented by Kundan Singh Joint work with Henning Schulzrinne, Jonathan Lennox, Xiaotao Wu, Wenyu Jiang, Sankaran Narayanan, Ajay Nambi and Ali Khwaja Computer Science Department, Columbia University April 25, 2003

2 Purpose What are media services? What is CINEMA? What services are provided by CINEMA? Conferencing Streaming media Unified messaging Interactive voice response How can we use CINEMA? The discussion on services and specific CINEMA implementation components are intermixed. N N Not relevant or not implemented On-going work Yes, Implemented O Y

3 IP telephony services (PSTN) Call routing services: pre-call, one party speed dial call forwarding “follow me” call filtering/blocking (in/out) do not disturb distinctive ringing Call handling features autoanswer Multi-party features call waiting call transfer (blind, consultative) conference call call park call pickup music on hold call monitoring O IP-telephony can’t win by just providing what PSTN has

4 IP telephony services (Internet) Presence-enabled calls place call only if callee is available Presence-enabled conferencing call conference participants when all are online and not busy IM conference alerts receive IM when someone joins a conference Unified messaging receive , IM alert for new voic s O

5 Where the services reside? Common gateway interface (CGI) Call processing language (CPL) SIP servlet Make call when boss is online … Enter your authentication PIN for billing… B2BUA Double ringing sound when boss calls… Endpoint Forward to office phone during day, and home phone during evening… Proxy/registrar Endpoint Language for End System Services (LESS) Use finger for locating user… O

6 Program Call routing “Bricks” in media applications SIPSAP RSVPRTCP RTP Media G.711 MPEG RTSP Signaling Quality of service Media transport Internet Telephony Internet Radio/TV Messaging and Presence Interactive voice response Unified messaging Video conferencing Physical layer Link layer Network (IPv4, IPv6) Transport (TCP, UDP) Application layer Voice XML DTMFMixing Speech/ text N SDP

7 … Audio Packet Transfer Digitization (e.g., sampling at 8kHz, 16 bits per sample, i.e, 128 kb/s or 320 bytes per 20 ms) Real-time compression/encoding (e.g., G.729A at 8 kb/s) Transport to remote IP address and port number over UDP (Why not TCP?) Processing on receiver side is the reverse N

8 Unreliable UDP a)Packet loss b)Out-of-order (very rarely) c)Jitter (delay variation) timeline Sender Receiver (a) (b) Problems with UDP N

Sender Receiver Receive Buffer Receive buffer: to absorb jitter Tradeoff in buffer size Adaptive delay adjustment Sequence number: to detect packet loss; Just ignore the loss! Y

10 Encoded Audio RTP Header UDP header IP header msg sendto(…, msg, …) recvfrom(…, msg, …) Sequence number Optional contributors’ list (CSrc) Source identifier (SSrc) Timestamp (proportional to sampling time) Payload typeCCMV PX Real-time Transport Protocol Y RTP: media transport RTCP: QoS feedback

11 Streaming media Messages RTSP vs HTTP UDP, Stateful, Control, Timed Applications Recording, Playback, Live media, Conference Web browser Media player file Web server HTTP Web browser Media player meta file HTTP RTSP RTP Media server Y

12 Columbia rtspd Existing Playback/recording of  - law (Sun.au files) and raw Unicast address Multi-threaded; Portable (Unix/win32) Works with Apple QuickTime client UDP + TCP IPv4 + IPv6 Planned Multicast MPEG video Mp3, wav audio Media over TCP Live media Y

13   Mixer Transcoder  -law G.729  -law RTP-based Conference N

14 What is the IP address of Alice’s host? What audio/video encoding can it support? bob’s host Alice’s host REGISTER as alice=> Call alice Why do we need signaling? Session Initiation Protocol -like address: DNS Proxy, redirect Y home.com

15 AliceBob INVITE I can support  -law and G.729 Send me audio at :6780 OK; I can support  -law Send me audio at : To port 6780 To port 8000 RTP Session description Y

16 Columbia sipc Y

17 Emergency service Calling E911, location Assistance Audio/video, remote-help Notification Earthquake, fire O

18 Presence/event notification PA registrar Presence server office.com SUBSCRIBE NOTIFY REGISTER PUA PUA + PA Y

19 Columbia IM and presence Y

20 Endpoint call control Language for End System Services (LESS) for endpoint service creation Direct user interaction, direct media control Handle converged information, e.g., call, presence, Example: when buddy is online, make a call <less name="online_call" require="generic presence ui"> <alert sound=“ring.au" text="Calling …" /> O

21 Network call control SIP-CGI CPL SIP servlets Priority.pl SIP_FROM SIP_TO stdin CGI-PROXY-REQUEST stdout SIP proxy Urgent Low-priority Voic Phone Y if (defined $ENV{SIP_FROM} && $ENV{SIP_FROM} =~ { foreach $reg (get_regs()) { print "CGI-PROXY-REQUEST $reg SIP/2.0\n"; print "Priority: urgent\n\n"; }

22 Call transfer REFER Blind/ consultation/ attended active call REFER C Referred-By: B INVITE C Referred-By: B BYE A A B C active call O

23 B2BUA and third-party call control Back-to-back UA Incoming call triggers outgoing call Services Calling card Anonymizer INVITE A B C SIP OK (SDP1) ACK INVITE (SDP1) OK (SDP2) ACK INVITE (SDP2) OK ACK O

24 Voic Problems in PSTN Design alternatives Issues Redirect after 10s vmail.pl SIP_FROM SIP_TO stdin CGI-PROXY-REQUEST stdout If no response accept after 15s Endpoint redirects Voic acts like a phone Proxy controls Y

25 Columbia sipum Goals Universal access Scalability Provider independent Why SIP and RTSP? Reuse existing infrastructure and tools (1) INVITE INVITE OK CANCEL (3) OK (2) SETUP (4) RTP (5) BYE rtspd sipum Y

26 Columbia sipum Retrieval Web interface rtsp://server/alice /inbox/1677.au, sip:alice press 1 to listen… Configuration Folders Options Y

27 VoiceXML Telephone PSTN Voice gateway Web server Service logic (CGI, servlet, JSP) Voice and telephony functions VoiceXML browser Internet user VXML HTML Internet IVR platform Voice and telephony functions (ASR, TTS, DTMF) Service logic (application specific) sipvxml Gateway Y

28 VoiceXML contd. Enter your Id: Your ID, please. Telephony, speech synthesis or audio output, user input and grammar, program flow, variable and properties, error handling, … Y

29 DTMF Telephone SIP/PSTN gateway PSTN Internet Internet user D D Audio SIP INFO RTP/ RFC2833 Y

30 Columbia sipvxml Unified messaging access by phone Event notification and scheduling Audio volume control for conference Advanced conference control Telephone SIP/PSTN gateway PSTN sipvxml SIP phone Web server Internet .tcl Media server rtspd TTS, ASR, DTMF, XML, HTTP, RTSP Y

31 + phone Login formatting Listen, reply, delete, compose, forward Navigation -next, previous, jump formatting SIP based Text-to-speech VoiceXML browser servlet JSP DB procmail to IM Call Interne t TTS important mails SIP Interne t SIP HTTP by phone to phone Y Inbox

32 IM + voice call Who can initiate? authentication, billing, … Feedback to voice user to IM user initial IM greeting Talk-spurt detection Speech recognition IM Call TTS ASR SIP MESSAGE SIP INVITE RTP Y

33 Unified messaging The gaps among different media (audio, video, text), devices (PC, phone) and means of communications ( , SIP, IM) disappear for messaging O

34 Notification Calendar Events Conferences Web server Calendar.cgi “at 6:00pm” Schedule from a browser SIP call IM O

35 Phone announcement server Input Range:93970?? List: A, B, C Example Announcement Emergency Issues Voic Failure detection PAS TTS Destinations SIP Text or audio... Y

36 RTSP + TTS + ASR Media server TTS ASR SETUP rtsp://server/tts.cgi ?text=How+are+you. PLAY RTP SETUP rtsp://server/asr.cgi RECORD RTP SET_PARAMETER Text=I am fine, thank you. O

37 Audio conference models (non-multicast) A B C D  B A+C+D A B C A B+C+D A+B+D C D D A+B+C  A B C D     Topology star full-mesh ad-hoc AdvantagesHeterogeneous simple clients No central point of failure DisadvantagesExternal server with high bandwidth link Complex endpoints Complex signaling Typically only three party conferences N

38 Centralized conferencing Conference as URL On the fly conferences Basic task: join/leave Dial in, Refer dial in Dial out, Refer dial out REFER INVITE REFER INVITE server O

39 Columbia sipconf G.711, GSM, DVI, G.722 mixing (decode-mix-encode) Video replication; IM; text; VNC screen sharing D D D E E E A B C G711 DVI GSM Linear G711 DVI G711 Playout delay Periodic timer M=A+B+C Mixed linear M - A=B+C M - B M - C Receive thread Send thread Y

40 Columbia sipconf Y

41 Conference control Access control Floor control Notification State Participant: active, departed, booted, failed dialog state, floor state Media sending: received-by-all, muted receiving: receiving-all, anchor-only active departed O

42 Conference + VoiceXML sipvxml Caller sipconf 1. INVITE sipvxml 2. Call accepted 3. Enter your four digit PIN 4. Entered Authenticate user, 4683=>Alice 6. Enter the conference identifier 7. Entered 2-3-# 8. Permission to join, 23=>meet 9. REFER 10.Terminate the old call 11.INVITE Call transfer vs bridged mode Y

43 New conference applications sipvxml Caller 1. INVITE sipvxml 2. Menu 1. Vol Check 2. Mic Check 3. User enters 2 4. User speaks out a voice sample 5. Voice sample is analyzed 7. User adjusts the vol level. The ease & flexibility of sipvxml enables us to build custom telephonic applications to suit our needs. e.g., Volume Check Application sipconf 8. User now joins conference. 6. SipVXML: Vol level too high/low/… Y

44 Conferencing Automatic volume adjustments Automatic load balancing Delay adjustments Conference recording Local or RTSP Ysipc SIP323 SIP/PSTN Recording in a media server

45 Ncast MPEG distribution INVITE Y

46 My owner’s SIP address is Help!!! (invoke sipc to call INVITE Y

47 Summary of media services Streaming media Playback, recording, live-media Unified messaging Voic , video/IM integration,... Conferencing Mixing, load-balancing Interactive voice response Tele-banking, VoiceXML Y

48 CINEMA Columbia InterNet Extensible Multimedia Architecture CINEMA Distributed component architecture Libraries and APIs Management and control Application and proxy servers User-agent (client) applications Y

49 Columbia SIP user agent: sipc Media Audio, video, text, white board Screen sharing Shared web browsing Advanced Presence, instant messaging Programmable call handling Conference control (on-going) Emergency notification and handling (on-going) Device control Language for End System Services (on-going) Y

50 Columbia SIP servers (CINEMA) Internal Telephone Extn: 7040 SIP/PSTN Gateway Department PBX Web based configuration Web server Telephone switch SQL database sipd: Proxy, redirect, Registrar server NetMeeting H.323 rtspd: media server sipum: Unified messaging Quicktime RTSP clients RTSP 713x Single machine SNMP (Network Management) sipconf: Conference server siph323: SIP-H.323 translator Local/long distance PSTN Video encoder Y

51 Libraries and APIs sipdsip323sipconfsipumsipvxmlrtspd CINEMA Libraries libNT Win32 stub libcine Utilities parsing IPv6 libsip Basic SIP library libsip++ SIP UA library libmixer RTP audio mixer libdict Hash table libdb++ mySQL intf RTSP media server SIP proxy server SIP/H.323 gateway SIP/RTP conferencing SIP/RTSP unified messaging SIP/VoiceXML browser Xerces-C OpenH323 MySQL PWLib Resparse librtsp RTSP client librtp RTP library libsnmp SIP MIB ViaVoice/Flite Xerces-C Server Applications Parsing, SIP, SDP, RTP, mySQL interface, SNMP interface, Portability stubs, etc. CPL, CGI, Servlet Y

52 Status of CINEMA services Features/servicesStatus SIP proxy, redirect, registration serverProduct SIP Presence agent; programmable call routingBeta Dial-in conference with audio, video, screen sharing, instant messaging, recording, load balancing, volume or delay adjustment, and IVR Beta Dial-out or refer dial-in conferencing, conference controlOn-going Voice mail with retrieval, deletion and IVRBeta Media streaming server for basic audio record and playBeta SIP/VXML browser for VoiceXML 1.0 with basic functionalityAlpha Call transfer, back-to-back user agentOn-going Phone announcement server, IM/call converter, -by-phone and -to-phone, SIP anonymizer Demo Calendar event notification to IM, phone and On-going Web-based configuration and controlAlpha Media server with text-to-speech, speech-recognition, video, live mediaOn-going

53 What’s next? from multimedia communications to collaboration Synchronous communications Conferencing, IM Asynchronous communications Voic s, message board, file sharing Ubiquitous computing i-button, ID-card Service creation User friendly, end-point/network

Media Services in CINEMA BACKUP SLIDES

55 DO SIP/2.0 ….. turn lamp on serial port lamp X10 device SIP beyond telephony SUBSCRIBE NOTIFY DO INVITE

56 PSTN interworking Translating audio (PCMU/PCMA) Translating signaling (PRI/T1,ISUP) Overlap signaling Advanced features in SIP are lost in PSTN Translating identifiers (phone number) Determining transition points Telephone network SIP/PSTN gateway SIP serverIP endpoint Telephone subscriber

57 PSTN to IP Gateway knows the SIP server ENUM DNS => e164.arpa => Suitable for relatively “static” contacts

58 IP to PSTN Static mapping xxxx ITGW information is dynamic: Overlapping networks Multiple providers Load balancing TRIP Route advertisement Can be implemented in outbound proxy Suitable for current hierarchical network at 4¢/min at 1¢/min free

59 SIP-CGI Programming language independent Maintains state via an opaque token For SIP proxies and endpoints: call routing controlling forking call rejection call modification (Priority, Call-Info) RFC 3050 Upload via web or REGISTER

60 SIP CGI contd. Block if (defined $ENV{SIP_FROM} && $ENV{SIP_FROM} =~ { print "SIP/ I can't talk right now\n\n"; } Make calls from boss as urgent: if (defined $ENV{SIP_FROM} && $ENV{SIP_FROM} =~ { foreach $reg (get_regs()) { print "CGI-PROXY-REQUEST $reg SIP/2.0\n"; print "Priority: urgent\n\n"; }

61 CPL XML-based; interpreted Control Restricted to branching and subroutines no variables, no loops structured tree describing actions performed on call setup event Event incoming and outgoing events Switch address, language, time, priority Action Proxy, redirect, reject (busy) Graphical tools Protocol-independent SIP, H.323 Extensions CPL for presence Logging Call not match match actions

62 CPL contd. <time dtstart=" T140000Z" dtend=" T150000Z">

63 SIP servlet Java-based Runs in SIP server Receives SIP objects and processes them public class RejectServlet extends SipServletAdapter { public boolean doInvite(SipRequest req) { SipResponse res = req.createResponse(); res.setStatus(603); res.send(); return true; }