1. Media Services in CINEMA Presented by Kundan Singh Joint work with Henning Schulzrinne, Jonathan Lennox, Xiaotao Wu, Wenyu Jiang, Sankaran Narayanan, Ajay Nambi and Ali Khwaja Computer Science Department, Columbia University April 25, 2003

2. 2 Purpose What are media services? What is CINEMA? What services are provided by CINEMA? Conferencing Streaming media Unified messaging Interactive voice response How can we use CINEMA? The discussion on services and specific CINEMA implementation components are intermixed. N N Not relevant or not implemented On-going work Yes, Implemented O Y

3. 3 IP telephony services (PSTN) Call routing services: pre-call, one party speed dial call forwarding “follow me” call filtering/blocking (in/out) do not disturb distinctive ringing Call handling features autoanswer Multi-party features call waiting call transfer (blind, consultative) conference call call park call pickup music on hold call monitoring O IP-telephony can’t win by just providing what PSTN has

4. 4 IP telephony services (Internet) Presence-enabled calls place call only if callee is available Presence-enabled conferencing call conference participants when all are online and not busy IM conference alerts receive IM when someone joins a conference Unified messaging receive email, IM alert for new voicemails O

5. 5 Where the services reside? Common gateway interface (CGI) Call processing language (CPL) SIP servlet Make call when boss is online … Enter your authentication PIN for billing… B2BUA Double ringing sound when boss calls… Endpoint Forward to office phone during day, and home phone during evening… Proxy/registrar Endpoint Language for End System Services (LESS) Use finger for locating user… O

6. 6 Program Call routing “Bricks” in media applications SIPSAP RSVPRTCP RTP Media G.711 MPEG RTSP Signaling Quality of service Media transport Internet Telephony Internet Radio/TV Messaging and Presence Interactive voice response Unified messaging Video conferencing Physical layer Link layer Network (IPv4, IPv6) Transport (TCP, UDP) Application layer Voice XML DTMFMixing Speech/ text N SDP

7. 7 … Audio Packet Transfer Digitization (e.g., sampling at 8kHz, 16 bits per sample, i.e, 128 kb/s or 320 bytes per 20 ms) Real-time compression/encoding (e.g., G.729A at 8 kb/s) Transport to remote IP address and port number over UDP (Why not TCP?) Processing on receiver side is the reverse N

8. 8 Unreliable UDP a)Packet loss b)Out-of-order (very rarely) c)Jitter (delay variation) 123576 1234567 timeline Sender Receiver (a) (b) Problems with UDP N

9. 9 123576 1234567 Sender Receiver 890234 8901234 127 1257 89023 123567 8902 Receive Buffer Receive buffer: to absorb jitter Tradeoff in buffer size Adaptive delay adjustment Sequence number: to detect packet loss; Just ignore the loss! Y

10. 10 Encoded Audio RTP Header UDP header IP header msg sendto(…, msg, …) recvfrom(…, msg, …) Sequence number Optional contributors’ list (CSrc) Source identifier (SSrc) Timestamp (proportional to sampling time) Payload typeCCMV PX Real-time Transport Protocol Y RTP: media transport RTCP: QoS feedback

11. 11 Streaming media Messages RTSP vs HTTP UDP, Stateful, Control, Timed Applications Recording, Playback, Live media, Conference Web browser Media player file Web server HTTP Web browser Media player meta file HTTP RTSP RTP Media server Y

12. 12 Columbia rtspd Existing Playback/recording of  - law (Sun.au files) and raw Unicast address Multi-threaded; Portable (Unix/win32) Works with Apple QuickTime client UDP + TCP IPv4 + IPv6 Planned Multicast MPEG video Mp3, wav audio Media over TCP Live media Y

13. 13   Mixer Transcoder  -law G.729  -law RTP-based Conference N

14. 14 What is the IP address of Alice’s host? What audio/video encoding can it support? bob’s host Alice’s host 128.59.19.194 REGISTER as alice=>128.59.19.194 Call alice 128.59.19.194 Why do we need signaling? Session Initiation Protocol Email-like address: alice@home.com DNS Proxy, redirect Y home.com

15. 15 AliceBob INVITE alice@home.com I can support  -law and G.729 Send me audio at 202.16.49.27:6780 OK; I can support  -law Send me audio at 128.59.19.194:8000 202.16.49.27 128.59.19.194 To port 6780 To port 8000 RTP Session description Y

16. 16 Columbia sipc Y

17. 17 Emergency service Calling E911, location Assistance Audio/video, remote-help Notification Earthquake, fire O

18. 18 Presence/event notification PA registrar Presence server office.com SUBSCRIBE NOTIFY REGISTER alice@home.com bob@office.com PUA PUA + PA Y

19. 19 Columbia IM and presence Y

20. 20 Endpoint call control Language for End System Services (LESS) for endpoint service creation Direct user interaction, direct media control Handle converged information, e.g., call, presence, email Example: when buddy is online, make a call <less name="online_call" require="generic presence ui"> <alert sound=“ring.au" text="Calling …" /> O

21. 21 Network call control SIP-CGI CPL SIP servlets Priority.pl SIP_FROM SIP_TO stdin CGI-PROXY-REQUEST stdout SIP proxy Urgent Low-priority Voicemail Phone Y if (defined $ENV{SIP_FROM} && $ENV{SIP_FROM} =~ /sip:boss@mycompany.com/) { foreach $reg (get_regs()) { print "CGI-PROXY-REQUEST $reg SIP/2.0\n"; print "Priority: urgent\n\n"; }

22. 22 Call transfer REFER Blind/ consultation/ attended active call REFER C Referred-By: B INVITE C Referred-By: B BYE A A B C active call O

23. 23 B2BUA and third-party call control Back-to-back UA Incoming call triggers outgoing call Services Calling card Anonymizer INVITE A B C SIP OK (SDP1) ACK INVITE (SDP1) OK (SDP2) ACK INVITE (SDP2) OK ACK O

24. 24 Voicemail Problems in PSTN Design alternatives Issues alice@vmail.yahoo.com Redirect after 10s vmail.pl SIP_FROM SIP_TO stdin CGI-PROXY-REQUEST stdout If no response accept after 15s Endpoint redirects Voicemail acts like a phone Proxy controls Y

25. 25 Columbia sipum Goals Universal access Scalability Provider independent Why SIP and RTSP? Reuse existing infrastructure and tools (1) INVITE INVITE OK CANCEL (3) OK (2) SETUP (4) RTP (5) BYE email rtspd sipum Y

26. 26 Columbia sipum Retrieval Web interface rtsp://server/alice /inbox/1677.au, sip:alice-1677- retrieve@server press 1 to listen… Configuration Folders Options Email Y

27. 27 VoiceXML Telephone PSTN Voice gateway Web server Service logic (CGI, servlet, JSP) Voice and telephony functions VoiceXML browser Internet user VXML HTML Internet IVR platform Voice and telephony functions (ASR, TTS, DTMF) Service logic (application specific) sipvxml Gateway Y

28. 28 VoiceXML contd. Enter your Id: Your ID, please. Telephony, speech synthesis or audio output, user input and grammar, program flow, variable and properties, error handling, … Y

29. 29 DTMF Telephone SIP/PSTN gateway PSTN Internet Internet user D D Audio SIP INFO RTP/ RFC2833 Y

30. 30 Columbia sipvxml Unified messaging access Email by phone Event notification and scheduling Audio volume control for conference Advanced conference control Telephone SIP/PSTN gateway PSTN sipvxml SIP phone Web server Internet Email.tcl Media server rtspd TTS, ASR, DTMF, XML, HTTP, RTSP Y

31. 31 Email + phone Login Email formatting Listen, reply, delete, compose, forward Navigation -next, previous, jump Email formatting SIP based Text-to-speech VoiceXML browser Email servlet JSP DB procmail Email Email to IM Call Interne t TTS important mails SIP Interne t SIP HTTP Email by phone Email to phone Y Inbox

32. 32 IM + voice call Who can initiate? authentication, billing, … Feedback to voice user to IM user initial IM greeting Talk-spurt detection Speech recognition IM Call TTS ASR SIP MESSAGE SIP INVITE RTP Y

33. 33 Unified messaging The gaps among different media (audio, video, text), devices (PC, phone) and means of communications (Email, SIP, IM) disappear for messaging O

34. 34 Notification Calendar Events Conferences Web server Calendar.cgi “at 6:00pm” Schedule from a browser SIP call 2129397063 IM kundan@ Email Kns10@… O

35. 35 Phone announcement server Input Range:93970?? List: A, B, C Example Announcement Emergency Issues Voicemail Failure detection PAS TTS Destinations SIP Text or audio... Y

36. 36 RTSP + TTS + ASR Media server TTS ASR SETUP rtsp://server/tts.cgi ?text=How+are+you. PLAY RTP SETUP rtsp://server/asr.cgi RECORD RTP SET_PARAMETER Text=I am fine, thank you. O

37. 37 Audio conference models (non-multicast) A B C D  B A+C+D A B C A B+C+D A+B+D C D D A+B+C  A B C D     Topology star full-mesh ad-hoc AdvantagesHeterogeneous simple clients No central point of failure DisadvantagesExternal server with high bandwidth link Complex endpoints Complex signaling Typically only three party conferences N

38. 38 Centralized conferencing Conference as URL staffmeet@office.com On the fly conferences my.adhoc@server Basic task: join/leave Dial in, Refer dial in Dial out, Refer dial out REFER INVITE REFER INVITE server O

39. 39 Columbia sipconf G.711, GSM, DVI, G.722 mixing (decode-mix-encode) Video replication; IM; text; VNC screen sharing D D D E E E A B C G711 DVI GSM Linear G711 DVI G711 Playout delay Periodic timer M=A+B+C Mixed linear M - A=B+C M - B M - C Receive thread Send thread Y

40. 40 Columbia sipconf Y

41. 41 Conference control Access control Floor control Notification State Participant: active, departed, booted, failed dialog state, floor state Media sending: received-by-all, muted receiving: receiving-all, anchor-only active departed O

42. 42 Conference + VoiceXML sipvxml Caller sipconf 1. INVITE sipvxml 2. Call accepted 3. Enter your four digit PIN 4. Entered 4-6-8-3 5. Authenticate user, 4683=>Alice 6. Enter the conference identifier 7. Entered 2-3-# 8. Permission to join, 23=>meet 9. REFER meet@conference 10.Terminate the old call 11.INVITE meet@conference Call transfer vs bridged mode Y

43. 43 New conference applications sipvxml Caller 1. INVITE sipvxml 2. Menu 1. Vol Check 2. Mic Check 3. User enters 2 4. User speaks out a voice sample 5. Voice sample is analyzed 7. User adjusts the vol level. The ease & flexibility of sipvxml enables us to build custom telephonic applications to suit our needs. e.g., Volume Check Application sipconf 8. User now joins conference. 6. SipVXML: Vol level too high/low/… Y

44. 44 Conferencing Automatic volume adjustments Automatic load balancing Delay adjustments Conference recording Local or RTSP Ysipc SIP323 SIP/PSTN Recording in a media server

45. 45 Ncast MPEG distribution sip.cgi?sip:xyz@cs.columbia.edu INVITE sip:xyz@cs.columbia.edu Y

46. 46 My owner’s SIP address is sip:john@cs.columbia.edu sip:john@cs.columbia.edu Help!!! (invoke sipc to call sip:john@cs.columbia.edu) INVITE sip:john@cs.columbia.edu Y

47. 47 Summary of media services Streaming media Playback, recording, live-media Unified messaging Voicemail, video/IM integration,... Conferencing Mixing, load-balancing Interactive voice response Tele-banking, VoiceXML Y

48. 48 CINEMA Columbia InterNet Extensible Multimedia Architecture CINEMA Distributed component architecture Libraries and APIs Management and control Application and proxy servers User-agent (client) applications Y

49. 49 Columbia SIP user agent: sipc Media Audio, video, text, white board Screen sharing Shared web browsing Advanced Presence, instant messaging Programmable call handling Conference control (on-going) Emergency notification and handling (on-going) Device control Language for End System Services (on-going) Y

50. 50 Columbia SIP servers (CINEMA) Internal Telephone Extn: 7040 SIP/PSTN Gateway Department PBX Web based configuration Web server Telephone switch SQL database sipd: Proxy, redirect, Registrar server NetMeeting H.323 rtspd: media server sipum: Unified messaging Quicktime RTSP clients RTSP 713x Single machine SNMP (Network Management) sipconf: Conference server siph323: SIP-H.323 translator Local/long distance 1-212-5551212 PSTN Video encoder Y

51. 51 Libraries and APIs sipdsip323sipconfsipumsipvxmlrtspd CINEMA Libraries libNT Win32 stub libcine Utilities parsing IPv6 libsip Basic SIP library libsip++ SIP UA library libmixer RTP audio mixer libdict Hash table libdb++ mySQL intf RTSP media server SIP proxy server SIP/H.323 gateway SIP/RTP conferencing SIP/RTSP unified messaging SIP/VoiceXML browser Xerces-C OpenH323 MySQL PWLib Resparse librtsp RTSP client librtp RTP library libsnmp SIP MIB ViaVoice/Flite Xerces-C Server Applications Parsing, SIP, SDP, RTP, mySQL interface, SNMP interface, Portability stubs, etc. CPL, CGI, Servlet Y

52. 52 Status of CINEMA services Features/servicesStatus SIP proxy, redirect, registration serverProduct SIP Presence agent; programmable call routingBeta Dial-in conference with audio, video, screen sharing, instant messaging, recording, load balancing, volume or delay adjustment, and IVR Beta Dial-out or refer dial-in conferencing, conference controlOn-going Voice mail with retrieval, deletion and IVRBeta Media streaming server for basic audio record and playBeta SIP/VXML browser for VoiceXML 1.0 with basic functionalityAlpha Call transfer, back-to-back user agentOn-going Phone announcement server, IM/call converter, email-by-phone and email-to-phone, SIP anonymizer Demo Calendar event notification to IM, phone and emailOn-going Web-based configuration and controlAlpha Media server with text-to-speech, speech-recognition, video, live mediaOn-going

53. 53 What’s next? from multimedia communications to collaboration Synchronous communications Conferencing, IM Asynchronous communications Voicemails, message board, file sharing Ubiquitous computing i-button, ID-card Service creation User friendly, end-point/network kns10@cs.columbia.edu http://www.cs.columbia.edu/~kns10/ http://www.cs.columbia.edu/IRT/cinema

54. Media Services in CINEMA BACKUP SLIDES

55. 55 DO sip:lamp@cs.columbia.edu SIP/2.0 ….. turn lamp on serial port lamp X10 device SIP beyond telephony SUBSCRIBE door@alice.home.net NOTIFY alice@work.com DO light@alice.home.net INVITE camera@alice.home.net

56. 56 PSTN interworking Translating audio (PCMU/PCMA) Translating signaling (PRI/T1,ISUP) Overlap signaling Advanced features in SIP are lost in PSTN Translating identifiers (phone number) Determining transition points Telephone network SIP/PSTN gateway SIP serverIP endpoint Telephone subscriber +1 212 9397063 sip:bob@home.com

57. 57 PSTN to IP Gateway knows the SIP server ENUM DNS +1 212 9397042 => 2.4.0.7.9.3.9.2.1.2.1.e164.arpa => sip:hgs@cs.columbia.edu Suitable for relatively “static” contacts

58. 58 IP to PSTN Static mapping 1-212-939xxxx => @itgw1.cs.columbia.edu ITGW information is dynamic: Overlapping networks Multiple providers Load balancing TRIP Route advertisement Can be implemented in outbound proxy Suitable for current hierarchical network +1 @service.mci.com at 4¢/min +1212 @nyc.gw.com at 1¢/min +1212939 @itgw1.columbia.edu free

59. 59 SIP-CGI Programming language independent Maintains state via an opaque token For SIP proxies and endpoints: call routing controlling forking call rejection call modification (Priority, Call-Info) RFC 3050 Upload via web or REGISTER

60. 60 SIP CGI contd. Block *@anonymous.com: if (defined $ENV{SIP_FROM} && $ENV{SIP_FROM} =~ "sip:*@anonymous.com") { print "SIP/2.0 600 I can't talk right now\n\n"; } Make calls from boss as urgent: if (defined $ENV{SIP_FROM} && $ENV{SIP_FROM} =~ /sip:boss@mycompany.com/) { foreach $reg (get_regs()) { print "CGI-PROXY-REQUEST $reg SIP/2.0\n"; print "Priority: urgent\n\n"; }

61. 61 CPL XML-based; interpreted Control Restricted to branching and subroutines no variables, no loops structured tree describing actions performed on call setup event Event incoming and outgoing events Switch address, language, time, priority Action Proxy, redirect, reject (busy) Graphical tools Protocol-independent SIP, H.323 Extensions CPL for presence Logging Email Call not match match actions

62. 62 CPL contd. <time dtstart="20020830T140000Z" dtend="20020830T150000Z">

63. 63 SIP servlet Java-based Runs in SIP server Receives SIP objects and processes them public class RejectServlet extends SipServletAdapter { public boolean doInvite(SipRequest req) { SipResponse res = req.createResponse(); res.setStatus(603); res.send(); return true; }