Pervasive Pixels (Columbia University Dept. of Computer Science) Henning Schulzrinne (PI) Steven K. Feiner Gail Kaiser John Kender Kathleen McKeown

Proposed Research Goal: seamless mobile multimedia collaboration across distance u Integrate advances across fields u Collaborative work u Graphical and visual interfaces u Spoken language understanding and generation u Vision sensing and understanding u Networking and security

Contributions u Contextual information management u use workflow to determine display content u multimedia summaries of past and present sessions u Harmonizing physical and virtual environments u map changing virtual information onto physical displays u map layout of physical environment onto virtual space u Network services u clear, flexible interface to common services u authentication and privacy support u infrastructure for persistent large displays

Features of Research Infrastructure u Large numbers of instrumented multi- display workspaces u Networked mobile devices of various capabilities u Transparent and automatic adaptability to changes of place, platform or group u Support for a wide range of hardware and software, from commercial to novel

Proposed Research Infrastructure u Outfit informal areas for collaboration u Public areas for walk-by interaction u Multiple touch displays, cameras, audio u Portable units u Stationary setups u Multiple displays, video cameras, audio u Seminar room, meeting rooms u 12 faculty offices u User-based personalization: user location u Triangulation on mobile devices u Visual tracking u Standard methods (e.g., active badge)

Public areas – walk by stations Multiple touch displays, video projectors and cameras, embedded computers, speakers and microphones

IR/RF badge network PC proj. camera card reader loudspeaker ceiling electronic whiteboard microphone array Design for Walk-by Collaboration Station

Public Areas – informal gatherings

Meeting Room Remote- controlled pan- tilt video cameras and projectors, Omnicam, conference table microphones, automatic audio mixer, ceiling speakers

Faculty Office Mimio electronic whiteboard, XGA video projector, Ethernet speaker phone, wall-mounted pan-tilt video camera, PocketPCs

Seminar room Omnicam omnidrectional audience camera, high- resolution DV video camera, 2 pan-tilt speaker cameras, ceiling mounted microphones, electronic whiteboard, XGA high-brightness video projectors

Functionalities u Conferencing u Internet conferencing server to mix IP and PSTN audio streams u Interconnection with analog phone u Digital hybrid connects digital or analog sound to existing telephone system in classroom u Network voice-over-IP interface attached to Nortel Meridian PBX for 20 simultaneous conversations u Multi-processor servers and IA64 compute and database server u File storage u Face, speaker and fingerprint recognition u Backup facilities: 2 printers and tape library system

Initial results HCI: gesture-based user interface for public kiosk mouse replacement for pointing and selecting uses frontal and side camera Security: disCFS and WebDAVA secure file systems disCFS: NFS with credentials instead of authorization WebDAVA: grant restricted access to resources using HTTP and Java applets Web-based collaboration: content on all kinds of devices pass DOM through a series of filters and transformations  HTML Ubiquitous multimedia communications infrastructure being commercialized; I2 demonstration input into standardization (IETF)

Ubiquitous Computing Traditionally, focus on closed environments proprietary protocols single (trusted) user class single site (room, lab, home, …) stand-alone components (“video conferencing”) PP focuses on whole system and user experience Pervasive Pixels networking component: standard protocols: SIP for media configuration, event notification, instant multimedia messaging SLP for service discovery integration of presence and user context standardization in the IETF (RPID) location-based services user context user authorization service location

Mobility in Pervasive Pixels Terminal mobility application-layer mobility complements L3 mobility Session mobility move active sessions to devices found in the environment  service discovery Service mobility move configuration to new devices Personal mobility one user, many devices

Location-based services Traditionally, focus on geospatial location (e.g., GPS) But other aspects as important: civil location (often more intuitive) type of place (home vs. office; outdoors vs. theatre) behavioral: distraction, privacy, appropriateness Experimenting with low- complexity location mechanisms: IR/RF active badges with low installation cost (Ivistar) BlueTooth location beacons LAN backtracking and DHCP swipe cards and i-buttons DHCP server 458/17  Rm /18  Rm. 816 DHCP answer: sta=DC loc=Rm815 lat= long= :0:20:ab:d5:d CDP + SNMP 8:0:20:ab:d5:d  458/17

Some initial lessons learned Usage: remote presence from UKy during sabbatical research group meetings departmental site visit thesis proposals and defenses Perception: “Multimedia collaboration is a mature field” Reality: It doesn’t work much better than in 1992 still fails in hard-to-diagnose ways quality better, but echo, feedback and level issues remain Integration between synchronous and asynchronous collaboration integrating documents, minutes, … Transition from call-focused to presence-focused much larger use of asynchronous collaboration ( , bulletin boards, …) Working with start-up company: new IP-based departmental communication system to replace PBX

Columbia SIP servers (CINEMA) Internal Telephone Extn: 7040 SIP/PSTN Gateway Department PBX Web based configuration Web server Telephone switch SQL database sipd: Proxy, redirect, registrar server Extn: 7134 NetMeeting H.323 rtspd: media server sipum: Unified messaging Quicktime RTSP clients RTSP Extn: x Single machine SNMP (Network Management) sipconf: Conference server siph323: SIP-H.323 translator Local/long distance

Larger lessons for multimedia systems research Software tool support for multimedia communications lacking most are applications, not building blocks cross-platform research media tools are getting very old and creaky (vic, rat, etc.) multi-party support very weak (multicast never happened) Components designed to be operated by humans IP phones only have HTTP/HTML interface video projectors just proprietary configuration Lots of components, but hard to evaluate in real use still mostly barely demo quality: audio delay, echo, random failures people will fall back to good ol’ PSTN quickly

Conclusion Pervasive Pixels = attempt to integrate multiple modalities into system, not just grouping of components Evaluation in real usage, not just demos Spread throughout the department, not just lab