Streaming Architectures for Very Large Collections Jim DeRoest Director Streaming Media Technologies, ResearchChannel Assistant Director, University Computing Services University of Washington

Who We Are And Support ResearchChannel/UWTV - Exploring digital convergence and educational outreach. National Echostar and CATV to over 5 million homes International VoD and live streams – I2 Abilene Experiments - First cinema quality HD over IP KEXP – UW and EMP collaboration exploring digital radio Uncompressed 1.4Mb live stream Real time web playlist generation Real time web playlist generation World music format – DJ choice Faculty, Staff, Student, Clinical, K20 Other Affiliate/Collaborative Projects

Managing Content Large Collection Problem Disparate Asset Types Disparate Asset Types HD, Broadcast TV, audio libraries, pictures, rich text Instantiation: capture, encoding, metadata, storage Instantiation: capture, encoding, metadata, storage Multiple sources, formats, schema Interfaces Interfaces TV post production, radio DJ, viewer/listener Discovery and Administration Discovery and Administration Naming, categorization, relations, internationalization Distributed management, sharing, searching Access Controls Access Controls Authentication, authorization, rights, privacy, security

MediaWeb Project 18 month technology/requirements review Requirements Interviewed engineers, DJ’s, librarians and consumers Interviewed engineers, DJ’s, librarians and consumers Reviewed vendor products Proprietary, 70% solution, consumer specific Proprietary, 70% solution, consumer specific Created prototype during review process Pluggable, scalable, standards based architecture Pluggable, scalable, standards based architecture Buy, build, both, call it quits? Decision to go with prototype Decision to go with prototype

Robotic Tape Robotic Tape Disk Cache Disk Cache Media Store Asset Metadata Asset Metadata Remote Stream Manager Remote Stream Manager Request Director Transcoding Metadata Creation, Admin Asset Management Services TV Broadcast Services Radio DJ Services Delivery Services Image, Quicktime, Windows, Real, HD, Webcasting … UW MediaWeb – Digital Media Asset Management System

Core Services Media Store HSM online/nearline multi-vendor HSM online/nearline multi-vendor High Performance Storage System (HPSS) Gigabit interconnects. Gigabit interconnects. Distributed, redundant, recoverable Distributed, redundant, recoverable Block I/O stream to servers (smoke and mirrors Block I/O stream to servers (smoke and mirrors Metadata Service SQL, directory, imbedded objects, access control, rights SQL, directory, imbedded objects, access control, rights Stream Manager Maintains asset locality state and aging Maintains asset locality state and aging Request Re-Director Re-direct requests to target stream or application service Re-direct requests to target stream or application service

Feeds Video Airtime/Dubtime to 50Mb MPEG2 i-frame base Louth/Mav triggers/APIs Louth/Mav triggers/APIs Metadata generation Looking at Virage plugins Looking at Virage pluginsAudio CD ripping raw audio base Metadata map CDDB sites (Muse, AMG, …) DJ metadata support Transcode from base formats as needed Transcode from base formats as needed

Metadata Schema – How rich can we afford Consumer specific Standards – Dublin Core, MPEG7 Vendor (Muse, AMG, Louth playlists, …) Draft submitted for comment I2-VidMid VoD and ViDe Access/Rights controls Vendor specific, MPEG21 Auth*2 – NetID Pubcookie, Shibboleth, … Imbedded – Honored by clients Trust Metadata Objects SQL (base), LDAP, Imbedded objects/methods

Delivery Streaming Servers Standard file system or IO API required Stream initiated as first blocks arrive to local cache Tape, disk, OS and network speed vs. hybrid file systems Selectable stream stub support First +30 seconds of clip hide mount time content on tape Low bit rate (no MPEG1) Quicktime, Real, Windows Media High bit rate Windows Media – KEXP 1.4Mb live stream Optivision – RC 5.6Mb live stream to remote sites MPEG2 – Concurrent Computer Xstreme 5.6Mb VoD MPEG4 – Looking at IBM Videocharger – interface issues HD – In-house (>270Mb), other vendor lower bit rates

MediaWeb Project Status Prototype operational with simple search Geographically separated/mirrored server environment Video: RC/UWTV content (~700 hrs MPEG2, lower bit rates) Audio: Sample audio Refining schema Object experiments Imbedded controls (MPEG7) Reviewing large vendor schemas Deploying admin tools Building feed systems (RC, +35,000 disc KEXP library) Incorporating faculty/student mgt (expiration, quotas, …) Evaluating other large content requests Ongoing evaluation, scaling, benchmarking architecture Looking for opportunities to collaborate!

Experiments UW CSci animation teaching tool “How Networks Change Teaching: A Case Study” Louis Fox, Kris Martin, Barbara Mones-Hattal I2 VIMM Friday, October 5 th 1:30EDT KEXP live stream archives KEXP automated playlist EMP live webcasting and archival UW Catalyst collection interface HD IP Videoconferencing