Immersive Media Vision Create seamless immersive environment for distributed, interactive real and virtual events Reproduce audio, video and other senses with fidelity approaching the limits of human perception Relevant to any type of human interaction: entertainment, education, communications

IMSC Entertainment Vision: RMI A fusion of internet browsing with a theater-like immersive experience HD Video at up to 45 Mbits/sec 10.2 ch Immersive Audio (16 Mbits/sec) Streaming over the Internet on-demand to a mouse click (20 min prog) RMI demonstrates IMSC innovations in several critical technical areas: Immersed in a college football game Doctors assisting in a remote procedure Business people negotiating like they are in the same room Students visiting an aquarium a thousand miles away Streaming media servers Immersive audio capture and rendering Network protocols for error correction Synchronization Applications

Premier Demonstration May 9, 2002 Cross-country streaming from ISI-East in Virginia NY Times coverage in Circuits section NBC-TV and KTLA

Internet2 Fall 2002 Member Meeting Audio: 10.2 channel, immersive sound system Video: HDTV 1920x1080i New World Symphony, Miami Beach 550 seat USC Bing Theater

University of Maryland Server USC USC Client New World Symphony Server Georgia Tech Server RMI Experimental Setup Synchronized immersive audio and HDTV streamed playback from Yima server over Internet2 Control of end-to-end process: capturing, network interface, transmission, rendering CENIC 2003 Gigabit or Bust Award

Remote Master Class LA Philharmonic cellist located at our USC lab Student at New World Symphony in Miami Beach Three hour session with MPEG2 video and 10.2 immersive audio Teacher reports that student "was really there" with immersive audio Many psychophysical, perceptual and artistic tests to be done

Recent Developments... Two-way audio communication to New World Symphony in Miami Beach, Florida 8-channel audio – low latency Live video capture/transmission (DVCam) Near term prospects RMI theater at Inha University (Korea) 2-way live HD-video/immersive audio streaming for Internet collaboration

Distributed Immersive Performance Create seamless immersive interactive environment Distributed musicians, conductor (active) and audience (passive) Q: Why a musical performance? A: A distributed orchestra is one of the most challenging immersive, interactive environments Scenario: Multiple soloists scheduled in different cities Under served areas can be reached

IMSC Communication Vision InterAct! UWB console captures 3D model and problem details PDA provides spatial array of participants and sonic translation of details Immersive workstation coordinates video, avatars, graphics, audio, and haptics to facilitate problem resolution

Research Highlights immersive audio multichannel and HRTF approaches - holistic DSP approach streaming servers distributed and scalable architecture computer vision computational framework for grouping based on tensor voting, tracking for augmented realities and SFX graphics & animation 3D DSP mesh processing, compression, mesh operations, hair modeling and animation multimodal emotive interfaces Speech and dialog, vision sensing of body and hands, facial expressions analysis and expressive avatars virtual reality -- chair of VR2003 in LA applications to psychology (ADD diagnosis), haptics applications, and user studies perception & cognitive modeling theoretical foundations for understanding benefits of media immersion UWB wireless leadership in technology, FCC regulation, and commercialization IMSC has produced ground breaking fundamental research in:

Sensory Interfaces (SI) Robust Vision Systems Segmentation, motion, 3D body and hand tracking, tensor voting, activity analysis, recognition and tracking Speech Recognition and Synthesis Emotive dialog, translation, emotion analysis Immersive Audio Sonic visualization, spatialization, autocalibration Facial Analysis -- Gestures and Animation Expression signatures for emotion and recognition Video fusion with 3D models for Surveillance Structure extraction from LiDAR and texture projection Digital Geometry (3D) Processing Compression, surface and topology smoothing, mesh simplification