Interactive Video Tours MSR Interactive Visual Media Group //msrweb/vision/IBR Rick Szeliski, Sing Bing Kang, Matt Uyttendaele, Simon Winder, Antonio Criminisi

Being There: Video-Based Reality Project A new computer-mediated interactive medium Goal: Combine the interactivity of 3D games and virtual worlds with the realism and richness of film and video

Being There: Video-Based Reality Project How? Capture real-world video footage with an omni- directional camera and add interactive video/graphic elements Application: Video-Based Walkthroughs [IEEE CG&A May/June 2004]

Virtual Tours  Move camera along a rail (“dolly track”) and play back a 360  video  Applications: –Homes and architecture –Outdoor locations (tourist destinations)

Surround video acquisition system OmniCam (six-camera head)

OmniCam  Built by Point Grey Research (Ladybug)  Six camera head  Portable hard drives, fiber-optic link  Resolution per image: 1024 x 768  FOV: ~100 o x ~80 o  Acquisition speed: 15 fps uncompressed

Acquisition platforms  Robotic cart  Wearable

Calibration: Lens distortion Original (distorted) imageOutput (corrected, perspective) image

Calibration: Vignetting Before correctionAfter correction

Stitching (Only 4 of 6 images shown here)

Problem with simple feathering

Multiperspective plane sweep With MPPSWithout MPPS

Feature tracking for calibration and ego-motion Points tracked in all frames in all 6 cameras Edges tracked in all frames in all 6 cameras

Stabilization Before motion stabilizationAfter motion stabilization Align frame- to-frame and distribute  heading

Augmented surround video A movie clip from the original surround data. The augmented movie clip. Surround augmented with video

Viewing Software  D3D + DirectShow + XML scene graphs  Model: 6 texture mapped planes –Rendered using D3D hardware  Rendering software: –Selective decompression –Joystick controller –Overlays support

Demo A Virtual Home Tour

 Head-mounted outdoor scene acquisition  Map control  Localized audio for a richer experience  Complex path navigation (greater freedom of motion) A second scenario: Bellevue Botanical Gardens

Data acquisition  Walk through the garden with the Ladybug camera mounted on helmet  Acquisition done in about 15 minutes  44 Gb worth of video and audio data

Map control The garden map Drawing the acquisition path Mapping video frames positions on map Placing sound sources (background and dynamic) Output is a descriptive XML file

Audio processing and streaming All audio sources are: 1.Attenuated (based on distance from current position), 2.Mixed together, 3.Streamed to sound card (DirectX Audio). Two types of sound: background and dynamic Some audio sources Current ego-position

Bifurcation handling  Hypothesize, align, choose best pair as bifurcation  Choice of path depends on current heading and bifurcation point orientation Current heading

Bifurcation handling  Hypothesize, align, choose best pair as bifurcation  Choice of path depends on current heading and bifurcation point orientation Current heading

Demo 2 A walk in Bellevue Botanical Garden

 High-Dynamic range (HDR) imaging  Complex indoor path navigation 3 rd scenario: High-End Home (architectural walkthrough)

 Combine multiple exposures to obtain a high dynamic range (HDR) image. High dynamic range imaging Overexposed Combined Underexposed

All frames Underexposed framesOverexposed frames  HDR imaging is challenging for videos.  Image registration is necessary.

Demo 3 A High-End Home

Sample walkthroughs Garden Tour High-End Home Tour Simple Home Tour

Features and statistics  Rapid scene acquisition  Map control  Localized audio for a richer experience  Complex path navigation  High data rates: –44 Gb worth of video and audio data in 15 minutes

Current/future work  Scene Modeling –Motion estimation and map generation –More interactive 3D elements  Compression & selective decoding –Tiled video representation  Rendering –Better use of GPU  Interactive DVD deployment