1. Interactive Video Tours MSR Interactive Visual Media Group //msrweb/vision/IBR Rick Szeliski, Sing Bing Kang, Matt Uyttendaele, Simon Winder, Antonio Criminisi

2. Being There: Video-Based Reality Project A new computer-mediated interactive medium Goal: Combine the interactivity of 3D games and virtual worlds with the realism and richness of film and video

3. Being There: Video-Based Reality Project How? Capture real-world video footage with an omni- directional camera and add interactive video/graphic elements Application: Video-Based Walkthroughs [IEEE CG&A May/June 2004]

4. Virtual Tours  Move camera along a rail (“dolly track”) and play back a 360  video  Applications: –Homes and architecture –Outdoor locations (tourist destinations)

5. Surround video acquisition system OmniCam (six-camera head)

6. OmniCam  Built by Point Grey Research (Ladybug)  Six camera head  Portable hard drives, fiber-optic link  Resolution per image: 1024 x 768  FOV: ~100 o x ~80 o  Acquisition speed: 15 fps uncompressed

7. Acquisition platforms  Robotic cart  Wearable

8. Calibration: Lens distortion Original (distorted) imageOutput (corrected, perspective) image

9. Calibration: Vignetting Before correctionAfter correction

10. Stitching (Only 4 of 6 images shown here)

11. Problem with simple feathering

12. Multiperspective plane sweep With MPPSWithout MPPS

13. Feature tracking for calibration and ego-motion Points tracked in all frames in all 6 cameras Edges tracked in all frames in all 6 cameras

14. Stabilization Before motion stabilizationAfter motion stabilization Align frame- to-frame and distribute  heading

15. Augmented surround video A movie clip from the original surround data. The augmented movie clip. Surround augmented with video

16. Viewing Software  D3D + DirectShow + XML scene graphs  Model: 6 texture mapped planes –Rendered using D3D hardware  Rendering software: –Selective decompression –Joystick controller –Overlays support

17. Demo A Virtual Home Tour

18.  Head-mounted outdoor scene acquisition  Map control  Localized audio for a richer experience  Complex path navigation (greater freedom of motion) A second scenario: Bellevue Botanical Gardens

19. Data acquisition  Walk through the garden with the Ladybug camera mounted on helmet  Acquisition done in about 15 minutes  44 Gb worth of video and audio data

20. Map control The garden map Drawing the acquisition path Mapping video frames positions on map Placing sound sources (background and dynamic) Output is a descriptive XML file

21. Audio processing and streaming All audio sources are: 1.Attenuated (based on distance from current position), 2.Mixed together, 3.Streamed to sound card (DirectX Audio). Two types of sound: background and dynamic Some audio sources Current ego-position

22. Bifurcation handling  Hypothesize, align, choose best pair as bifurcation  Choice of path depends on current heading and bifurcation point orientation Current heading

23. Bifurcation handling  Hypothesize, align, choose best pair as bifurcation  Choice of path depends on current heading and bifurcation point orientation Current heading

24. Demo 2 A walk in Bellevue Botanical Garden

25.  High-Dynamic range (HDR) imaging  Complex indoor path navigation 3 rd scenario: High-End Home (architectural walkthrough)

26.  Combine multiple exposures to obtain a high dynamic range (HDR) image. High dynamic range imaging Overexposed Combined Underexposed

27. All frames Underexposed framesOverexposed frames  HDR imaging is challenging for videos.  Image registration is necessary.

28. Demo 3 A High-End Home

29. Sample walkthroughs Garden Tour High-End Home Tour Simple Home Tour

30. Features and statistics  Rapid scene acquisition  Map control  Localized audio for a richer experience  Complex path navigation  High data rates: –44 Gb worth of video and audio data in 15 minutes

31. Current/future work  Scene Modeling –Motion estimation and map generation –More interactive 3D elements  Compression & selective decoding –Tiled video representation  Rendering –Better use of GPU  Interactive DVD deployment