1. Construction of an Immersive Mixed Environment Using an Omnidirectional Stereo Image Sensor
Jun Shimamura, Naokazu Yokoya, Haruo Takemura and Kazumasa Yamazawa
Graduate School of Information Science
Nara Institute of Science and Technology JAPAN

2. IEEE Workshop on Omnidirectional Vision 2000
Motivation
How to digitize a dynamic outdoor scene for creating an immersive virtualized environment where the followings are possible.
(1) interactive viewing of the environment with 3-D sensation and
(2) computer graphics (CG) object synthesis maintaining correct occlusion between real and virtual objects.
June 12, 2000
IEEE Workshop on Omnidirectional Vision 2000

3. IEEE Workshop on Omnidirectional Vision 2000
Our approach
A full panoramic 3-D model is made from a cylindrical panoramic image using a video-rate omnidirectional stereo image sensor
CG objects are merged into the full panoramic 3-D model of real scene with correct occlusion
A prototype of immersive mixed reality system using a large cylindrical screen is developed
Our approach is based on acquiring full
panoramic 3-D models of real worlds using a video-rate
omnidirectional stereo image sensor [5]. The sensor can
produce a pair of high-resolution omnidirectional binocu-lar
images that satisfy the single viewpoint constraint; that
is, each omnidirectional image is obtained from a single
effective viewpoint. A full panoramic 3-D model is made
from a cylindrical panoramic image and depth to each pixel
from the center the cylinder. The secondary objective is to
construct a prototype system which presents a mixed environment
consisting of real scene model and CG model.
CG objects are merged into the full panoramic 3-D model
of real scene maintaining consistent occlusion between real
and virtual objects. Thus it is possible to yield rich 3-D
sensation with binocular and motion parallax.
We have developed a prototype of immersive mixed reality
system using a large cylindrical screen, in which a user can
walk through the mixed environment and can manipulate
CG objects in real time.
June 12, 2000
IEEE Workshop on Omnidirectional Vision 2000

4. Omnidirectional Stereo Imaging Sensor
Twelve CCD cameras and two hexagonal pyramidal mirrors.
June 12, 2000
IEEE Workshop on Omnidirectional Vision 2000

5. Generation of panoramic stereo images
Elimination of geometric distortion in images,
Color adjustment of different camera images,
Concatenation of six images for completing upper and lower omnidirectional images of stereo pair.
June 12, 2000
IEEE Workshop on Omnidirectional Vision 2000

6. Panoramic stereo images
June 12, 2000
IEEE Workshop on Omnidirectional Vision 2000

7. Virtualizing a Dynamic Real Scene
Layered representation of a dynamic scene
Static scene image generation
Moving object extraction
Depth estimation from static panoramic stereo image and moving object
Generation of layered 3-D model
June 12, 2000
IEEE Workshop on Omnidirectional Vision 2000

8. IEEE Workshop on Omnidirectional Vision 2000
A pair of panoramic stereo images of a static scene without moving objects.
Majority
Filtering
Dynamic Panoramic
Images
Static Image
June 12, 2000
IEEE Workshop on Omnidirectional Vision 2000

9. Extracted moving object regions in the upper panoramic image.
June 12, 2000
IEEE Workshop on Omnidirectional Vision 2000

10. IEEE Workshop on Omnidirectional Vision 2000
Depth estimation from panoramic stereo images
June 12, 2000
IEEE Workshop on Omnidirectional Vision 2000

11. Generating Depth Image
June 12, 2000
IEEE Workshop on Omnidirectional Vision 2000

12. Panoramic depth map generated from panoramic stereo images
June 12, 2000
IEEE Workshop on Omnidirectional Vision 2000

13. Generating Full Panoramic Scene Model
June 12, 2000
IEEE Workshop on Omnidirectional Vision 2000

14. Bird’s-eye view of texture-mapped 3-D scene model.
June 12, 2000
IEEE Workshop on Omnidirectional Vision 2000

15. IEEE Workshop on Omnidirectional Vision 2000
Cylindrical 3-D model
June 12, 2000
IEEE Workshop on Omnidirectional Vision 2000

16. Immersive Mixed Reality System
Prototype system with 330 degree cylindrical screen
Merging virtual objects into a virtualized real world scene
Superimposing dynamic event layers onto static scene layer
June 12, 2000
IEEE Workshop on Omnidirectional Vision 2000

17. Cylindrical Display System
6144x768 pixel resolution
June 12, 2000
IEEE Workshop on Omnidirectional Vision 2000

18. Cylindrical Display (Video)
6 Projectors
June 12, 2000
IEEE Workshop on Omnidirectional Vision 2000

19. Merging virtual objects
Background:
13400 Polygons
CG Tree:
41340 Polygons
Total:
54740 polygons
June 12, 2000
IEEE Workshop on Omnidirectional Vision 2000

20. Superimposing dynamic event
June 12, 2000
IEEE Workshop on Omnidirectional Vision 2000

21. Superimposing dynamic event
June 12, 2000
IEEE Workshop on Omnidirectional Vision 2000

22. IEEE Workshop on Omnidirectional Vision 2000
Summary
A novel method of constructing a large scale mixed environment.
A combination of cylindrical 3-D model of a real scene and polygonal CG objects.
Depth values are appended to the cylindrical panoramic image of real scene, so that CG and 3-D model has correct occlusion.
A prototype system was built with 330 degree cylindrical screen system.
June 12, 2000
IEEE Workshop on Omnidirectional Vision 2000

23. IEEE Workshop on Omnidirectional Vision 2000
Future work
Extending the model far larger by using multiple panoramic stereo images.
Implementing an algorithm that smoothly switches constructed models when user’s viewpoint changes.
June 12, 2000
IEEE Workshop on Omnidirectional Vision 2000

24. IEEE Workshop on Omnidirectional Vision 2000
Thank you for your attention
June 12, 2000
IEEE Workshop on Omnidirectional Vision 2000