Creating and Exploring a Large Photorealistic Virtual Space INRIA / CSAIL / Adobe First IEEE Workshop on Internet Vision, associated with CVPR 2008.

Outline Introduction Constructing the image space Navigating the virtual 3D space Limitations and Conclusion

Introduction We present a system for exploring large collections of photos in a virtual 3D space. Let users navigate within each theme using intuitive 3D controls that include move left/right, zoom and rotate.

In a similar fashion we can create infinite zoom effects that resemble the ”Droste effect”.

Constructing the image space The image database – We have collected ~6 million images from Flickr based on keyword and group searches typical image size is 500x375 pixels 720GB of disk space (jpeg compressed)

Image representation Color layout GIST [Oliva and Torralba’01] Original image

Obtaining semantically coherent themes We further break-up the collection into themes of semantically coherent scenes: Train SVM-based classifiers from 1-2k training images [Oliva and Torralba, 2001]

Basic camera motions Forward motionCamera rotation Camera pan Starting from a single image, find a sequence of images to simulate a camera motion:

3. Find a match to fill the missing pixels Scene matching with camera view transformations: Translation 1. Move camera 2. View from the virtual camera 4. Locally align images 5. Find a seam 6. Blend in the gradient domain

4. Stitched rotation Scene matching with camera view transformations: Camera rotation 1. Rotate camera 2. View from the virtual camera 3. Find a match to fill-in the missing pixels 5. Display on a cylinder

Steps Collect images Classify images into topic themes Calculate the descriptors: – GIST – RGB Build the graph Find the path for given query image(s) – Dijkstra algorithm Alignment – Gradient decent alignment Composition – Poisson blending

More “infinite” images – camera translation

Forward Rotate (left/right) Pan (left/right) Nodes represent Images Edges represent particular motions: Edge cost is given by the cost of the image match under the particular transformation Image graph Navigating the virtual 3D space Virtual space as an image graph

Virtual image space laid out in 3D

Limitations and Conclusion The larger the database, the better the results. Compositing two distinct images is always a challenge and at times, the seam is quite visible. This system can be used to create photorealistic visual content for large online virtual 3D worlds like Second Life. Create infinite panoramas or use the image taxi to generate tailor-made tours in the virtual 3D space. These applications can find use in games, movies and other media creation processes.

Thank you ！