Tracking and Data Fusion for 4D Visualization Ulrich Neumann Suya You Computer Science Department Integrated Media Systems Center University of Southern.

Slides:



Advertisements
Similar presentations
Free-viewpoint Immersive Networked Experience February 2010.
Advertisements

For Internal Use Only. © CT T IN EM. All rights reserved. 3D Reconstruction Using Aerial Images A Dense Structure from Motion pipeline Ramakrishna Vedantam.
Structured Light principles Figure from M. Levoy, Stanford Computer Graphics Lab.
Interactive Video Tours MSR Interactive Visual Media Group //msrweb/vision/IBR Rick Szeliski, Sing Bing Kang, Matt Uyttendaele, Simon Winder, Antonio Criminisi.
Centre for Integrated Petroleum Research University of Bergen & Unifob, Norway Walter Wheeler & Simon Buckley Unifob, UiB Bergen, Norway Lidar laser scanning.
1 Autonomous Registration of LiDAR Data to Single Aerial Image Takis Kasparis Nicholas S. Shorter
Vision Sensing. Multi-View Stereo for Community Photo Collections Michael Goesele, et al, ICCV 2007 Venus de Milo.
MASKS © 2004 Invitation to 3D vision Lecture 7 Step-by-Step Model Buidling.
Vision Based Control Motion Matt Baker Kevin VanDyke.
Fast and Extensible Building Modeling from Airborne LiDAR Data Qian-Yi Zhou Ulrich Neumann University of Southern California.
GIS and Image Processing for Environmental Analysis with Outdoor Mobile Robots School of Electrical & Electronic Engineering Queen’s University Belfast.
A Versatile Depalletizer of Boxes Based on Range Imagery Dimitrios Katsoulas*, Lothar Bergen*, Lambis Tassakos** *University of Freiburg **Inos Automation-software.
Dana Cobzas-PhD thesis Image-Based Models with Applications in Robot Navigation Dana Cobzas Supervisor: Hong Zhang.
A new approach for modeling and rendering existing architectural scenes from a sparse set of still photographs Combines both geometry-based and image.
Plenoptic Stitching: A Scalable Method for Reconstructing 3D Interactive Walkthroughs Daniel G. Aliaga Ingrid Carlbom
Copyright  Philipp Slusallek Cs fall IBR: Model-based Methods Philipp Slusallek.
RANSAC-Assisted Display Model Reconstruction for Projective Display Patrick Quirk, Tyler Johnson, Rick Skarbez, Herman Towles, Florian Gyarfas, Henry Fuchs.
Automated 3D Model Construction for Urban Environments Christian Frueh John Flynn Avideh Zakhor Next Generation 4D Distributed Modeling and Visualization.
Multi-view stereo Many slides adapted from S. Seitz.
Image-Based Rendering using Hardware Accelerated Dynamic Textures Keith Yerex Dana Cobzas Martin Jagersand.
Visual Odometry for Ground Vehicle Applications David Nister, Oleg Naroditsky, James Bergen Sarnoff Corporation, CN5300 Princeton, NJ CPSC 643, Presentation.
Augmented Reality: Object Tracking and Active Appearance Model
CSCE 641: Computer Graphics Image-based Rendering Jinxiang Chai.
Multiple View Geometry : Computational Photography Alexei Efros, CMU, Fall 2006 © Martin Quinn …with a lot of slides stolen from Steve Seitz and.
55:148 Digital Image Processing Chapter 11 3D Vision, Geometry Topics: Basics of projective geometry Points and hyperplanes in projective space Homography.
David Luebke Modeling and Rendering Architecture from Photographs A hybrid geometry- and image-based approach Debevec, Taylor, and Malik SIGGRAPH.
Overview and Mathematics Bjoern Griesbach
Research & Innovation 1 An Industry Perspective on VVG Research Oliver Grau BBC Research & Innovation VVG SUMMER SCHOOL '07.
Supporting Beyond-Surface Interaction for Tabletop Display Systems by Integrating IR Projections Hui-Shan Kao Advisor : Dr. Yi-Ping Hung.
Automatic Registration of Color Images to 3D Geometry Computer Graphics International 2009 Yunzhen Li and Kok-Lim Low School of Computing National University.
Facial animation retargeting framework using radial basis functions Tamás Umenhoffer, Balázs Tóth Introduction Realistic facial animation16 is a challenging.
Accuracy Evaluation of Stereo Vision Aided Inertial Navigation for Indoor Environments D. Grießbach, D. Baumbach, A. Börner, S. Zuev German Aerospace Center.
Consistent Visual Information Processing Axel Pinz EMT – Institute of Electrical Measurement and Measurement Signal Processing TU Graz – Graz University.
Tracking for Scene Augmentation & Visualization Ulrich Neumann Computer Science Department Integrated Media Systems Center University of Southern California.
Dynamic Meshing Using Adaptively Sampled Distance Fields
Visual Perception PhD Program in Information Technologies Description: Obtention of 3D Information. Study of the problem of triangulation, camera calibration.
MESA LAB Multi-view image stitching Guimei Zhang MESA LAB MESA (Mechatronics, Embedded Systems and Automation) LAB School of Engineering, University of.
Exploitation of 3D Video Technologies Takashi Matsuyama Graduate School of Informatics, Kyoto University 12 th International Conference on Informatics.
Monitoring, Modelling, and Predicting with Real-Time Control Dr Ian Oppermann Director, CSIRO ICT Centre.
Digital Image Processing & Analysis Spring Definitions Image Processing Image Analysis (Image Understanding) Computer Vision Low Level Processes:
High-Resolution Interactive Panoramas with MPEG-4 발표자 : 김영백 임베디드시스템연구실.
WSCG2008, Plzen, 04-07, Febrary 2008 Comparative Evaluation of Random Forest and Fern classifiers for Real-Time Feature Matching I. Barandiaran 1, C.Cottez.
A Camera-Projector System for Real-Time 3D Video Marcelo Bernardes, Luiz Velho, Asla Sá, Paulo Carvalho IMPA - VISGRAF Laboratory Procams 2005.
Adaptively Sampled Distance Fields Representing Shape for Computer Graphics Ronald N. Perry and Sarah F. Frisken Mitsubishi Electric Research Laboratories.
Towards real-time camera based logos detection Mathieu Delalandre Laboratory of Computer Science, RFAI group, Tours city, France Osaka Prefecture Partnership.
Stereo Many slides adapted from Steve Seitz.
Acquiring 3D models of objects via a robotic stereo head David Virasinghe Department of Computer Science University of Adelaide Supervisors: Mike Brooks.
Spring 2015 CSc 83020: 3D Photography Prof. Ioannis Stamos Mondays 4:15 – 6:15
MIRALab Where Research means Creativity SVG Open 2005 University of Geneva 1 Converting 3D Facial Animation with Gouraud shaded SVG A method.
Lec 22: Stereo CS4670 / 5670: Computer Vision Kavita Bala.
MASKS © 2004 Invitation to 3D vision. MASKS © 2004 Invitation to 3D vision Lecture 1 Overview and Introduction.
112/5/ :54 Graphics II Image Based Rendering Session 11.
Yizhou Yu Texture-Mapping Real Scenes from Photographs Yizhou Yu Computer Science Division University of California at Berkeley Yizhou Yu Computer Science.
Visual Odometry David Nister, CVPR 2004
Visual Odometry for Ground Vehicle Applications David Nistér, Oleg Naroditsky, and James Bergen Sarnoff Corporation CN5300 Princeton, New Jersey
Image-Based Rendering Geometry and light interaction may be difficult and expensive to model –Think of how hard radiosity is –Imagine the complexity of.
Model Refinement from Planar Parallax Anthony DickRoberto Cipolla Department of Engineering University of Cambridge.
MASKS © 2004 Invitation to 3D vision. MASKS © 2004 Invitation to 3D vision Lecture 1 Overview and Introduction.
CIRP Annals - Manufacturing Technology 60 (2011) 1–4 Augmented assembly technologies based on 3D bare-hand interaction S.K. Ong (2)*, Z.B. Wang Mechanical.
MAV Optical Navigation Software System April 30, 2012 Tom Fritz, Pamela Warman, Richard Woodham Jr, Justin Clark, Andre DeRoux Sponsor: Dr. Adrian Lauf.
Rendering Pipeline Fall, 2015.
Jun Shimamura, Naokazu Yokoya, Haruo Takemura and Kazumasa Yamazawa
Paper – Stephen Se, David Lowe, Jim Little
A Forest of Sensors: Using adaptive tracking to classify and monitor activities in a site Eric Grimson AI Lab, Massachusetts Institute of Technology
Image Based Modeling and Rendering (PI: Malik)
Eric Grimson, Chris Stauffer,
Sensor Fusion Localization and Navigation for Visually Impaired People
Stereo vision Many slides adapted from Steve Seitz.
Closing Remarks.
Presentation transcript:

Tracking and Data Fusion for 4D Visualization Ulrich Neumann Suya You Computer Science Department Integrated Media Systems Center University of Southern California MURI Review June 2002

Combine all manner of 3D models, images, video, and data in a coherent dynamic visualization to support spatio-temporal data understanding, information extraction, and dynamic scene change detection Text Hyper-links Communications … LiDAR Laser Stereo … Satellite Aerial Ground … 3D modelVideo/image Dat a Information Fusion & Visualization Data Fusion Tracking Model refinement Needs: Research Goals

Robust tracking for outdoor unprepared environment –Portable hybrid tracking and data acquisition/fusion system –Natural feature tracking using vision sensors Fusion of 2D video/images and 3D model –LiDAR data tessellation and model reconstruction –Real-time video texture projection and visualization 6DOF Auto-calibration technology –Detect and calibrate scene features (points and lines) to refine models and aid in tracking We pursue basic algorithm research and testbed implementations that are feasible with current or near term technology Research Highlights

People with sensors (or unmanned sensors) moving in environment provide textures and data for visualizations… Where are they? Where are they looking? Need 6DOF pose tracking over wide area outdoors –Varying sensor data availability and data rates vision, GPS, inertial sensors –Varying certainty of measurements spatial and temporal noise and precision –Fusion models and algorithms underdetermined system needs constraints real-time acquisition and execution on portable systems Developed two tracking systems –Portable hybrid tracking and real-time data acquisition system –Natural feature tracking for computing motion of video sensor Tracking in Unprepared Environments

System configuration: RTK differential GPS (Ashtech Z-Sensor base/mobile) 3D inertial tracker (Intersense IS300) Real-time stereo head (SRI International) PIII 866Hz laptop DGPS receiver 3DOF Gyro sensor Stereo camera head Data fusion and storage Com1 Com2 Firewire Portable Acquisition System

Hybrid DGPS and Inertial sensor provide real-time 6DOF pose tracking High-resolution digital camera pairs are used to capture video streams for texture projection and façade reconstruction Complete self-contained system in a backpack Acquisition in real-time (all data streams are time- stamped and synchronized) Includes data capture and playback tools Real-Time Data Acquisition

Problem Most vision tracking methods require a priori knowledge about the environment Pre-calibrated landmarks Active sensors Scene models Active control or modification of an outdoor environment is unrealistic Our approach Detect and use naturally occurred features Robust tracking 1D (point) and 2D feature (region) SFM (structure from motion) from tracked features Neither camera ego-motion nor structure information is known Natural Feature Tracking Using Vision Sensor

Approach provides camera pose and structure estimates Related pose tracking can be directly used for augmented reality overlays Structure estimates allows continually tracking also can be used to improve/refine existing models Framework allows further sensor fusion (GPS, gyroscopes) for absolute pose reconstruction Video streams Pose & Structure Estimate Feature verification 2D Feature detection (new feature) 2D Feature tracking Structure Pose Feature list Natural Feature Tracking Using Vision Sensor

Natural Feature Tracking

Vision Tracking Used for AR Overlays

Imagine dozens of video streams from people, UAVs, and sensors distributed and moving through scene…. Use sensor models and 3D models of the scene to integrate video/image data from different sources Visualize the imagery in unique, innovative ways that maximize information extraction and comprehension Produce dynamic visualizations from arbitrary viewpoints - Static textures pre-compute mapping of images to models - dynamic projections onto models – like “slide projectors” Fusion of 2D video/image and 3D model

LiDAR provides 3D accurate (sub-meter) position samples and cm height accuracy - Use as base model/context for video visualization Raw LiDAR comes as a 3D point-cloud - Need tools for data resampling, tessellation, and model reconstruction 3D Model: LiDAR from flyover

Data tessellation –Data re-sampling (irregular sample cloud to regular grid) –Surface interpolation (hole filling) 3D models represented as triangle meshes - Easily converted to many other geometric representations - Supports many level-of-detail techniques - Easily add photometric information (texture projections) - Hardware acceleration for fast image rendering Use VRML as the standard model representation - supports web applications and open tool-base Data Tessellation and Model Reconstruction

LIDAR data acquired for USC campus Resampled range imageReconstructed 3D model Data Resampling and Model Reconstruction

 Texture Projection vs. Texture Map –dynamic vs. static –texture image and position both change each video frame Image/Video Texture Projection 3D modelimage texture

Dynamic view and placement control during visualization Update pose and “paint” the scene each frame HW texture transformation during rendering Real-time visualization with HW accelerator (Video) Virtual Texture Projector

Putting it all together… −Accurate 3D models −Accurate 3D sensor models −calibration and 6DOF tracking −Projection transformation computes texture mapping during rendering −Visibility and occlusion processing −multi-pass algorithm ensure that only visible surfaces are textured Dynamic Texture Projection

video texture projected on USC model (GPS/inertial tracking)

video texture projected on USC model (vision/GPS/inertial tracking)

video texture projected on USC LiDAR/model (vision/GPS/inertial)

Visualizations from arbitrary viewpoints User control of viewpoint as well as image inclusion (blending, and projection parameters) Multi-texture projectors simultaneously visualize many image sources projected on the same model Hardware acceleration with G-Force graphics card with use of pixel-shader features Visualization wall (8x10-foot tracked stereo) LiDAR/Projection Rendering System

Autocalibration computes 3D scene information during tracking –allows tracking in regions beyond models or landmarks –Provides the necessary scale factor data to create the 6th DOF that is lacking from vision-only tracking –Provides absolute pose data for stabilizing the multi- sensors data fusion –Provided estimated 3D information of structure features (point, line and edge) to improve/refine models 6 DOF Autocalibration

Camera pose and features are estimated simultaneously along the motion path –Point features (developed in past 2-3 years) –Line features (this past year) Pose estimate from 3D calibrated lines or points Autocalibration of 3D lines –Unique line representation (minimal) Four parameters N1, N2  (x, y, 1) for a given (R 1 T 1, R 2 T 2 ) to uniquely determine a 3D line L –EKF based estimator Update per feature or measurement Adapt to different sample rate 6 DOF Autocalibration

Simulation Results Simulation with 100 inch volume and 50 lines

Tracked line features are marked as green Red lines are the projections of auto- calibrated lines A virtual lamp and chair are inserted into the real scene based on the estimated camera pose Autocalibration Results

Automate steps in data capture and fusion for vision/gps/inertial tracking and visualization –do 8-12 captures around 3-4 buildings on campus –visualize streams simultaneously –correspondences from video to LiDAR/models edges, windows, … Model refinement (w/Cal, GT) –constrained autocalibration for estimating building features simplification and definition of edges, planar faces, windows, … Temporal data management (w/Cal, GT, UCSC, USyr) –accumulation of persistent textures –user management and temporal blending (update) Future Plans

 Benefits and capabilities  Real-time multi-source data fusion –Models, imagery, video, maps...  Enhanced data understanding –Dynamic controls of viewpoint as well as image inclusion –Rapid update to reflect most recent information –Highly interactive real-time perspective view capability  3D data editing using photogrammetric methods –Enables reuse of models and incremental refinement Dynamic Texture Projection

Aerial view of projected image texture (campus of Purdue University) Sensor Image plane View frustum Projection on LiDAR Data