Online Tracking of Outdoor Lighting Variations for Augmented Reality with Moving Cameras Yanli Liu 1,2 and Xavier Granier 2,3,4 1: College of Computer Science, Sichuan University, P.R.China 2: INRIA Bordeaux Sud-Ouest, France 3:LP2N (CNRS, Univ. Bordeaux, Institut d'Optique) 4:LaBRI (CNRS, University of Bordeaux)

20 12  Augmented reality mobility Motivation

20 12 TTwo consistency Geometric consistency DDevices CCamera position GPS, UWB, Omnisense WiFi, cell information CCamera pose Linear accelerometers TTracking via computer vision [[Cornelis et al. LNCS 2001, zhang et al. CVPR 2007, Xu et al. image and Vision Computing 2008] Illumination consistency ooutdoor lighting is largely dependent on weather and time

20 12  Two problems Online process  first step toward real-time solutions Moving viewpoints  Handhold camera jitter

20 12 Previous Work  Markers or lighting probes [Debevec Siggraph’ 98, Agusanto ISMAR’03, Kanbara ICPR’04, Hensley I3D’07] too dense sampling our method does not require any supplemental devices Debevec Siggraph’ 98

20 12 Previous Work  Three components of shading BRDF geometry lighting  Fix other one or two components [Wang PG’02, Li ICCV’03, Hara PAMI’05, Andersen ICPR’06, Sun ICCV’09] 3D reconstruction controlled environment (indoor or lab) [Wang PG’02] original image rendered image

20 12 Previous Work  Time-lapse outdoor video analysis [Sunkavalli Siggraph’07, Sunkavalli CVPR 08] take whole video sequence as input Post-processing [Sunkavalli Siggraph’07]

20 12 Previous Work  Learning based outdoor illumination estimation [Liu TVC’09, Liu CAVW’10, Xing C&G’11] offline stage learning fixed viewpoint Liu CAVW’10 moving viewpoints

20 12 Our Method  Key ideas Tracking illumination variation by tracking feature points Feature points tracking is error prone. Select reliable feature points using global illumination constraint and spatial-temporal coherency.

20 12  Outdoor lighting [Sunkavalli SIG’07, Sunkavalli CVPR 08, Madsen InTech 2010] the sunlight  directional light  colored intensity  sun direction the skylight  ambient light  colored intensity Illumination and BRDF model

20 12  Neutral reflection model [Lee PAMI’90, Montoliu LNCS’05, Eibenberger ICIP 2010, ICCV 2011] the color of the specular reflection is the same as the color of the incident lighting.  Phong-like model

20 12 System Initialization Tracking illumination variation by tracking feature points  3D geometry vs normals  planar feature points KLT feature-points clustered feature-points first frame plane segmentation [Hoiem IJCV’07] mean-shift color segmentation threshold-based Shadow detection

20 12 System Initialization  BRDF initialization pixels difference at in sun lit regions depend on specular parameters and : Assuming piecewise constant, and Spatially varying diffuse

20 12  Energy function Outdoor lighting is nearly constant during time intervals less than 1/5 second. control the smooth degree of skylight Alignment-based weight Tracking Lighting Variation with Reliable Feature Points

20 12 Tracking Reliable Features and Their Attributes  Feature points labeling Three attributes:  Normal (plane, homography matrix)  BRDF parameters  Shadow situation Spatial & temporal coherency

20 12 Tracking Reliable Features and Their Attributes  Feature points labeling t -1 t paired point is labeled in current point is not in shadow compute lighting

20 12 Results and Discussion  Quantitative results PC: Intel i7 2.67GHz and 6GB RAM MATLAB Video resolution Average fps and average number of feature points estimated on 1,000 frames

20 12 Results and discussion  Quantitative results Average percentage of different steps in total computational cost

20 12 Results and Discussion  Visual results Building scene Wall scene

20 12 Conclusion  Fully image-based pipeline online tracking of lighting variations of outdoor videos.  Manages lighting changes and misalignment of feature points  Ensure a stable estimation on a sparse set feature points.

20 12 Limitations and Future Work  Rough shadow detection 3D reconstruction vs shadow detection Sun-lit features  Initialization automatic initialization: easy but may fail in some cases manual initialization: may be tedious for a non-expert user. Semi-assisted initialization

20 12 Limitations and Future Work  Tracking independently on R, G, and B channels priori model of outdoor illumination color or spectra difficult to optimization  The first step of a long march to a seamless and real-time AR solution for videos with moving viewpoints.

20 12 Thanks for your attention! Questions?