V. Martin et al. 1 (18) WFDPVA, ENEA Frascati 28/03/12 3D Scene Calibration for Infrared Image Analysis V. Martin, V. Gervaise, V. Moncada, M.H. Aumeunier, M. Firdaouss, J.M. Travere (CEA) S. Devaux (IPP), G. Arnoux (CCFE) and JET-EFDA contributors Workshop on Fusion Data Processing Validation and Analysis, ENEA Frascati, March 2012

V. Martin et al. 2 (18) WFDPVA, ENEA Frascati 28/03/12 Issue: a complex thermal scene 1.Wide angle views with high geometrical effects: depth of field and curvature 2.Many metallic materials (Be, W) with different and changing optical (reflectance) and thermal (emissivity) properties Objective: Match each pixel with the 3D scene model of in-vessel components for: 1.getting the real geometry of the viewed objects 2.reliable linking between viewed objects and their related properties Applications 1.Image processing (event characterization) 2.IR data calibration: T surf = f(material emissivity) 3D IR Scene Calibration JET #81313 KL7 (images in DL) Bulk W Bulk Be Be coated linconel W coated CFC Bulk Be W coated CFC

V. Martin et al. 3 (18) WFDPVA, ENEA Frascati 28/03/12 IR Data Calibration Methodology Image Stabilization 2D/3D Scene Model Mapping Image Processing Image Correction Camera NUC Dead pixel Map Reference image2D/3D scene models Knowledge base of the thermal scene Calibration chain Registered & Calibrated Image

V. Martin et al. 4 (18) WFDPVA, ENEA Frascati 28/03/12 Illustration of Motion in Images Camera vibrations lead to misalignments of ROIs (PFC RT protection) = false alarms or worth missed alarms Image stabilization is a mandatory step for analysis based on ΔT(t) estimation (e.g. heat flux computation)

V. Martin et al. 5 (18) WFDPVA, ENEA Frascati 28/03/12 Important factors for method selection Deformation type: planar (homothety), non-planar Target application: real-time processing, off-line analysis Data quality and variability: noise level, pixel intensity changes, image entropy Aimed precision level: pixel, sub-pixel Applications in tokamaks (non-exhaustive list) Image Stabilization Motion causeDeformation type Target applicationAimed precision level Difficulty Tore Supra RF antenna IR 1. antenna positions 2. camera vibrations 1. non-planar 2. planar (Δx, Δy) RT PFC protectionpixel levelchanges of scene appearance JET KL7 wide-angle IR camera vibrationsplanar (Δx, Δy)offline analysis (heat load, disruptions…) sub-pixellow image entropy JET KL9 divertor tiles IR sensor affected by magnetic field planar (Δx, Δy)offline analysis (heat load, disruptions…) sub-pixellow resolution, slow motion, aliasing JET KL8 fast visible intensifier affected by magnetic field planar (Δx, Δy, Ө)physics (transient events) pixel levelnoisy, low image entropy, aliasing

V. Martin et al. 6 (18) WFDPVA, ENEA Frascati 28/03/12 Classical Methodology 1.Feature Detection Local descriptors: Harris corners, MSER, codebooks, Gabor wavelets (see Craciunescu talk), SIFT, SURF, FAST… Global descriptors: Tsallis entropy (see Murari talk), edge detectors… Fourier analysis: spectral magnitude & phase, pixel gradients, log-polar mapping… 2.Feature Matching Spatial cross-correlation techniques: normalized cross-correlation, Hausdorff distance… Fourier domain: normalized cross-spectrum and its extensions 3.Transform Model Estimation Shape preserving mapping (rotation, translation and scaling only) Elastic mapping: warping techniques… 4.Image transformation 2D Interpolation: nearest neighboor, bilinear, bicubic… Image Stabilization See Zitovas survey, Image and Vision Computing, vol. 21(2003), pp

V. Martin et al. 7 (18) WFDPVA, ENEA Frascati 28/03/12 Proposed Algorithm 1. Masked FFT-based image registration [1] Deterministic computing time Accelerating hardware compatible algorithm (e.g. FFT on GPU) real time applications Local analysis with dynamic intensity-based pixel masking (e.g. mask the divertor bright region) 2. with sub-pixel precision [2] Slow drift compensation 3. and dynamic update of the reference image Robust to image intensity changes (context awareness) Evaluation of registration quality over time [1] D. Padfield, IEEE CVPR10, pp , 2010 [2] M. Guizar-Sicairos et al., Opt. Lett., vol. 33, no. 2, pp , 2008

V. Martin et al. 8 (18) WFDPVA, ENEA Frascati 28/03/12 Principle of Fourier-based Correlation Let I ref a reference image, I t an image at time t and DFT the Discrete 2D Fourier transform such as I t ( x, y ) = I ref ( x-x 0, y-y 0 ) NCC is the Normalized Cross Correlation figure (image) and the position of the peak gives the coordinates of the translation ( x 0, y 0 ) I ref ItIt max (NCC( I ref, I t )) NCC( I ref, I t )

V. Martin et al. 9 (18) WFDPVA, ENEA Frascati 28/03/12 Sub-pixel Precision Up-sample k times the DFT of NCC (trigonometric interpolation) : The peak coordinates ( x 0, y 0 ) give F the translation with 1/k pixel of precision:

V. Martin et al. 10 (18) WFDPVA, ENEA Frascati 28/03/12 Reference Image Updating Use the NCC value to trigger the update of I ref : High NCC confidence, no I ref update needed T min T max NCC failed, no I ref update update I ref

V. Martin et al. 11 (18) WFDPVA, ENEA Frascati 28/03/12 Results JET #81313, KL7, 480x512 pixels, 50 Hz, 251 frames k=1/4 pixel

V. Martin et al. 12 (18) WFDPVA, ENEA Frascati 28/03/12 Results k=1/2 pixel JET #80827, KL7, 128x256 pixels, 540 Hz, frames

V. Martin et al. 13 (18) WFDPVA, ENEA Frascati 28/03/12 Results k=1/16 pixel JET #82278, KL9B, 32x96 pixels, 6 kHz, 4828 frames

V. Martin et al. 14 (18) WFDPVA, ENEA Frascati 28/03/12 Computational Performance Influence of image size and precision level on frame rate GPU NVidia TM GTX 580 : 512 processor core units 32 threads per processor Up to 1 Tflops! 256x256, k =1/4 700 fps GPU/CPU performance gain Number of CPU cores GPU is at least 15 times faster than CPU (same generation) FFT 256x256 pixels: CPU v/s GPU performance

V. Martin et al. 15 (18) WFDPVA, ENEA Frascati 28/03/12 From 2D to 3D Challenge –transform pixel coordinates into machine coordinates: (x, y) (r, θ, φ) Method –Ray-tracing method from simplified CAD files

V. Martin et al. 16 (18) WFDPVA, ENEA Frascati 28/03/12 3D Scene Model for Image Processing S. Palazzo, A. Murari et al., RSI 81, , 2010 V. Martin et al. Blobs 1 & 2 must not be merged! mm Z Map (depth) 2 1 2m 7m

V. Martin et al. 17 (18) WFDPVA, ENEA Frascati 28/03/12 Plasma ImagiNg data Understanding Platform (PINUP) IR Data Calibration Methodology An integrated software for IR data registration & calibration Image Stabilization 2D/3D Scene Model Mapping Image Processing Image Correction Camera NUC Dead pixel Map Reference image2D/3D scene models Knowledge base of the thermal scene Registered & Calibrated Image Used for PFC protection Used for temperature evaluation Used for event triggering Set sub-pixel precision factor Set mask Load/save translations

V. Martin et al. 18 (18) WFDPVA, ENEA Frascati 28/03/12 Conclusion Summary –Complex IR scenes require a new calibration approach including image stabilization, 3D mapping for reliable data analysis. –A robust and fast image stabilization algorithm with sub-pixel precision has been proposed. –A first integration of 3D model for IR data analysis has been performed. –An integrated software (PINUP) is available for users Outlook –Test of the stabilization algorithm on visible imaging data (JET KL8) with rotation compensation –Full integration of 3D scene models into PINUP –Improvement of image processing algorithms (e.g. hot spot detection) with 3D information