1. An Efficient Automatic Geo-registration Technique for High Resolution Spaceborne SAR Image Fusion IGARSS 2011 28/July 2011 Woo-Kyung Lee and A.R. Kim Korea Aerospace University wklee@kau.ac.kr

2. Motivation Simple approach to the SAR image registration and fusion - As the resolution level improves, * the unique feature of the radar imaging becomes prominent and the task of image fusion with optical image becomes complicated, * the number of pixels increases and the amount of resources for calculation such as memory and time consumption escalates exponentially. To relieve the burden of the work and make it done in real time.Efficient image matching in both rural and urban regions One click Let the machine do the rest of the job in almost real time

3. SAR Sensor and Geometric Characteristic Side-looking Observation -Image processing depends on the surface characteristics and structures -Radar images suffer from unrealistic distortions -Non-linear distortions along range, Shortening from shadow region System error - Inaccurate Doppler parameter estimation leads to geocoding errors - Unstability in internal system clock and orbit parameters Image acquisition SAR vs. optics images

4. Error correction method Effect of Error Error Source SAR Sensor and Geometric Characteristic Source of SAR geocoding errors Platform - Image Orientation Error - Squint Angle - Doppler Centroid Platform - Image Orientation Error - Squint Angle - Doppler Centroid Earth - Earth Rotation - Side-looking - Target Height Earth - Earth Rotation - Side-looking - Target Height SAR sensor - Electronic Time Delay - Slant Range Error - Incidence Angle Estimation - PRF Fluctuation SAR sensor - Electronic Time Delay - Slant Range Error - Incidence Angle Estimation - PRF Fluctuation Platform - Inclination Angle - Yaw Angel Error - Pitch Angle Error Platform - Inclination Angle - Yaw Angel Error - Pitch Angle Error - Geometric Calibration - Deskew - Ground Projection - Image Rotation - Terrain Correction - Geometric Calibration - Deskew - Ground Projection - Image Rotation - Terrain Correction Earth - Azimuth Skew - Range Non-Linearity - Foreshortening, Layover, Shadowing Earth - Azimuth Skew - Range Non-Linearity - Foreshortening, Layover, Shadowing Effect of Error - Range Location - Range Scale - Azimuth Scale Effect of Error - Range Location - Range Scale - Azimuth Scale

5. SAR Sensor and Geometric Characteristic Geometrical distortions in SAR images - Mismatch between SAR and Optical images OpticsSAR (a) Azimuth Distortion (b) Non linear Range Error(c) Deskew

6. SAR Geo-correction with satellite internal data Ground projection example -Slant range function has non linear scale variations Azimuth Range Sland based Ground Based Slant range image Ground range image Reference image (EO image) -Discrepancy exist compared with the reference image - Distortion between SAR and EO are case-sensitive

7. GCP based geometry registration Basic Principle - A reference image is chosen to be used for geometric correction or fusion - Multiple GCP(Ground Center Point)s are selected and directly applied to individual position error calculation and correction i - Based on the selected GCPs, image transforml function is characterized that best describes the discrepancy between the images - Original image is re-sampled and re-arranged by the generated transform function - In general, distinctive features such as road, building, bridges, reflectors are chosen that are easily discriminated for convenience - Manually? Or Automatically?? Who will chose what points?? Choice of GCP - To perform geometrical calibration and restore distortion, the GCPs in the SAR images would be re-arranged into the true ground positions - It becomes most essential to pick up the best candidates of GCPs

8. Selection of GCPs within SAR images Difficulty of GCP choice - Speckle noise inherence in SAR image makes it difficult to guarantee to pinpoint precise positions that correspond to the reference points. - A human work of manual GCP selection is never reliable - The number of available points are case-sensitive and still limited by the existence of the distinctive features - The precision of the GCP location is not fully guaranteed and the error variance may increase in coarse resolution images. Optical image GCP SAR image GCP

9. Methodology SURF algorithm -Speeded-Up Robust Features (SURF) is known to have highly robust performance -Scale, rotation and illumination-invariant feature descriptor. - Adaptive for noisy environment and mutll-scale images - Only summing operation is involved in producing integral image to match the scale and calculation is speeded up Selection of GCP and matching - Selection of matching points(GCP) are performed using feature vectors described by Hessian Matrix -The size of the constructed Hessian matix can be varied and can be increased to multiple dimensions as desired -The number of dimensions is limited by the complexity, time consumption and precision of the image matching.- case sensitive - Parameters required for the decision algorithms is set intuitively - This work is motivated to find the decision parameters automatically compromising the performance and the complexity

10. Block diagram for GCP pair selection

11. Integral image generation -For the given image, an integral set of points are summed together -The size is variable depending on the scale and complexity of the image - Simple summations of intensity levels are performed over two dimensional domain : A +B + C + D

12. GCP candidate generation - The Hessian matrix, corresponding to each pixel, is simplified by summation with adjacent cells - The image scale is varied and the simplified Hessian matrix is obtained for each scale space - Harr-wavelet responses are calculated and the feature descriptor is generated - The polarity of the image intensity variation is investigated and stored X direction Y direction X, Y direction Harr wavelet

13. GCP selection Principle - Find a pair of feature descriptors that best fit to each other - One-by-one comparison is straightforward but time-consuming and does not guarantee successful matching due to increased ambiguity - Construct a look-up table for the feature descriptor - Each feature descriptor is indexed depending on their size, variation rate, orientation - For a given GCP, a “search process” is performed within other look-up table generated from reference image and the best matching pair is selected - Nearest neighbor search is adopted to find the correct matching pair

14. - Among the selected GCPs, the Euclidian distance (x, y) – (x’, y’) are estimated to find the nearest points with similar feature descriptor -The rates of intensity variations along orientations (denoted as A and B) are considered as weighting factors - Distance multiplication is performed -The number of orientation can be increased in order to reduce ambiguity and avoid wrong decision. - Appropriate threshold level is required to compare with the distance multiplication and make a decision -The GCP match is confirmed when the distance multiplication is less than the threshold level - Image Projective Transform function is deduced from the matching GCPs GCP pair selection Define threshold level

15. Overall procedure diagram for image matching GCP generationGCP selectionTransform function

16. GCP extraction demonstration GCP extraction from SAR images (a) Stripmap image(b) Scan image -ScanSAR image is exposed to higher noise level and composed of extended resolution cells. -GCP candidates are extracted from both images using the same Hessian matrix structure - The number of GCP points appear to be close to each other despite the gap in the image quality GCP number SURF a1870 b1667

17. Geo-registration demonstration -Original SAR image is corrected using GCP matching and transformation -Strip mode SAR image over Vancouver, Canada is geo-registered using the reference image in Radarsat-1 SSG format - The threshold level is set to be zero for convenience Raw Reference GCP # vs. Threshold levelTime consumption vs Th. Level GCP selection for raw image GCP selection for reference image 881 557 912 544 GCP selection GCP # vs. Threshold levelTime consumption vs Th. Level

18. Image Matching Demonstration -Original image is geo-corrected Fusion image

19. Measure of registration errors -RMSE(Root Mean Square Error) is calculated for the selected GCPs CorrectedReference Displacement Error in pixel xy RMSE (x, y) 0.630.8 RMSE (average)1.02 The average deviation is about one point pixel size

20. Application to higher resolution images -Reference image of TerraSAR in EEC format, Toronto, - The number of GCP increases consistently when the level of correlation between the two images are high Original Reference GCP selection 1595 252 2680 404 1.72 0.81 3.21 1.47 GCP variation rate As the similarity of the images are high, the GCP increases consistently as the “Threshold Level” decreases

21. After fusion Image Matching Demonstration

22. Mismatch Error Estimate The average position error is less than one pixel The performance of the matched GCP selection is affected by the image resolution Mismatch error is reduced as the image resolution improves CorrectedReference Displacement Error in pixel xy RMSE (x, y) 0.70.32 RMSE (average)0.77

23. Image fusion of SAR over EO Number of GCPs a375 b1326 (a) JERS SAR image(b) LANDSAT EO image - This case is where SAR image constitute a small portion of the EO image -GCPs from the two images are distinguished - The matching GCPs are easily identified by the nearest search algorithm

24. Automated threshold level selection - Threshold variance Threshold 500, Matching image (14 points) Threshold 350, Matching image (15 points)Threshold 200, Matching image (15 points) There exist a turning point, where further reduction of the threshold level stops affecting the number of GCP matching points Computer traces the variation of the available GCP matches and find the turning point

25. GCP matching and Image transformation -Matching GCP selection process stops automatically and image transform function is obtained from the selected points Feature points extraction Source imageReference image xyxy 1398874247 11914335937 45917594203 254232137234 350268365366 509294514425 192402231209 503160137234 23622194203 362272390386 266224173218 280283177372 349250365329 44029687391 383237467270 Transform equation  15 points extraction Find equation

26. Result of the geo-registration Overlaid image -RMSE is affected by the resolution discrepancy and inherent image property. - Here it is given as 1.26 pixel

27. Automated Geo-registration software -Usually, the threshold level is manually set-up by user looking into the complexity of the images and resultant fusion performance - This procedure is replaced by compute search algorithm, where the threshold level is traced to find the turning point - Total elapsed time is within several minutes and will be further reduced by adaptive search algorithm Original ReferenceCorrected GCP selection and matching Fusion

28. Performance analysis vs. Resolution - The number of total GCP is not affected by modes(Stripmap and Scan) (a) Stripmap image(b) Scan image GCP number SURF a1870 b1667 - However, the RMSE is measured as 0.94 for ScanSAR mode while it is 0.34 for stripmap mode - It appears that the performance improves as the resolution improves

29. Insufficient information for SAR geometry Limited information -Internal data within SAR instrument fails to retrieve shadow region -There is non-linear discrepancy between slant and ground ranges -Generate Errors in geometrical coordinate -Need external references to retrieve broken information and correct errors in ground range allocations - foreshortening, layover, shadowing Foreshortening Layover Shadowing

30. Impact of the ground characteristics - Diverse ground geometry becomes a source of matching errors - Mountain areas are severely distorted from the EO case - Need to adopt separate transform functions within the image Coast line area Mountain area After correction Coast line fusion Mountain area fusion After correction

31. Matching Performance Comparison - Image distortion is not compensated for by the simple GCP transformation - Need to divide blocks and adopt modified transform functions separately Coast line Error XY RMSE (x, y) 0.330.26 RMSE0.42 Mountain Error XY RMSE (x, y) 1.461.73 RMSE2.27

32. Modified Transform functions - Image is divided into blocks according to the geographical properties Mountain Area Errors xy RMSE (x, y) 1.351.2 RMSE1.8 Average 1.8 RMSE error Application of separate transform function leads to the reduction of RMSE Urban Area Errors xy RMSE (x, y) 0.60.3 RMSE0.67 Average 0.67 RMSE Error

33. Summary - SURF provides an efficient tool to perform SAR image geo-registration - A choice of threshold level is required to perform efficient of GCP matching and it can be automated by tracing its variation curve - The image matching algorithm works with various SAR and EO images and the average RMSE is measured to be around 1 pixel. - Image blocks containing mountain areas need separate GCP matching and transform function to compensate for image distortion - Need to develop optimized transfer function for different type of ground characteristics - Indexing of GCP is performed based on their intensity and variation vector -With the introduction of adaptive indexing table for selected GCP, the automated image matching is expected to be carried out in real time - ConclusionFurther work