Mirza Muhammad Waqar GEOREFERENCING OF IMAGES BY EXPLOITING GEOMETRIC DISTORTIONS IN STEREO IMAGES OF UK DMC 1 Final Defense

2  Advisor  Dr. Rafia Mumtaz (SEECS, NUST)  Guidance & Examination Committee  Dr. Ejaz Hussain (IGIS, NUST)  Dr. Rizwan Bulbul (IGIS, NUST)  Muhammad Hussan (IGIS, NUST)

Contents 3  Overview  Novel Contribution  Objectives & Scope  Research Novelty  Literature Review  Thermo elastic Model  Model Inversion  Results  Conclusions

 Georeferencing  It is the process of assigning geographic coordinates to a digital image.  It is a process for correcting spatial location and orientation of a satellite image  Types of Georeferencing  Direct Georeferencing  Indirect Georeferencing  Required for  Geospatial Analysis Change Detection Urban Planning  Map update Overview 4

 Traditional Method of Georeferencing  Require Ground Control Points (GCPs)  GCPs are acquired manually and hence an expensive task  Regions like deserts which lack salient features and have homogeneous texture, selection of GCP is difficult  A large number are required for complex terrain with varied surface elevation.  Accuracy of exterior orientation depends on the accuracy of GCP  Not suitable for push broom imagery as every scanned line possesses a different set of exterior orientation parameters  Proposed method is based on Direct Georeferencing  Does not require Ground Control Points (GCPs)  Utilizes Satellite position/velocity data Attitude data Sensor Configuration to determine the pixel’s geo location. 5

Objectives 6  The primary goal of this research is to provide accurate georeferenced imagery using no GCPs by mitigating thermo elastic effect.  This will be accomplished by the following objectives  Modeling thermo elastic effect as a transformation matrix  Model inversion in order to extract the thermal deformation from the image offsets  Find geodetic coordinates by correcting pointing error  Validate the complete model by testing it on UK-DMC imagery

Novel Contributions 7  Modeled the thermo elastic effect as transformation matrix by exploiting the inter image offsets present between the pair of images.  Inverted the model to extract the thermal deformation knowledge to remove the pointing error.  Developed a new direct georeferencing method capable of modeling the pointing errors.

UK DMC – Sensor Geometry Launch by Surrey Space Center, UK in 2003 operated by DMC international imaging  Sensor: Push broom  Made up of 6 CCD Channels  Making 2 banks of 3 (Port and Star-Board Array)  Separate by an angle β across track  Separate by an angle α along track  Both Port & Star-board Array have10000 linear CCD detectors  pixel image width with 500 pixels overlap  GSD = 32 m  Spectral Bands: Green, Red, NIR  Ground Swath = 640 km DMC Multi Spectral Imager (MSI) Star- Board Array Port Array 8

Direct Georeferencing  Direct Georeferencing  Transforms the image coordinate in camera frame to geodetic frame. Satellite onboard attitude data Satellite position and velocity data  One crucial point  Accuracy of estimated geo-locations is directly dependent on the accuracy of onboard sensor measurements  Main Advantage  Requires no GCPs  Becoming more robust and accurate every year with the ongoing development in GPS and inertial equipment.  Can be applied with aerial camera, hyperspectral scanner, Synthetic Aperture Radar, LIDAR 9

Thermo-elastic Effect  Due to thermo-elasticity  Satellite cools and heat up periodically which causes it to contract and expand.  This introduces changes in the relative orientations of the attitude sensors (e.g. star tracker) and the imager. 10 (Attitude of Imager) (From Attitude Sensor)

Literature Review on Thermo elastic Effect 11  In 2003, work on “ Active pixel array devices in space missions “ has been done and it has been found that  The X-ray Telescope for NASA’s Swift mission incorporates a Telescope Alignment Monitor (TAM) to measure thermo- elastic misalignments between the telescope and the spacecraft star tracker.  In 2005, research on “Pleiades-HR Image System Products, Quality And Geometric Accuracy” has been done.  It has been found that attitude sensors are mechanically fixed on the telescope support to minimized the thermo elastic distortions.

12  Radhadevi et al in “In-flight geometric calibration of fore and aft cameras of CARTOSAT-1”devised a method for in-flight geometric calibration for Cartosat-1  The objective of this study is to ensure the best absolute 3D pointing accuracy and relative location accuracy of the cameras.  It is concluded that accuracy of direct orientation observation could be brought down to better than 100 m with the inclusion of in-flight calibrated parameters in to the adjustment model. Literature Review on Thermo elastic Effect

13  In 2008, a research is conducted on, “Attitude Performance Requirements and Budgeting for RASAT Satellite”  In this study various sources of errors are identified for RASAT satellite. These includes Star Tracker Errors Controller Errors Thermo elastic Error  Finally these errors are combined together in an error budgeting tool for analysis. Literature Review on Thermo elastic Effect

Research Novelty  In direct geo-referencing the major source of error  Thermo-elastic effect Creates misalignment between the imaging sensor and the attitude sensor onboard. Measured and realized orientations will be different.  Mechanical design changes to reduce the distortion by mounting the imager/star-tracker assembly together on compliant mounts But this solution will be costly and require changes in the physical mounting of the sensors.  Towards such ends we proposed to model the thermo elastic effect as a transformation matrix by using the inter-image offsets present between the pair of images. 14 (Attitude of Imager) (From Attitude Sensor)

Modeling Thermo-elastic Effect  The thermal deformation could be a rotation, scaling or some sort shearing in the pixels. T = T rotation + T scale + T shear  Rotation It has the major effect on the accuracy of geo-locations. Small deviations from the true orientation cause a large displacement on the ground.  Scaling Due temperature changes the focal length of the imager contracts and expands. This effects the scaling in the height (Z-direction)  Shearing Non parallel projection of imager results in shearing of pixels. 15

Modeling Thermo-Elastic Effect as a Rotation  This can be determined by  Rotation of Imager  Determined by inter-image offsets present between stereo pair.  Mathematical model has been developed  The inter image offsets equations are the function of senor configuration angles and attitude components. 16 (From Attitude Sensor) (Attitude of Imager)

 Let the port and starboard pixel in body frame be and  Let T be the matrix that represents the thermo elastic deformation.  Let A be the attitude matrix  Next step is to find the equations for the inter-image offsets. 17 Modeling Thermo-Elastic Effect as a Rotation

 Column Shift  Determine the corresponding Pixels  Measure the distance of port and starboard pixel End of their respective array Difference in the distances gives the column shift  Row Shift  Time separation  Delay for the corresponding point to appear in the second image 18 Where Modeling Thermo-Elastic Effect as a Rotation

19 Direction in which satellite is moving ΔrΔr ΔcΔc Δr be the row shift Δ c be the column shift Star Board Array Port Array Overlapping region Max Row Shift Min Column Shift Min Row Shift Min Column Shift Modeling Thermo-Elastic Effect as a Rotation

Inter-Image Offsets 20 Row ShiftColumn Shift At Nominal Attitude and nominal thermo-elastic effect T =

Offset Modeling as Parabola & Straight Line 21  Offset Modeling  Column shift represents parabola  Row shift represents straight line  Column shift as Parabola  Express parabola parameters in terms of attitude components. H (peak value of parabola) x o (x-intercept for peak value of parabola) a (shape of parabola) Attitude matrix Where Thermo elastic matrix Parabola

Offset Modeling as Parabola & Straight Line 22  Row shift as straight line  Express slope m and intercept c in terms of attitude parameters Straight Line

Effect of Attitude on the Offset Parameters Image offset Paramet ers RollPitchYaw c m γpγp H

Model Inversion - Modeling Thermo elastic Effect as Rotation  Attitude is determined  Estimating the unknown components of attitude matrix by solving the offsets equations linearly. Use properties of attitude matrix  Mapping the estimated components to general attitude matrix  Determine the attitude by using the standard equations Properties of Attitude Matrix 24 Attitude matrix

Modeling Thermo-elastic Effect as Scaling  Temperature changes effects the focal length of the imager  Introduces height changes along the Z axis  Change in scale (along z-axis) effects ground separation of the port and starboard imaging planes.  This will effect the row shift magnitude.  Hence row shift constant equation will be used to extract 25

26  The cofactor of the T scaling matrix can be written as,  Where  The cofactor terms appear in expression of row and column offset parameters. Modeling Thermo-elastic Effect as Scaling

 Column offset Equations  Where 27

Modeling Thermo-elastic Effect as Scaling  Row Offset Equations Similarly using cofactors, the expression for b 1, b 2, b 3, b 4, b 5, b 6, b 7 and b 8 can be reduced to Using above values, the row shift parameters can be written as 28

Simulations – Sensitivity of Scaling w.r.t offset parameters 29

Scale Extraction from row offset  With scaling matrix the row offset equation will take the form  From the above equation can be found as 30

Modeling Thermo-elastic Effect as Shearing  Shearing slides one edge of an image along the X or Y axis, creating a parallelogram.  The amount of the shear is controlled by a shear angle 31

32  The cofactors of shear matrix can be written as  The cofactor terms appear in the expression of row and column offset parameters.  Hence Modeling Thermo-elastic Effect as Shearing

33  Column Offsets Equations  Row Offset Equations Modeling Thermo-elastic Effect as Shearing Where

Simulations – Sensitivity of Shearing w.r.t Offset Parameters 34

Shear Extraction from column offset  Shear pushes the pixels across the track thus effecting the column offset. Therefore column offset will be used to find the shear factor where 35

Thermo-elastic Matrix  Thermo-elastic matrix will be the sum of rotation, scale and shear extracted from the image offsets  Now inserting this matrix between the body and orbital frame will mitigate the misalignment between the imager and the attitude sensor. 36

Direct Georeferencing using onboard Attitude Sensor 37  Study Area: UK  Date: May 23, 2004  Rows:  Columns:  GPS Positions  X = m  Y = m  Z = m  Attitude Values  Roll = 32 centidegree  Pitch = -25 centidegree  Yaw = 47 centidegree

Attitude & GPS data Provided by SSTL 38 GPS Data Attitude Data

Direct Georeferencing using exterior orientation data  By using the onboard exterior orientation data, the accuracy of Km has been achieved. 39

40 Row ShiftColumn Shift Offset estimation using Window based Scheme

Measured Image offsets over the entire Scene length 41

Stereo Angle Estimation 42  Prior to model inversion, the sensor configuration angles of UK DMC must be determined which are α and β.  The β is being determined by SpaceMetric  β = o.  However the magnitude of α can be found from  Rearranging equation of row offset’s constant  For UK DMC band 3 image pair, the value of α was found to be

Direct Georeferencing using Imager Attitude  By using imager attitude, the accuracy of 7- 10Km has been achieved. 43

Thermo elastic Rotation 44

Direct Georeferencing using thermo elastic Rotation  By applying the thermo elastic  Rotation  The accuracy of 1- 5Km has been achieved. 45

Thermo elastic Scale 46

Direct Georeferencing using thermo elastic Rotation and Scale  By applying the thermo elastic  Rotation  Scale  The accuracy of 1- 3Km has been achieved. 47

Thermo elastic Shear 48

Georeferencing using thermo elastic Rotation, Scale and Shear  By applying the thermo elastic  Rotation  Scale  Shear  The accuracy of Km has been achieved. 49

Potential Benefits 50  No GCPs required:  The major benefit of this approach is that it does not based on GCPs. Collection of GCPs is time consuming and expensive Difficult to collect GCPs having homogeneous texture Large number of GCPs are required for complex terrain Unsuitable for pushbroom imagery  Thermo elastic correction  To date, modeling of thermo elastic effect as a transformation matrix has not been explored.  The thermo elastic effect is deemed as major source of error in this work  Not only provide cost effective solution but also mitigate pointing error  No additional hardware  Low cost and low mass system for obtaining geolocation  Can work with conventional EO cameras with no additional hardware  Frequent Attitude Observations  Due to small baseline or angular separation between the sensors, the registration time for the corresponding pixels is very small which results in rapid attitude observations.

Conclusions  A novel method for measuring geolocation without GCPs has been developed.  The error in geolocations estimate has been addressed by modeling thermo elastic effect as a deformation matrix.  Mathematical model has been developed by exploiting inter image offset between pair of images.  The possible deformations that has been considered are  Rotation, Scaling & Shearing  These deformations have been modeled individually and has been summed at the end to represent the entire thermo elastic deformation.  The row and column offset parameters have been simulated individually to determine  The best candidate for extracting these individual deformation.  Developed mathematical model has been validated on UK DMC imagery.  Accuracy of 1-1.5Km is achieved using newly developed georeferencing method. 51

Future Work 52  Developing Generic Model for thermo elastic effect  Can be applied on other celestial bodies (e.g. Moon)

53 1. Rafia, M, P.L.Palmer,. Waqar, M.M. Georeferencing of images without Ground Control Points by Exploiting Geometric Distortions in Stereo Earth Images. Journal of Remote sensing of the Environment (Manuscript Submitted) Impact Factor ~ Waqar, M.M., Johum, F.M., Rafia, M., Ejaz, H. (2012) Development of New Indices for extraction of Built-up area & Bare Soil from Landsat Data. Journal of Geophysics & Remote Sensing. (Manuscript Accepted). 3. Waqar, M.M., Rafia, M., Sufyan, N., Mustafa, M. (2012) Accuracy Assessment of Geo-locations using Multi-Resolution Interpolated DEMs th International Conference on Digital Image Processing. Kuala Lumpur, Malaysia. Published by SPIE. Research Publications

Acknowledgement 54

55 Questions - Discussion