Dorota A. Grejner-Brzezinska Multi-sensor Integration and Calibration Aspects Dorota A. Grejner-Brzezinska Civil and Environmental Engineering and Geodetic Science The Ohio State University 470 Hitchcock Hall Columbus, OH 43210 Tel. (614) 292-8787 E-mail: dorota@cfm.ohio-state.edu OSU

Presentation Outline Multi-sensor system AIMS Calibration experiences Camera calibration Lever arm calibration OTF INS calibration (GPS) Boresighting misalignment (AT) Summary, outlook

Multi-sensor System Kinematic platform, upon which multiple sensors have been integrated and synchronized to a common time base Provide three-dimensional near-continuous positioning of both the platform and simultaneously collected geo-spatial data In general direct georeferencing is used In principle, no external information, such as ground control, is needed except for the GPS base station Calibration of the components and the link between components -- a challenging task

Multi-sensor System: Objectives Automation of the map making process: the ultimate goal of digital photogrammetry Sensors based on different physical principles record complementary and often redundant information, leading ultimately to a more consistent scene description Offer a feasibility of automation of the photogrammetric tasks

Multi-sensor Systems: Design Level Specifications of the accuracy requirements Sensor specifications Architecture of the sensor mount Time synchronization circuit (crucial) Most challenging: Algorithms for sensor calibration Algorithms for data processing Methods for testing the performance

Multi-sensor Systems Variety of new spatial data acquisition sensors CCD-based cameras LIDAR (light detection and ranging) multi/hyper-spectral sensors SAR/IFSAR Optimal sensor fusion more consistent scene description complementary information redundant information

Direct georeferencing (direct platform orientation DPO) Geometric data fusion  time-space registration (georeferencing) Basis for higher level fusion (multi-sensor imaging systems) GPS/INS integration complementary and competitive data high accuracy (5-20 cm, 10-30 arcsec) high reliability fault-resistant cost effective (less ground control) mandatory for new spatial data sensors (LIDAR, SAR, multi/hyperspectral) experimental systems (University of Calgary, OSU AIMS) commercial (Applanix) GPS multi-antenna systems for less demanding applications

Effects of Errors in Direct georeferencing on the 3D Ground Point Positioning Depend on: Quality of GPS/INS error modeling Quality of boresight transformation Quality of GPS/INS lever arm estimation Quality of individual sensor calibration Rigidity of the mount (platform)

3D Ground Point Positioning Error Resulting from Errors in the Exterior Orientation: Simulation

3D Ground Point Error as a Function of Increasing Errors in Attitude for Variable Error Levels in Exposure Center Location (300 m altitude )

3D Ground Point Error as a Function of Increasing Errors in Exposure Center Location for Variable Error Levels in Attitude (300 m altitude )

Calibration of a Multi-sensor System Based on AIMS GPS/INS/CCD GPS Antenna Imaging PC GPS Base Station BigShot™ Hasselblad Camera Trimble 4000SSI LN-100 INS/GPS PC

OSU Center for Mapping AIMS Over 20 airborne tests Several land-based tests Typical standard deviations (covariance analysis) position 2-4 cm attitude 5-6 arcsec for pitch and roll 7-10 arcses for heading Typical fit to ground truth 2-20 cm for flight altitude ~ 300m 0.2-3 cm for land-based applications (10-20 m object distance) Calibration – crucial component

System Calibration Concept GPS Rover OTF GPS Error Estimation and Ambiguity Resolution Lever Arm Calibration Camera Calibration CCD INS INS OTF Calibration Sensor Mount Boresight Misalignment GPS Base

Hardware Configuration for Land-based Tests Digital camera GPS antenna LN 100

Integrated System Calibration Camera calibration GPS/INS lever arm calibration Boresight calibration between the camera body and the INS body frames linear offsets and the rotation matrix determined from the imagery containing images of the ground control points

Integrated System Calibration Camera calibration focal length, principle point coordinates, and radial distortions performed at the indoor test range in-flight calibration (limited) USGS polynomial model (3+6) especially important for non-metric digital cameras must to be repeated any time after the CCD was detached

Lever Arm Test (2)

Radial Symmetric Distortion Surface Carl Zeiss Distagon 4/50 mm microns

Radial Distortion Difference Between Two Calibrations microns

50-mm lens equipped 4K by 4K CCD frame Camera Calibration

CCD Camera Calibration Experiences Recalibration of AIMS™ digital camera Insignificant changes in distortion surfaces Linear parameters vary (especially principal point coordinates) No check on changing radiometric behavior Dust accumulates (factory cleaning) System performance unaffected

Lever Arm Determination Linear offsets from GPS phase center to INS body frame center (usually pre-surveyed) Have to be known with sufficient accuracy especially important for embedded systems where INS directly aids the carrier-phase-tracking loop Could be determined by the integrated filter speed of estimation depends on the vehicle dynamics

Calibration of the GPS Lever Arm Errors GPS Lever Arm Error Estimation GPS Lever Arm Error Estimation 10 4 5 2 Bias Estimate (x-axis, cm) Bias Estimate (z-axis, cm) -5 -2 -10 -4 500 1000 1500 500 1000 1500 time(s) time(s) Standard Deviation Standard Deviation 10 10 8 8 6 6 Standard Deviation (x-axis, cm) Standard Deviation (z-axis, cm)) 4 4 2 2 500 1000 1500 500 1000 1500 time(s) time(s)

Effects of GPS Lever Arm Errors DD Carrier Phase Residuals: level arm errors not calibrated DD Carrier Phase Residuals: level arm errors estimated 20 20 15 15 10 10 5 5 Residual (cm) Residual (cm) -5 -5 -10 -10 -15 -15 -20 -20 200 400 600 800 1000 1200 1400 1600 200 400 600 800 1000 1200 1400 1600 time(s) time(s)

Lever Arm Test Lever arm offsets pre-surveyed: 0, 0.95, -0.04 m distorted: 0, 0.50, 0 m INS x-axis oriented towards south during the initial 50-sec stationary period 200-sec vehicle maneuvering

Van Trajectory start

Difference between solutions with correct and incorrect lever arm components Motion starts

Difference between solutions with correct and incorrect lever arm components Motion starts

Difference between solutions with correct and incorrect lever arm components (no initial maneuvering)

Difference between solutions with correct and incorrect lever arm components (no initial maneuvering)

Camera/INS Boresight Misalignment

Death Valley, JPL Test

INS/Camera Mount

Boresight Misalignment Since DPO rotational components are naturally related to the INS body frame, they must be transformed to the imaging sensor frame The angular and linear misalignments between the INS body frame and the imaging sensor frame are known as boresight components The boresight transformation must be determined with sufficiently high accuracy

Boresight Misalignment Calibration Angular and linear misalignments between the INS body frame and the imaging sensor frame Resolved by comparison of the GPS/INS positioning/orientation results with independent AT solution or as a part of a modified bundle adjustment with constraints Should be performed at a specialized test range No flex or rotation of the common mount of the imaging and the georeferencing sensors can occur

Direct Georeferencing ZBINS XBINS XC YC YBINS ZM XM YM rM,k rm,i,j rM,INS 3D INS coordinates in mapping frame 3D object coordinates in model frame (derived from i,j stereo pair) attached to C-frame 3D coordinates of point k in M-frame boresight matrix between INS body frame and camera frame C rotation matrix between INS body frame and mapping frame M, measured by INS boresight offset components scaling factor s

Error in Object Coordinates Due to Errors in Boresighting

Boresight Calibration: Example Aerotriangulation Results for Boresight Calibration SoftPlotter data reduction and adjustment packages 2 cm accuracy assumed for control points 7  for image coordinate observations GPS/INS Results for Boresight Calibration 5-8 arcsec estimated standard deviations of attitude components 1-2 cm estimated standard deviations of INS center position

Boresight Matrix Estimation: Example RPht - photogrammetrically derived attitude matrix RINS - GPS/INS derived attitude matrix S - reflection matrix to approximately align INS and camera frame axes (could be absorbed directly by RB RB - boresight matrix

Boresight Estimation Quality Performed on ground control points (natural objects) The standard deviations for the boresight components linear displacements : 0.22, 0.08 and 0.06 m rotation angles , , : 0.01, 0.03 and 0.04 deg Possible reasons for modest quality: Photogrammertic processing accuracy affected by poor signalization of control points Mechanical problems with the camera body/mount Image time tagging (rather unlikely)

IMU OTF Error Calibration The main sources of errors in an inertial navigation are due to the following factors: The time rates of change of the velocity errors are driven chiefly by accelerometer errors and gravity disturbance The attitude error rates are driven primarily by gyroscope errors In the aligned INS the main errors in tilt are due to accelerometer bias and the gravity disturbance vector It is important that the IMU errors are properly estimated and applied by the feedback loop Platform maneuvers are needed to separated error sources

IMU Error Calibration m/s2 Straight portion of the flight

IMU Error Calibration arcsec/h

IMU Error Calibration – Flight Trajectory

Accelerometer Scaling Factor for Different Atmospheric Conditions GPS time [s] GPS time [s]

Positioning Error Growth During GPS Losses of Lock Due to Obstruction or Interference (forward direction, end of the gap)

Summary Direct orientation can be achieved with high accuracy Proper system calibration is crucial AT is needed for boresight calibration Radio interference poses problems (AT is needed) Continuity of the GPS lock can pose serious problems in land-based applications Rigidity of the multi-sensor system mount -- embedded systems are preferred Precise GPS/INS time synchronization is crucial