Preprocessing for Hyperspectral Analysis Calibration, Noise, Bad Bands, Atmosphere

AVIRIS Reflectance Scene: Cuprite, Nevada

Overview Hyperspectral data require extensive preprocessing so that they can produce useable spectral information Typically convert to ground reflectance so that data can be compared to spectral libraries Calibration (conversion of DNs to radiance) Atmospheric correction Often need to remove “bad bands” and create spectral subsets May need to identify pixels that represent spectral endmembers for unmixing

Goal is to compare sensor-derived spectral curves to spectral libraries or ground measurements. From Map and Image Processing System (MIPS) online user’s manual

Identification and flagging of bad bands Not uncommonly, many bands in a hyperspectral dataset can be unusable Low (poor) SNR No or low radiance due to atmospheric absorption (causes low SNR because signal is low) Sensor problems

AVIRIS Band 3 from Cuprite, Nevada reflectance image

AVIRIS Band 112 from Cuprite, Nevada reflectance image

Atmospheric transmittance across spectrum Band 112 From Map and Image Processing System (MIPS) User manual (online)

Erdas tool for flagging bad bands in a dataset

Choosing spectral subsets Accuracy of classifications can actually decrease with increasing number of bands Sometimes called the Hughes Phenomenon or the “curse of dimensionality” High dimensional statistics require more input data (e.g., training pixels) to work effectively Many bands in a hyperspectral dataset are redundant (correlated) and therefore not needed. Many bands may be irrelevant for certain targets or applications.

Erdas tool for omitting unwanted bands from an analysis

Which areas might you omit (or choose) if you need to ID montmorillonite?

Techniques for choosing spectral subsets. Removal of atmospheric absorption regions (often “bad bands”) Awareness of key spectral features of target(s) Principal Components Analysis (PCA) Can be used to reduce dimensionality of hyperspectral data IF you are not interested in the actual spectral reflectance curves Lower PCs contain most of the information in a dataset. PCA can also help identify endmembers for spectral unmixing. Can be problematic when amount of noise varies from band to band in an image Minimum Noise Fraction (MNF) – a variation of PCA that adjusts for bands that have uneven amounts of noise.

Minimum Noise Fraction (MNF) When noise varies from band to band, PCA produces a set of components that don’t have increasing noise with increasing component # (PC1, PC2, PC3…PC224). It is often desirable to cloister the noise in the higher order components (PCs) MNF Analysis is a modified version of PCA that creates new “bands” (components) with increasing noise content as you go up through the components Can invert to re-create bands that have a Gaussian noise distribution. Concentrates the information in low order components and noise in high order components.

Erdas MNF tool.

Calibration Calibration is the process of converting raw DNs to radiance. Requires internal calibration information for each sensor Often improved by on-the-ground measurement of known targets

Uncalibrated at-sensor brightness: combination of atmospheric effects, sensor characteristics, and reflectance of the lake bed target (red square) From Map and Image Processing System (MIPS) User manual (online)

Spectral curve from lakebed in same AVIRIS image after conversion to reflectance

Atmospheric correction (Conversion of at-satellite radiance to ground reflectance) Hyperspectral analysis usually requires correction for both scattering (path radiance) and transmittance Often use radiative transfer models (based on the physics of the interaction of light with atmospheric components) Can be accomplished with ground measurements or image-based techniques e.g., flat field correction, empirical line method

Atmospheric correction programs are available "off the shelf" or modified MODTRAN: MODerate resolution atmospheric TRANsmission – models transmission of light through the atmosphere 6S: Second Simulation of a Satellite Signal in the Solar Spectrum ACORN: Atmospheric CORrection Now ATREM: ATmospheric REMoval (modeled after MODTRAN) FLAASH: Fast Line-of-site Atmospheric Analysis of Spectral Hypercubes ATCOR: ATmospheric CORrection HATCH Others…

ATREM removal of atmospheric interferance from satellite radiance AVIRIS data: Kansas City Water vapor image “removed” by ATREM

Kansas City image

Atmospheric transmittance Strongly affected by several atmospheric constituents Water vapor Carbon dioxide Ozone Nitrous oxide Carbon monoxide Methane Oxygen Also affected by amount of aerosol in atmosphere (larger particles)

How do these programs work? Two basic strategies Use field data (e.g., from a radiosonde) to model atmospheric conditions in the “column” between ground and top of atmosphere (e.g., MODTRAN) Infer atmospheric conditions from key hyperspectral bands that correspond to atmospheric absorption dips (e.g. ATREM)

Example: MODTRAN4 User’s Manual (current version is 5) More info available at MODTRAN website: http://www.modtran5.com Note that MODTRAN is complicated (manual is 99 pages long) and requires a lot of investment just to learn to run. Some companies have created more user-friendly front end interfaces for it.

Standard MODTRAN atmospheres Default atmospheric profiles include seasonal representations of: Atmospheric pressure Temperature Density Water vapor Ozone

Other MODTRAN parameters include Scattering options Geometry Amounts of atmospheric constituents Solar irradiance Sensor characteristics Etc.

Example: ATREM calibration of AVIRIS data in Park City, Utah USGS Spectroscopy Lab used ATREM to correct AVIRIS data using a fairly uniform dam site Data correction programs like ATREM don’t do a perfect job and need to be calibrated (fudged) for particular images Data calibration was for a project characterizing abandoned mine lands (AML) in Utah

Simplified schematic of procedure AVIRIS data ATREM model Per pixel ground reflectance

Procedure Choose a calibration site on the ground Should be large (multiple pixels) Should be spectrally uniform Should be spectrally neutral (not extremely bright or dark) Collect spectral data on the ground using a handheld spectral device (e.g., ASD spectroradiometer)

Calibration Site: Deer Creek, Utah

Field spectra collection using Analytical Spectral Devices (ASD) radiometer

Procedure (cont.) Run the ATREM model on the AVIRIS image Translate ground spectral bands to match the spectral band position and width of corrected AVIRIS data Correct for path radiance using dark pixels from another part of image (ATREM tends to overcorrect for path radiance) Dark pixels chosen for path radiance correction

Before calibration!

Procedure (cont.) Dark pixel spectrum is used to create an “offset spectrum” that will be subtracted from ATREM corrected AVIRIS data (essentially dark pixel subtraction)

Procedure (cont.) Extract field calibration site pixels from corrected AVIRIS data

Procedure (cont.) Divide field spectrum by ATREM-corrected spectrum to create a “multiplier spectrum” which can be multiplied by every AVIRIS pixel spectrum to calibrate.

Procedure (cont.) Apply offset and multiplier spectra to ATREM corrected imagery Applied to all pixels, not just the ones on the calibration site Allows corrected image to more closely match what would be measured if you were on the ground

Other atmospheric correction strategies Empirical line method (image based) Measure reflectance on ground at two sites with relatively high and low brightness, but which also have relatively constant reflectance across bands Get satellite radiance for the two sites and graph points Use slope and intercept of the line connecting the two to correct data

Other methods (cont.) Flat field correction Find an area in the image that has relatively constant (“flat”) and high reflectance (to minimize noise) across the bands you are correcting. Extract spectral reflectance of flat field from image Divide all pixels in all bands by the corresponding flat field values Gives relative reflectance (not absolute) Can be difficult to find a flat field Dry lakebeds work well. Spectral features (where it isn’t spectrally “flat”) in the flat field will cause problems

Summary Image pre-processing for hyperspectral analysis is a rigorous and time-consuming task that needs to be done well or projects will not be successful. Worth investing considerable time into pre-processing.