1. AGU 2004 Fall Meeting, May 19, 2015Slide 1 Jointly Retrieving Surface Soil Moisture from Active and Passive Microwave Observations using Cubist Data-Mining Xiwu Zhan UMBC-GEST/NASA-GSFC, Code 974.1, Greenbelt, MD Paul Houser NASA-GSFC HSB, Code 974, Greenbelt, MD Jeffrey Walker University of Melbourne, Victoria, Australia

2. AGU 2004 Fall Meeting, May 19, 2015Slide 2 OBJECTIVES There will be high resolution (up to 1km) radar backscatter observations of land surface soil moisture from NASA ESSP Hydros mission. Radiation transfer models are usually inversed to retrieve soil moisture value. Can data- mining provide an alternative? There are many global high resolution satellite data sets of land surface parameters that are related to soil moisture. Can we use them to derive soil moisture when radar data are not available? What are the accuracies of these alternative soil moisture retrieval methods?

3. AGU 2004 Fall Meeting, May 19, 2015Slide 3 METHODOLOGY Use the 1km geophysical and biophysical data fields and microwave emission and backscatter models (MEBM) from the Observation System Simulation Experiment (OSSE in Crow et al, 2004, Zhan et al, 2005) of NASA ESSP Hydros Mission; Inverse the simulated radiometer and radar observations with the MEBM for soil moisture retrievals; Use the Cubist data-mining tool to generate Cubist models and use the models to obtain fine resolution soil moisture retrievals Use the update equation of the Extended Kalman Filter (EKF) to combine course resolution and fine resolution soil moisture estimations for an optimal soil moisture retrieval data product; Compute the RMSEs of soil moisture retrievals of the three methods (INV, Cubist, & EKF) against the original soil moisture data fields.

4. AGU 2004 Fall Meeting, May 19, 2015Slide 4 Hydros : Hydrosphere States Mission Spinnin g 6m dish A NASA Earth System Science Pathfinder mission; Surface soil moisture w/  4%vol. accuracy and Freeze/Thaw state transitions; Revisit time: Global 3 days, boreal area 2 days L-band (1.41GHz) Radiometer sensing 40km brightness temp. with H & V polarization; L-band (1.26GHz) Radar measuring 1-3km backscatters with hh, vv, hv polarization; Soil moisture products: 3km radar retrievals, 40km radiometer retrievals, 10km radar and radiometer combined retrievals and 5km 4DDA results.

5. AGU 2004 Fall Meeting, May 19, 2015Slide 5 Hydros OSSE: Data Layers 36 km T Bh, T Bv data from Hydros radiometer simulator 9 km soil moisture retrieval product 3 km  hh,  vv,  hv data from Hydros radar simulator 1 km soil moisture data from nature run 1 36km pixel 9 9km pixels 144 3km pixels 1296 1km pixels

6. AGU 2004 Fall Meeting, May 19, 2015Slide 6 “Truth” Validation Radiometer Inversion 3/9/36km SM Retrieval Error 1 km Nature Run (Input: LC, ST, NDVI, Rainfall, Met data) 3/9km  Hydros Instrument Simulator 1km SM 36km Tb Aggregate to 3/9/36km 1km SM 3/9/36km SM 3/9km SM EKF Algorithm 36km SM Tb &  Errors Radiometer Tb Forward Model Radar  Forward Model EKF Algorithm Innovations Calculate optimized SM Error Models Based on red And white noise 3/9km  3/9/36km SM 36km Radiometer Forward Model Tb Iteration 36km SM 36km Tb Observations From Hydros instrument simulator Background From radio- meter inversion 1km T soil 1km T skin 36km SM Data Flow for Using EKF to Retrieve SM from T b &  Observations Radar Inversion 3km Radar Forward Model  Iteration 1km Radiometer Tb Forward Model Aggregate to 36km White Noise 1km Radar  Forward Model Aggregate to 3/9km Red Noise White Noise

7. AGU 2004 Fall Meeting, May 19, 2015Slide 7 “Truth” Validation Radiometer Inversion 3/9/36km SM Retrieval Error 1 km Nature Run (Input: LC, ST, NDVI, Rainfall, Met data) 3/9km  Hydros Instrument Simulator 1km SM 36km Tb Aggregate to 3/9/36km 1km SM 3/9/36km SM 1km SM EKF Algorithm 36km SM 1km SMErrors EKF Algorithm Innovations Calculate optimized SM Error Models Based on red And white noise 1km ndvi, Ts/  1/3/9/36km SM 36km Radiometer Forward Model Tb Iteration 36km SM 36km Tb Observations From Cubist model Background From radio- meter inversion 1km T soil 1km T skin 36km SM Cubist Model 1 km Cubist Models ndvi, Ts/  Observ. 1km SM 1km Radiometer Tb Forward Model Aggregate to 36km White Noise 1km Radar  Forward Model Aggregate to 3/9km Red Noise White Noise Data Flow for Using EKF to Retrieve SM from T b & other Observations

8. AGU 2004 Fall Meeting, May 19, 2015Slide 8 Radar observational are not handily available for retrieving soil moisture before the launch of Hydros in 2010: spatial coverage, revisit time; Radar radiation transfer models are not as mature as radiometer models for inversing soil moisture; Why Alternative for Radar Model? T* NDVI * Low Soil Moisture high Soil Moisture NDVI o NDVI s To Ts Visible/Infrared observations such as NDVI, LST and albedo from MODIS, Landsat and future VIIRS on NPOESS are available everyday at high spatial resolutions; The “Universal Triangle” relationships between soil moisture and the visible/infrared observations have been documented in literature for many years.

9. AGU 2004 Fall Meeting, May 19, 2015Slide 9 Cubist is used to build regression tree model of the relationships between soil moisture and its related land surface parameters such radar backscatter, or, NDVI, surface temperature and albedo; Regression tree is similar to the decision tree classifier in that it recursively splits training samples into subsets, two at each split; Instead of assigning class labels to the subsets, it develops a linear regression model for each of them; Each splitting is made such that the combined residual error of the models for the two subsets is substantially lower than the residual error of the single best linear model for the samples in the two subsets, and that the combined residual error of the split is the minimum of all possible splits Cubist: a Data-mining Computer Tool

10. AGU 2004 Fall Meeting, May 19, 2015Slide 10 Noise in data : Sigma -.5dB, Tb – 1K, ndvi – 10%, Ts-.5K, Roughness-5%, VWC-10% Cubist Model Compared with Radar Model For low noise data, Cubist model of radar backscatters may reduce the RMSEs of radar model inversions by about 1-2 %v/v;

11. AGU 2004 Fall Meeting, May 19, 2015Slide 11 Noise in data : Sigma - 1dB, Tb – 1K, ndvi – 20%, Ts-1K, Roughness-10%, VWC-20% Cubist Model Compared with Radar Model For high noise data, Cubist model of radar backscatters could reduce the RMSEs of radar model inversions by about 3-4 %v/v;

12. AGU 2004 Fall Meeting, May 19, 2015Slide 12 Cubist Model Applicability/Stability Cubist model of radar backscatters using same day or other day training data results very similar accuracy; Cubist model based on low noise data produces almost the same accuracy as based on high noise data, and the opposite is true too.

13. AGU 2004 Fall Meeting, May 19, 2015Slide 13 Cubist Model Using Visible/IR Data If the data noises are low, RMSEs of soil moisture retrievals from Cubist model using only visible/IR observations may be 3-5%v/v higher than radar backscatter model inversions.

14. AGU 2004 Fall Meeting, May 19, 2015Slide 14 Cubist Model Using Visible/IR Data RMSEs of soil moisture retrievals from Cubist model using only visible/IR observations may be 1-3%v/v higher than radar backscatter model inversions based on the high noise data. Radar observations are apparently more reliable than visible/IR obs as long as a radar model is known.

15. AGU 2004 Fall Meeting, May 19, 2015Slide 15 Extended Kalman Filter for SM Retrieval X a – soil moisture retrieval X b – background SM K – Kalman gain Z – observations h(X) – obs function H – obs operator P – bg error covariance R – obs error covariance Kalman filter is a statistical data assimilation technique that calculates an optimal observation correction term to the background value based on the relative magnitude of the error covariances of the observations and the background.

16. AGU 2004 Fall Meeting, May 19, 2015Slide 16 EKF Application Result - 1 EKF retrievals using Cubist model of 1km radar sigmas and 36km Tb inversion are marginally better than Cubist model estimates when the error covariance difference between Tb inversion and Cubist model is large;

17. AGU 2004 Fall Meeting, May 19, 2015Slide 17 EKF Application Result - 2 When the error covariance difference between Tb inversion and Cubist model are smaller, the advantage of the EKF retrievals is larger (2-4% less RMSE);

18. AGU 2004 Fall Meeting, May 19, 2015Slide 18 EKF Application Result - 3 EKF retrievals using Cubist model of 1km visible/IR obs and 36km Tb inversion are better than both Cubist model estimates and Tb inversion (2-3% less RMSE). But the Cubist model does not produce retrievals as good as using radar backscatter (4-5% larger RMSE).

19. AGU 2004 Fall Meeting, May 19, 2015Slide 19 EKF Application Result - 4 EKF retrievals using Cubist model of 1km visible/IR obs and 36km Tb inversion are marginally better than both Cubist model estimates and Tb inversion for high noise data. Cubist model of ndvi and Ts is not as good as radar backscatter models (1-3% higher RMSE).

20. AGU 2004 Fall Meeting, May 19, 2015Slide 20 Error Distribution of SM Retrievals 36km Tb Inversion1km Cubist Sigma Model 1km Radar Sigma Inversion 1km EKF Tb Inv + Cubist Sigma 9.1% RMSE in %v/v 3.3% 3.2%5.3% Low noise data, Day 155

21. AGU 2004 Fall Meeting, May 19, 2015Slide 21 Error Distribution of SM Retrievals 36km Tb Inversion1km Cubist Sigma Model 1km Radar Sigma Inversion 1km EKF Tb Inv + Cubist Sigma 9.1% RMSE in %v/v 7.1% 5.9%10.9% High noise data, Day 155

22. AGU 2004 Fall Meeting, May 19, 2015Slide 22 SUMMARY When radar backscatter (sigma) observations are available, a Cubist model of sigmas could be used to retrieve soil moisture with better accuracy (1-4% less RMSE) than a radar backscatter model. The same set of equations of the Cubist model based on one set of training data may be applicable to other sets of data. Thus a Cubist model could be an alternative to a radar backscatter model based on the data we used. EKF Data Assimilation method can combine high resolution and low resolution soil moisture estimations and improve retrieval accuracy. Based on the NDVI and Ts used currently, a Cubist model of NDVI and Ts is not as reliable as a Cubist model of radar sigma data. The difference of RMSE could be as high as 1-5%. However, the radar sigma inversions were obtained with the same radar backscatter model used to generate the radar backscatter data with noises added while the impacts of NDVI and Ts in the radar backscatter model may be as significant as in reality. Thus further investigations using real high resolution soil moisture, and visible/IP observational data are still needed.

23. AGU 2004 Fall Meeting, May 19, 2015Slide 23 Combining Optical/IR RS and MW RS for High Resolution Soil Moisture NDVI,LST,A or Sigmas – SM Relationships AMSR-E/CMIS SMOS, Hydros, MW Observations MODIS/VIIRS TM, SPOT, Hydros Observations of NDVI, LST, A or Sigmas Course Rez (20- 50km) Soil Moisture Retrievals Soil Moisture Truth Data from Airplane/Ground Observations High Rez (30m-3km) Soil Moisture Retrievals EKF Data Assimilation Algorithms Cubist: Data Mining Tools