1. What is Data Assimilation ? Data Assimilation: Data assimilation seeks to characterize the true state of an environmental system by combining information from measurements, models, and other sources. Typical measurements for hydrologic/earth science applications: Ground-based hydrologic and geological measurements (stream flow, soil moisture, soil properties, canopy properties, etc.) Ground-based meteorological measurements (precipitation, air temperature, humidity, wind speed, etc.) Remotely-sensed measurements (usually electromagnetic) which are sensitive to hydrologically relevant variables (e.g. water vapor, soil moisture, etc.) Mathematical models used for data assimilation: Models of the physical system of interest. Models of the measurement process. Probabilistic descriptions of uncertain model inputs and measurement errors. A description based on combined information should be better than one obtained from either measurements or model alone.

2. State estimation -- System is described in terms of state variables, which are characterized from available information Multiple data sources -- Estimates are often derived from different types of measurements (ground-based, remote sensing, etc.) measured at different times and resolutions. State variables may fluctuate over a wide range of time and space scales -- Different scales may interact (e.g. small scale variability can have large-scale consequences) Spatially distributed dynamic systems -- Systems are often modeled with partial differential equations, usually nonlinear. Uncertainty -- The models used in data assimilation applications are inevitably imperfect approximations to reality, model inputs may be uncertain, and measurement errors may be important. All of these sources of uncertainty need to be considered in the data assimilation process. The equations used to describe the system of interest are usually discretized over time and space -- Since discretization must capture a wide range of scales the resulting number of degrees of freedom (unknowns) can be very large. Key Features of Environmental Data Assimilation Problems

3. State-Space Framework for Data Assimilation State-space concepts provide a convenient way to formulate data assimilation problems. Key idea is to describe system of interest in terms of following variables: Input variables -- variables which account for forcing from outside the system or system properties which do not depend on the system state. State variables -- dependent variables of differential equations used to describe the physical system of interest, also called prognostic variables. Output variables -- variables that are observed, depend on state and input variables, also called diagnostic variables. Classification of variables depends on system boundaries: Precip. Land Atmosphere Precip. Land Atmosphere System includes coupled land and atmosphere -- precipitation and evapo-transpiration are state variables System includes only land, precipitation and evapo- transpiration are input variables ET

4. Components of a Typical Hydrologic Data Assimilation Problem The data assimilation algorithm uses specified information about input fluctuations and measurement errors to combine model predictions and measurements. Resulting estimates are extensive in time and space and make best use of available information. Measurement Eq: State Eq: True Output z i (e.g. radiobrightness) Measurement system Measured State y (t) (e.g. soil moist.) Time-invariant input  (e.g. sat. hydr. cond.) Hydrologic system Specified (mean) Random fluctuations Random error,  Random fluctuations Specified (mean) True Time-varying input u ( t) (e.g. precip) Data assimilation algorithm Estimated states and outputs Means and covariances of true inputs and output measurement errors

5. 100101102 1 1.5 2 2.5 3 3.5 9095100105110115120 -2 0 1 2 3 4 When models are discretized over time/space there are two sources of output measurement error: Instrument errors (measurement device does not perfectly record variable it is meant to measure). Scale-related errors (variable measured by device is not at the same time/space scale as corresponding model variable) Types of Measurement Errors When measurement error statistics are specified both error sources should be considered Large-scale trend described by model True value Measurement Instrument error Scale-related error * * * *

6. Types of Data Assimilation Problems - Temporal Aspects Z i = [ z 1, z 2, …, z i ] =Set of all measurements through time t i Smoothing: characterize system over time interval t  t i Use for reanalysis of historic data tt2t2 t1t1 titi Filtering/forecasting: characterize system over time interval t  t i Use for real-time forecasting titi t2t2 t1t1 t Interpolation: no time-dependence, characterize system only at time t = t i t=t i Use for interpolation of spatial data (e.g. kriging)

7. Types of Data Assimilation Problems - Spatial Aspects Downscaling: Characterize system at scales smaller than output measurement resolution Upscaling: Characterize system at scales larger than output measurement resolution States ( y 1 … y 4 ) Measurement ( z 1 ) Measurements ( z 1...z 4 ) Downscaling and upscaling are handled automatically if measurement equation is defined approriately State ( y 1 )

8. Characterizing Uncertain Systems What is a “good characterization” of the system states and inputs, given the vector Z i = [z 1,..., z i ] of all measurements taken through t i ? The posterior probability densities p(y| Z i ) and p(u| Z i ) are the ideal estimates since they contain everything we know about the state y or input u given Z i. ModeMean Std. Dev. p[y(t)| Z i ] y(t)y(t) In practice, we must settle for partial information about this density u: p(u) p(u | Z i ) y: p(y) p(y | Z i ) PriorConditional y = A(u) ZiZi Variational DA: Derive mode of p[y(t)| Z i ] by solving batch least-squares problem. Sequential DA: Derive recursive approximation of conditional mean (and covariance?) of p[y(t)| Z i ]

9. The Variational/Batch Approach is found with an iterative search. Search convergence is improved by the presence of the second (regularization) term in J B. Most variational methods use the mode of p u|z ( u| Z i ) as an estimate of uncertain input vector. State estimate is obtained by substituting into state equation: u1u1 u2u2 If  and u are multivariate normal is the value of that minimizes the following generalized least-squares error measure: Terms that do not depend on The state equation is often incorporated as a constraint, using adjoint methods.

10. The Sequential Approach Sequential methods are designed to propagate and update the conditional pdf in a series of discrete steps: zizi Z i = [Z i-1, z i ] t0t0 z1z1 Z 1 = [z 1 ] t1t1 z2z2 Z 2 = [Z 1, z 2 ] t2t2 titi t i+1 Algorithm initialized with unconditional (prior) PDF at t 0 p yi| zi-1 [ y i |Z i-1 ]p y,i+1| zi [ y i+1 |Z i ] p yi| zi [ y i |Z i ] Update i Propagation i to i +1 p y0 [ y 0 ] p y1 [ y 1 ] p y1| z1 [ y 1 |Z 1 ] Propagation 1 to 2 p y2| z1 [ y 2 |Z 1 ] Update 1 Propagation 0 to 1 Meas. i Meas. 1Meas. 2 In practice various approximations must be introduced.

11. Some Common Sequential Data Assimilation Methods A common approximation is to assume that the conditional PDF is multivariate Gaussian. The update for conditional mean has the form: Some common approximations: DirectUpdate forced to equal measurements where available, insertioninterpolated from meas. elsewhere Nudging:K = empirically selected constant OptimalK derived from assumed (static) covariance Interpolation: ExtendedK derived from covariances propagated with a linearized Kalman filter:model, input fluctuations and measurement errors must be additive. EnsembleK derived from a ensemble of random replicates propagated Kalman filter:with a nonlinear model, form of input fluctuations and measurement errors is unrestricted. K weights measurements vs. model predictions updatePropagated estimate

12. Example -- Microwave Measurement of Soil Moisture L-band (1.4 GHz) microwave emissivity is sensitive to soil saturation in upper 5 cm. Brightness temperature decreases for wetter soils. Objective is to map soil moisture in real time by combining microwave meas. and other data with model predictions (data assimilation). 00.20.40.60.8 1 0.5 0.6 0.7 0.8 0.9 1 saturation [-] microwave emissivity [-] sand silt clay

13. Case Study Area Aircraft microwave measurements SGP97 Experiment - Soil Moisture Campaign

14. Problem Specifications –SGP97 Ensemble Kalman Filter Example Hydrologic model: 1D (vertical) NOAH Land Surface Model (NOAA NCEP, Chen et al, 1996) applied at each estimation pixel Radiative Transfer Model: Jackson et al, 1999 model applied at each pixel Uncertain model inputs included in ensemble filter: Time-varying inputs: Precipitation (temporally uncorrelated) Time-invariant inputs: Porosity (upper bound on moisture content) Wilting point (lower bound on moisture content) Saturated hydraulic conductivity Minimum stomatal resistance Random fluctuations are multiplicative and lognormal (mean=1.0) Filter assumes that random fluctuations and measurement errors for different pixels are uncorrelated Random measurement errors included in ensemble filter: Additive radiobrightness measurement noise

15. Relevant Time and Space Scales Vertical Section Soil layers differ in thickness Note large horizontal-to-vertical scale disparity 5 cm 10 cm Typical precipitation events For problems of continental scale we have ~ 10 5 est. pixels, 10 5 meas, 10 6 states, 0.8 km 4.0 km Plan View Estimation pixels (large) Microwave pixels (small) 170 = 6/19/97 170175180185190195 0 0.005 0.01 0.015 0.02 0.025 mm/s *** ** ****** * *** * = ESTAR observation

16. Some Typical Spatially Variable Model Inputs –SGP97 Example 00.050.1 Clay fraction 00.20.40.60.8 Sand fraction RTM Inputs 02468 NOAH soil class 024681012 NOAH vegetation class NOAH Inputs Meteor. Stations El Reno 50 km Estimation region ~ 50 by 200 km (12 by 50 pixels 4 km on a side)

17. Brightness Temperatures at a Typical Pixel – SGP97 Example Brightness temp. deg. K. Brightness Temp. and Precip Time Series – El Reno Conditional mean Unconditional mean Brightness meas. Individual replicates Precip

18. Moisture Contents at a Typical Pixel – SGP97 Example Moisture content Unconditional mean Individual replicates Brightness meas. times Local spatial average of gravitimetric meas. Conditional mean Moisture Content and Precip. Time Series – El Reno Precip

19. Comparison of Some Data Assimilation Options Direct insertion, nudging, optimal interpolation Easy to implement + Updates do not account for system dynamics or input and measurement statistics – No information on estimation accuracy – Computationally efficient + Well-suited for real time applications, not optimal for smoothing problems +/- Provides information on estimation accuracy + Very flexible, modular, able to accommodate wide range of model error descriptions + No need for adjoint model or for linearizations or other approximations during propagation step + Approach is robust and easy to use + Update assumes states are jointly normal – Can be computationally demanding – Ensemble Kalman filter Can be adapted for real time or smoothing problems + Provides info. on estimation accuracy + Computationally demanding, limited capability to deal with model errors - Linearization approximation may be poor, tends to be unstable - Extended Kalman filter Well-suited for smoothing problems, less convenient for real-time applications + / - Does not provide information on estimation accuracy - Difficult to accommodate time-dependent model errors, not robust – Most efficient forms require derivation of an adjoint model - Variational methods