1. Data Assimilation Tristan Quaife, Philip Lewis

2. What is Data Assimilation? A working definition: The set techniques the combine data with some underlying process model to provide optimal estimates of the true state and/or parameters of that model.

3. What is Data Assimilation? It is not just model inversion. But could be seen as a process constraint on inversion (e.g. a temporal constraint)

4. e.g. Use EO data to constrain estimates of terrestrial C fluxes Terrestrial EO data: no direct constraint on C fluxes Combine with model

5. Data Assimilation is Bayesian Bayes’ theorem: P(A|B)= P(B|A) P(A) P(B)

6. What does DA aim to do? Use all available information about The underlying model The observations The observation operator Including estimates of uncertainty and the current state of the system To provide a best estimate of the true state of the system with quantified uncertainty

7. Kalman Filter DA: MODIS LAI product Data assimilation into DALEC ecological model

8. Lower-level product DA Ensure consistency between model and observations Assimilate low-level products (surface reflectance) Uncertainty better quantified Need to build observation operator relating model state (e.g. LAI) to reflectance Example of Oregon (MODIS DA) Quaife et al. 2008, RSE

9. Modelled vs. observed reflectance RedNIR Note BRF shape in red: can’t simulate with 1-D canopy (GORT here)

10. NEP results No assimilation Assimilating MODIS (red/NIR) DALEC model calibrated from flux measurements at tower site 1

11. Integrated flux predictions Flux (gC.m -2 ) Assimilated data 3yr total Standard Deviation NEP No assimilation240.2212.2 MODIS B1 & B2373.0151.3 Williams et al. (2005) 406.027.8 GPP No assimilation1646.4834.5 MODIS B1 & B22620.396.8 Williams et al. (2005) 2170.318.1 Flux (gC.m -2 ) Assimilated data Total Standard Deviation NEP Assimilation exc. snow 373.0151.3 Assimilation inc. snow 404.8129.6 Williams et al. (2005) 406.027.8 GPP Assimilation exc. snow 2620.396.8 Assimilation inc. snow 2525.642.7 Williams et al. (2005) 2170.318.1

12. Mean NEP for 2000-2002 1565 gC/m 2 /year 4.5 km Flux Tower Spatial average = 50.9 Std. dev. = 9.7 (gC/m 2 /year)

13. NEP – Site2 (intermediate) parameters, with/without DA Model running at Site 2, Oregon Site 1 model EO-calibrated at site 2 NEP observations from Site 2 Shows ability to spatialise

14. Data assimilation Low-level DA can be effective ‘easier’ data uncertainties Can be applied to multiple observation types Requires Observation operator(s) RT models Requires other uncertainties Ecosystem Model Driver (climate) Observation operator

15. Specific issues in land EOLDAS No spatial transfer of information Require full spatial coverage Atmosphere dealt with by an instantaneous retrieval (i.e. no transport model) All state vector members influence observations We are not interested in other variables!

16. Sequential Smoothers Variational Nominal classification of DA

17. Kalman Filter Variants - EKF Ensemble Kalman Filter Variants – Unscented EnKF Particle filters Lots of different types true MCMC technique Sequential methods

18. Propagation step: x=Mx - P=MP - M T + Q Analysis step: x * = x + K( y – Hx ) K= PH T ( HPH T +R ) -1 The Kalman filter State vector Model Covariance matrix Stochastic forcing Kalman gain Observation vector Observation covariance matrix Observation operator

19. The Kalman Filter Linear process model Linear observation operator Assumes normally distributed errors

20. Propagation step: x=m(x - ) P=M'P - M' T + Q Analysis step: x * = x + K( y – h(x) ) K= PH' T ( H'PH' T +R ) -1 The Extended Kalman filter Jacobian matrix

21. The Extended Kalman Filter Linear process model Linear observation operator Assumes normally distributed errors Problem with divergence

22. Propagation step: X=m(X - ) + Q no explicit error propagation Analysis step: X * = X + K( D – HX ) K= PH T ( HPH T +R ) -1 The Ensemble Kalman filter State vector ensemble Perturbed observations

23. The Ensemble Kalman Filter P estimated from X Non linear observations using augmentation: X a = h(X) X

24. The Ensemble Kalman Filter No assimilation Assimilating MODIS surface reflectance bands 1 and 2

25. The Ensemble Kalman Filter Avoids use of Jacobian matrices Assumes normally distributed errors – Some degree of relaxation of this assumption Augmentation assumes local linearisation

26. Particle Filters Propagation step: X=m(X - ) + Q Analysis step: e.g. Metropolis-Hastings

27. Particle Filters

28. No available observations

29. Particle Filters Fully Bayesian – No underlying assumptions about distributions Theoretically the most appealing choice of sequential technique, but… Our analysis show little difference with EnKF Potentially requires larger ensemble – But comparing 1:1 is faster than EnKF

30. Sequential techniques General considerations: – Designed for real time systems – Only consider historical observations – Only assimilates observations in single time step – Can lead to artificial high frequency components

31. Extension of sequential techniques All observations effect every time step Analogous to weighting on observations – [ smoothing-convolution / regularisation ] Difficult to apply in rapid change areas Smoothers

32. Smoothers - regularisation x = (H T R -1 H + λ 2 B T B) -1 H T R -1 y B is the required constraint. It imposes: Bf = 0 and the scalar λ is a weighting on the constraint. Constraint matrix Lagrange multiplier

33. Regularisation

34. Quaife and Lewis (2010) Temporal constraints on linear BRDF model parameters. IEEE TGRS, in press.

35. Regularisation Lots of literature on the selection of λ – Cross validation etc Permits insight into the form of Q

36. Variational techniques Expressed as a cost function Uses numerical minimisation Gradient descent requires differential Traditionally used for initial conditions – But parameters may also be adjusted

37. 3DVAR J(x) =( x-x - ) P -1 ( x-x - ) T + ( y-h(x) ) R -1 ( y-h(x) ) T Background Observations

38. 3DVAR No temporal propagation of state vector – OK for zero order approximations – Unable to deal with phenology

39. 4DVAR J(x) = ( x-x - ) P -1 ( x-x - ) T + ( y-h(x i ) ) R -1 ( y-h(x i ) ) T ΣiΣi Time varying state vector

40. Variational techniques Parameters constant over time window Non smooth transitions Assumes normal error distribution Size of time window? For zero-order case 3DVAR = 4DVAR – 4DVAR for use with phenology model Absence of Q - propagation of P?

41. Conclusions - technique DA is optimal way to combine observations and model Range of options available for DA Sequential Smoothers Variational Require understanding of relative uncertainties of model and observations Require way of linking observations and model state Observation operator (e.g. RT)

42. References T. Quaife, P. Lewis, M. DE Kauwe, M. Williams, B. Law, M. Disney, P. Bowyer (2008), Assimilating Canopy Reflectance data into an Ecosystem Model with an Ensemble Kalman Filter, Remote Sensing of Environment, 112(4),1347-1364. T. Quaife and P. Lewis (2010) Temporal constraints on linear BRF model parameters IEEE Transactions on Geoscience and Remote Sensing doi: 10.1109/TGRS.2009.2038901 http://www.ecmwf.int/newsevents/training/rcourse_notes/DATA_ASSIMILATION/ASSIM_CONC EPTS/Assim_concepts11.htmlhttp://www.ecmwf.int/newsevents/training/rcourse_notes/DATA_ASSIMILATION/ASSIM_CONC EPTS/Assim_concepts11.html http://www.cs.unc.edu/~welch/kalman/