Data Assimilation for the Space Environment Ludger Scherliess Center for Atmospheric and Space Sciences Utah State University Logan, Utah GEM 2003 Meeting Snowmass, CO June 2003

Data Assimilation  Create a coherent and objective ‘picture’ of the space environment by combing the information inherent in the physical model and in the data.  This ‘picture’ (analysis) should satisfy the physical laws and fit the data as best as possible (within their errors).  The analysis can than serve as the best initial condition for a forecast.

Historical Background Data Assimilation in the Atmosphere:  Initial Attemps started in the 1950th (NWP)  Currently adjoint/variational methods are used  Used now also in re-analysis runs to get consistency for the past 40 years. Data Assimilation in the Oceans:  Began with large scales (mean properties)  Regional effords (e.g., Gulf stream) [10-15 yrs ago]  El Nino, emphasis on equatorial pacific (coupled with atmosphere)  GODAE, produce operational upper ocean now- and forecast.

Data Assimilation in Space Sciences  Assimilative Mapping of Ionospheric Electrodynamics (AMIE, Richmond and Kamide, 1988)  Initial Testing of Kalman Filter for Ionospheric Electron Density Reconstructions (Howe et al., 1998)  Kalman Filter to construct the global 3-D Electron Density Field (GAIM) Until Recently only relatively Sparse Data were available

Global Assimilation of Ionospheric Measurements (GAIM) Multi-University Research Initiative (MURI):  Utah State University (PI R. W. Schunk)  University of Colorado - Boulder  University of Texas at Dallas  University of Washington Basic Approach: Use a Physics-Based Ionosphere-Plasmasphere Model as the Basis for Assimilating a Diverse Set of Real-Time (or Near Real-Time) Measurements.

Motivation  Data Assimilation has been proven very successful in Meteorology and Oceanography  Need for More Accurate Now-Casting and Forecasting of Plasma Distribution in the Ionosphere and Plasmasphere (and Magnetosphere?)  Availability of Large Quantities of Data  Diverse Data Sources (Apples and Oranges)  Mature Theoretical/Numerical Space Environment Models  Models contain our ‘knowledge’ of the physics  Data contain information about the ‘true’ state

Objectives  Goal: Optimally combine Data and the Model to create coherent Picture of the Space Environment  Solution satisfies the physical laws and ‘agrees’ with the data (within their error bounds) Both, however, are imperfect:  Models: Forcing, Parametrization, etc.  Data: Observational Errors

The Data Assimilation Cycle ‘Best-Guess’ Background Short-Term Forecast Analysis Data Collection Quality Control Forecast Physical Model creates a forecast which is adjusted by the Data to create an ‘analysis’, which serves as the start for the next model forecast. In the analysis the Data Errors and Model Errors are used as weights.

Data Assimilation Techniques  3-d Var  4-d Var  Kalman Filter x f = Mx +  P f = MPM T + Q y o = Hx +  K = P f H T (HP f H T + R) -1 x a = x f + K(y o - Hx f ) P a = (I-KH)P f

Fundamental Concept of 3D-Var Start with a forecast of the physical model. Produce an analysis by minimizing the difference between the analysis and a weighted combination of  the forecast (the background or best guess field) and  the observations.  At each time-step take a ‘snapshot’ of the state This ‘snapshot’ will serve as the initial condition for the next forecast. We want to minimize a “Cost Function” J which consists of: J = J B + J O +J C

The Cost Function J = J B + J O + J C J B : Weighted fit to the background field J O : Weighed fit to the observations J C : Constraint which can be used to impose physical properties (e.g., analysis should satisfy Maxwell’s equations, continuity equation, …) Strong Constraint: Exactly satisfied Weak Constraint: Approximately satisfied

A typical form for the J B term is: J B = (x A - x B ) T B -1 (x A - x B ) Where: x A : Analysis Variable (e.g., Electron Density,Temperature,…) x B : Background Field, obtained from the Model Forecast B : Background Error Covariance Matrix: How good is your Forecast: Good Forecast high weight Poor Forecast low weight The Cost Function, cont.

A typical form for the J O term is: J O = [y - H(x A )] T R -1 [y - H(x A )] Where: y : Represents all Observations H : Forward Operator which maps the Grid Point Values to Observations (can be linear or nonlinear) R : Observation Error Covariance Matrix: Make use of the probable Errors of the Data Good Data high weight Poor Data low weight (also includes the representativeness of the data) The Cost Function, cont. Constantly compares the data with the model

The Cost Function, cont. Analysis is found my minimizing J using “standard minimization techniques” (find the gradient of J). The analysis is found by:  Analyzing all points at once  Using all available data The Physical Properties/Model were used to:  Obtain the best possible background field  To constrain the Analysis  Cost function and the constraints are not explicitly time dependent

Fundamental Concepts of 4D-Var 4D-Var introduces the temporal dimension to data assimilation Find a close fit to the data that is consistent with the dynamical model over an extended period of time.  Find the the closest trajectory Physical model 4D-Var is a powerful technique to find:  Initial conditions  External Forcing

The Kalman Filter  x - Model State Vector  M - State Transition Matrix   - Transition Model Error  P - Model Error Covariance  Q - Transition Model Error Covariance  y - Data Vector  H - Measurement Matrix   - Observation Error  R - Observation Error Covariance  K - Kalman Gain In the Kalman filter the weight for the background (error covariance matrix) evolves with the same physical model as the state!

Kalman Filter Equations  Model-State Forecast  x - Model State Vector  M - State Transition Matrix   - Transition Model Error  P - Model Error Covariance  Q - Transition Model Error Covariance  y - Data Vector  H - Measurement Matrix   - Observation Error  R - Observation Error Covariance  K - Kalman Gain

Kalman Filter Equations  Model-State Forecast Error  x - Model State Vector  M - State Transition Matrix   - Transition Model Error  P - Model Error Covariance  Q - Transition Model Error Covariance  y - Data Vector  H - Measurement Matrix   - Observation Error  R - Observation Error Covariance  K - Kalman Gain

Kalman Filter Equations  Measurement Equation  x - Model State Vector  M - State Transition Matrix   - Transition Model Error  P - Model Error Covariance  Q - Transition Model Error Covariance  y - Data Vector  H - Measurement Matrix   - Observation Error  R - Observation Error Covariance  K - Kalman Gain

Kalman Filter Equations  Kalman Gain  x - Model State Vector  M - State Transition Matrix   - Transition Model Error  P - Model Error Covariance  Q - Transition Model Error Covariance  y - Data Vector  H - Measurement Matrix   - Observation Error  R - Observation Error Covariance  K - Kalman Gain

Kalman Filter Equations  Model-State Analysis  x - Model State Vector  M - State Transition Matrix   - Transition Model Error  P - Model Error Covariance  Q - Transition Model Error Covariance  y - Data Vector  H - Measurement Matrix   - Observation Error  R - Observation Error Covariance  K - Kalman Gain

Kalman Filter Equations  Model-State Analysis Error  x - Model State Vector  M - State Transition Matrix   - Transition Model Error  P - Model Error Covariance  Q - Transition Model Error Covariance  y - Data Vector  H - Measurement Matrix   - Observation Error  R - Observation Error Covariance  K - Kalman Gain

The Dynamical Model entered the Filter:  Evolution of the State Vector (make a Forecast)  Evolution of the Error Covariance Matrix  Error Covariance Matrix becomes time-dependent!  Model Error is explicitly included in the Assimilation

Data Assimilation Tasks  Develop Physical Model  Develop Assimilation Algorithm  Data Acquisition Software  Data Quality Control  Executive System  Validation Software

Kalman Filter Example Electron Density Along Field Line  Kalman Filter with synthetic data  Truth generated with IFM with Modified equatorial drift  1 Ne Measurement every 15 min at magnetic equator Observation Truth Climat Kalman

Kalman Filter Example Electron Density Along Field Line  Kalman Filter with synthetic data  Truth generated with IFM with Modified equatorial drift  1 Ne Measurement every 15 min at magnetic equator Observation Truth Climat Kalman

Kalman Filter Example Electron Density Along Field Line  Kalman Filter with synthetic data  Truth generated with IFM with Modified equatorial drift  1 Ne Measurement every 15 min at magnetic equator Observation Truth Climat Kalman

Kalman Filter Example Electron Density Along Field Line  Kalman Filter with synthetic data  Truth generated with IFM with Modified equatorial drift  1 Ne Measurement every 15 min at magnetic equator Observation Truth Climat Kalman

Kalman Filter Example Electron Density Along Field Line  Kalman Filter with synthetic data  Truth generated with IFM with Modified equatorial drift  1 Ne Measurement every 15 min at magnetic equator Observation Truth Climat Kalman

Next, consider the more complicated situation:  Global Reconstruction  Many observations  Different kinds of instruments measuring different quantities  Observations are in different places

We currently assimilate data from 16 globally distributed DISS Stations and more than 100 GPS Ground Stations. The GPS Stations are linked to the Fleet of GPS Satellites. In addition, we assimilate Electron Density Data from two DMSP Satellites and simulated data from the C/NOFS satellite in the Filter. Data Distribution

Kalman Filter Reconstruction TEC

In this “test,” the following are variables –TID Equatorward Speed –TID Width in Latitude –TID Amplitude History In this “test,” the assumptions are –TID is a perturbation on the background ionosphere. –TID perturbed densities are very noisy –TID moves along a meridian. –Observations lie along the meridian. Propagation with simple advective model A Model TID

1-D TID/TAD  1-D Propagation  Propagation along meridian  Density Perturbations  Network of ~10 observatories

1-D TID/TAD  1-D Propagation  Propagation along meridian  Density Perturbations  Network of ~10 observatories

1-D TID/TAD  1-D Propagation  Propagation along meridian  Density Perturbations  Network of ~10 observatories

1-D TID/TAD  1-D Propagation  Propagation along meridian  Density Perturbations  Network of ~10 observatories

The Observations  10 Stations  Density Perturbations  100% Noise

Kalman Filter Reconstruction  # of Stations = 10  100% Noise  True Velocity = 2  Reconstruction of TID Density Perturbations T=1

Kalman Filter Reconstruction  # of Stations = 10  100% Noise  True Velocity = 2  Reconstruction of TID Density Perturbations T=2

Kalman Filter Reconstruction  # of Stations = 10  100% Noise  True Velocity = 2  Reconstruction of TID Density Perturbations T=3

Kalman Filter Reconstruction  # of Stations = 10  100% Noise  True Velocity = 2  Reconstruction of TID Density Perturbations T=4

Kalman Filter Reconstruction  # of Stations = 10  100% Noise  True Velocity = 2  Reconstruction of TID Density Perturbations T=5

Kalman Filter Reconstruction  # of Stations = 10  100% Noise  True Velocity = 2  Reconstruction of TID Density Perturbations T=6

Kalman Filter Reconstruction  # of Stations = 10  100% Noise  True Velocity = 2  Reconstruction of TID Density Perturbations T=10

Kalman Filter Reconstruction  # of Stations = 10  100% Noise  True Velocity = 2  Reconstruction of TID Density Perturbations T=15

Kalman Filter Reconstruction  # of Stations = 10  100% Noise  True Velocity = 2  Reconstruction of TID Density Perturbations T=20

Kalman Filter Reconstruction  # of Stations = 10  100% Noise  True Velocity = 2  Guessed Velocity 50% Off  Reconstruction of TID Density Perturbations

Kalman Filter Reconstruction  # of Stations = 10  100% Noise  True Velocity = 2  Guessed Velocity 50% Off  Reconstruction of TID Density Perturbations

Kalman Filter Reconstruction  # of Stations = 10  100% Noise  True Velocity = 2  Reconstruction of TID Density Perturbations  Determination of TID Velocity T=150

Conclusions  Data Assimilation Techniques have proven to be very useful in Meteorological and Oceanographic Specifications and Forecasts.  Millions of Ionospheric Measurements Per Day Within 10 Years.  Physics-Based Data Assimilation Models Will Provide Real-Time Snapshots of the Global Ionosphere.  Many Outstanding Scientific Problems Will be Resolved.  Results Will be Available for Numerous Ionospheric Applications.