Jeff Anderson, NCAR Data Assimilation Research Section

Slides:



Advertisements
Similar presentations
Introduction to Data Assimilation NCEO Data-assimilation training days 5-7 July 2010 Peter Jan van Leeuwen Data Assimilation Research Center (DARC) University.
Advertisements

Uncertainty Representation. Gaussian Distribution variance Standard deviation.
Empirical Localization of Observation Impact in Ensemble Filters Jeff Anderson IMAGe/DAReS Thanks to Lili Lei, Tim Hoar, Kevin Raeder, Nancy Collins, Glen.
Nonlinear and Non-Gaussian Estimation with A Focus on Particle Filters Prasanth Jeevan Mary Knox May 12, 2006.
Machine Learning CUNY Graduate Center Lecture 3: Linear Regression.
Advanced data assimilation methods- EKF and EnKF Hong Li and Eugenia Kalnay University of Maryland July 2006.
Tracking with Linear Dynamic Models. Introduction Tracking is the problem of generating an inference about the motion of an object given a sequence of.
CSC2535: 2013 Advanced Machine Learning Lecture 3a: The Origin of Variational Bayes Geoffrey Hinton.
Binary Variables (1) Coin flipping: heads=1, tails=0 Bernoulli Distribution.
The impact of localization and observation averaging for convective-scale data assimilation in a simple stochastic model Michael Würsch, George C. Craig.
Ensemble Data Assimilation and Uncertainty Quantification Jeffrey Anderson, Alicia Karspeck, Tim Hoar, Nancy Collins, Kevin Raeder, Steve Yeager National.
EnKF Overview and Theory
Computational Issues: An EnKF Perspective Jeff Whitaker NOAA Earth System Research Lab ENIAC 1948“Roadrunner” 2008.
Federal Department of Home Affairs FDHA Federal Office of Meteorology and Climatology MeteoSwiss High-resolution data assimilation in COSMO: Status and.
Computer Vision - A Modern Approach Set: Tracking Slides by D.A. Forsyth The three main issues in tracking.
Probabilistic Robotics Bayes Filter Implementations.
Mean Field Variational Bayesian Data Assimilation EGU 2012, Vienna Michail Vrettas 1, Dan Cornford 1, Manfred Opper 2 1 NCRG, Computer Science, Aston University,
Mixture Models, Monte Carlo, Bayesian Updating and Dynamic Models Mike West Computing Science and Statistics, Vol. 24, pp , 1993.
Sophie RICCI CALTECH/JPL Post-doc Advisor : Ichiro Fukumori The diabatic errors in the formulation of the data assimilation Kalman Filter/Smoother system.
Computer Vision Group Prof. Daniel Cremers Autonomous Navigation for Flying Robots Lecture 6.1: Bayes Filter Jürgen Sturm Technische Universität München.
11 Background Error Daryl T. Kleist* National Monsoon Mission Scoping Workshop IITM, Pune, India April 2011.
Pg 1 A Loosely Coupled Ocean-Atmosphere Ensemble Assimilation System. Tim Hoar, Nancy Collins, Kevin Raeder, Jeffrey Anderson, NCAR Institute for Math.
Data assimilation and forecasting the weather (!) Eugenia Kalnay and many friends University of Maryland.
LEAST MEAN-SQUARE (LMS) ADAPTIVE FILTERING. Steepest Descent The update rule for SD is where or SD is a deterministic algorithm, in the sense that p and.
Deutscher Wetterdienst Vertical localization issues in LETKF Breogan Gomez, Andreas Rhodin, Hendrik Reich.
A kernel-density based ensemble filter applicable (?) to high-dimensional systems Tom Hamill NOAA Earth System Research Lab Physical Sciences Division.
Boosted Particle Filter: Multitarget Detection and Tracking Fayin Li.
The Unscented Particle Filter 2000/09/29 이 시은. Introduction Filtering –estimate the states(parameters or hidden variable) as a set of observations becomes.
Machine Learning CUNY Graduate Center Lecture 6: Linear Regression II.
A Random Subgrouping Scheme for Ensemble Kalman Filters Yun Liu Dept. of Atmospheric and Oceanic Science, University of Maryland Atmospheric and oceanic.
Pg 1 SAMSI UQ Workshop; Sep Ensemble Data Assimilation and Uncertainty Quantification Jeff Anderson National Center for Atmospheric Research.
Regularization of energy-based representations Minimize total energy E p (u) + (1- )E d (u,d) E p (u) : Stabilizing function - a smoothness constraint.
Future Directions in Ensemble DA for Hurricane Prediction Applications Jeff Anderson: NCAR Ryan Torn: SUNY Albany Thanks to Chris Snyder, Pavel Sakov The.
École Doctorale des Sciences de l'Environnement d’Île-de-France Année Universitaire Modélisation Numérique de l’Écoulement Atmosphérique et Assimilation.
École Doctorale des Sciences de l'Environnement d’ Î le-de-France Année Modélisation Numérique de l’Écoulement Atmosphérique et Assimilation.
PATTERN RECOGNITION AND MACHINE LEARNING CHAPTER 1: INTRODUCTION.
Jeffrey Anderson, NCAR Data Assimilation Research Section
Data Assimilation Research Testbed Tutorial
DEEP LEARNING BOOK CHAPTER to CHAPTER 6
Statistical Data Analysis - Lecture /04/03
Probability Theory and Parameter Estimation I
Using Sensor Data Effectively
Methods for dealing with spurious covariances arising from small samples in ensemble data assimilation Jeff Whitaker NOAA Earth.
Jeffrey Anderson, NCAR Data Assimilation Research Section
Data Assimilation Theory CTCD Data Assimilation Workshop Nov 2005
Line Fitting James Hayes.
Ch3: Model Building through Regression
Analyzing Redistribution Matrix with Wavelet
Jeff Anderson, NCAR Data Assimilation Research Section
Machine Learning Basics
Course: Autonomous Machine Learning
ECE 5424: Introduction to Machine Learning
Parallel Implementations of Ensemble Kalman Filters for Huge Geophysical Models Jeffrey Anderson, Helen Kershaw, Jonathan Hendricks, Nancy Collins, Ye.
Jeff Anderson, NCAR Data Assimilation Research Section
Statistical Methods For Engineers
Vertical localization issues in LETKF
FSOI adapted for used with 4D-EnVar
Where did we stop? The Bayes decision rule guarantees an optimal classification… … But it requires the knowledge of P(ci|x) (or p(x|ci) and P(ci)) We.
10701 / Machine Learning Today: - Cross validation,
Assimilation of Global Positioning System Radio Occultation Observations Using an Ensemble Filter in Atmospheric Prediction Models Hui Liu, Jefferey Anderson,
Generally Discriminant Analysis
Principles of the Global Positioning System Lecture 11
2. University of Northern British Columbia, Prince George, Canada
Introduction to Sensor Interpretation
Chapter 13 Additional Topics in Regression Analysis
Kalman Filter: Bayes Interpretation
Introduction to Sensor Interpretation
Sarah Dance DARC/University of Reading
Yalchin Efendiev Texas A&M University
Presentation transcript:

Jeff Anderson, NCAR Data Assimilation Research Section Exploiting Nonlinear Relations between Observations and State Variables in Ensemble Filters Jeff Anderson, NCAR Data Assimilation Research Section AMS, 9 January 2018

Schematic of a Sequential Ensemble Filter Use model to advance ensemble (3 members here) to time at which next observation becomes available. Ensemble state estimate after using previous observation (analysis) Ensemble state at time of next observation (prior)

Schematic of a Sequential Ensemble Filter Get prior ensemble sample of observation, y = h(x), by applying forward operator h to each ensemble member. Theory: observations from instruments with uncorrelated errors can be done sequentially. Can think about single observation without (too much) loss of generality.

Schematic of a Sequential Ensemble Filter Get observed value and observational error distribution from observing system.

Schematic of a Sequential Ensemble Filter Find the increments for the prior observation ensemble (this is a scalar problem for uncorrelated observation errors). Ensemble Kalman filters assume Gaussianity for this problem. Can compute increments without Gaussian assumptions. (Old news).

Schematic of a Sequential Ensemble Filter Find the increments for the prior observation ensemble (this is a scalar problem for uncorrelated observation errors). Students, faculty: Lots of cool problems still to be explored for non-Gaussian likelihoods. Contact me if interested.

Schematic of a Sequential Ensemble Filter Use ensemble samples of y and each state variable to linearly regress observation increments onto state variable increments. Theory: impact of observation increments on each state variable can be handled independently!

Schematic of a Sequential Ensemble Filter Use ensemble samples of y and each state variable to linearly regress observation increments onto state variable increments. Can solve this bivariate problem in other ways. Topic of this talk.

Schematic of a Sequential Ensemble Filter When all ensemble members for each state variable are updated, there is a new analysis. Integrate to time of next observation …

Focus on the Regression Step Standard ensemble filters just use bivariate sample regression to compute state increments. Dxi,n=bDyn, n=1,…N. N is ensemble size. b is regression coefficient. AMS, 9 January 2018

Focus on the Regression Step Will explore using different ‘regression’ for each ensemble member to compute increments for xi Dxi,n=bnDyn, n=1,…N. N is ensemble size. bn is ‘local’ regression coefficient. AMS, 9 January 2018

Nonlinear Regression Example Try to exploit nonlinear prior relation between a state variable and an observation. Example: Observation y~xn, for instance y=T4. AMS, 9 January 2018

Standard Ensemble Kalman Filter (EAKF) AMS, 9 January 2018

Nongaussian Filter (Rank Histogram Filter) AMS, 9 January 2018

Local Linear Regression Relation between observation and state is nonlinear. Try using ’local’ subset of ensemble to compute regression. What kind of subset? Cluster that contains ensemble member being updated. Lots of ways to define clusters. Here, use naïve closest neighbors in (x,y) space. Vary number of nearest neighbors in subset. AMS, 9 January 2018

Local Linear Regression Local ensemble subset is nearest ½ . Regression approximates local slope of the relation. AMS, 9 January 2018

Local Linear Regression Local ensemble subset is nearest ½ . Regression approximates local slope of the relation. Highlighted red increment uses least squares fit to ensemble members in region. AMS, 9 January 2018

Local Linear Regression Slope more accurate locally, but a disaster globally. Highlighted red increment uses least squares fit to ensemble members in region. AMS, 9 January 2018

Local Linear Regression Note similarity to Houtekamer’s method, except local ensemble members are used, rather than non-local. Highlighted red increment uses least squares fit to ensemble members in region. AMS, 9 January 2018

Local Linear Regression with Incremental Update Local slope is just that, local. Following it for a long way is a bad idea. Will use a Bayesian consistent incremental update. Observation with error variance s. Assimilate k observations with this value. Each of these has error variance s/k. AMS, 9 January 2018

Incremental Update This is an RHF update with 4 increments. Individual increments highlighted for two ensemble members. For an EAKF, posterior would be identical to machine precision. Nearly identical for RHF. AMS, 9 January 2018

Local Linear Regression with Incremental Update 2 increments with subsets ½ ensemble. Posterior for state qualitatively improving. AMS, 9 January 2018

Local Linear Regression with Incremental Update 4 increments with subsets ½ ensemble. Posterior for state qualitatively improving. AMS, 9 January 2018

Local Linear Regression with Incremental Update 8 increments with subsets ½ ensemble. Posterior for state qualitatively improving. AMS, 9 January 2018

Local Linear Regression with Incremental Update 8 increments with subset 1/4 ensemble. Posterior for state degraded. Increment is moving outside of local linear validity. AMS, 9 January 2018

Local Linear Regression with Incremental Update 16 increments with subset 1/4 ensemble. Posterior for state improved. AMS, 9 January 2018

Local Linear Regression with Incremental Update If relation between observation and state is locally a continuous, smooth (first two derivatives continuous) function: Then, in the limit of a large ensemble, fixed local subset size, and large number of increments: The local linear regression with incremental update converges to the correct posterior distribution. AMS, 9 January 2018

Local Linear Regression with Incremental Update If relation between observation and state is locally a continuous, smooth (first two derivatives continuous) function: Then, in the limit of a large ensemble, fixed local subset size, and large number of increments: The local linear regression with incremental update converges to the correct posterior distribution. This could be very expensive, No guarantees about what goes on in the presence of noise. AMS, 9 January 2018

Multi-valued, not smooth example. Similar in form to a wind speed observation with state velocity component. AMS, 9 January 2018

Multi-valued, not smooth example. Standard regression does not capture bimodality of state posterior. AMS, 9 January 2018

Multi-valued, not smooth example. Local regression with ½ of the ensemble does much better. Captures bimodal posterior. Note problems where relation is not smooth. AMS, 9 January 2018

Multi-valued, not smooth example. Local regression with1/4 of the ensemble does even better. No need for incremental updates here. AMS, 9 January 2018

Discontinuous example. Example of threshold process. Like measurement of rainfall rate and state variable of low-level temperature. AMS, 9 January 2018

Discontinuous example. Basic RHF does well on this, but it’s an ’accident’. AMS, 9 January 2018

Discontinuous example. Local regression with the two obvious clusters leads to ridiculous posterior. Lack of continuity of bivariate prior makes any regression-like update problematic. AMS, 9 January 2018

Local Regression with Noisy Priors Most geophysical applications have noisy bivariate priors. Usually hard to detect nonlinearity (even this example is still pretty extreme). AMS, 9 January 2018

Local Regression with Noisy Priors Basic RHF can still be clearly suboptimal. AMS, 9 January 2018

Local Regression with Noisy Priors This result is for local ensemble with nearest ½ of ensemble and 8 increments. Need bigger local ensembles to reduce sampling errors. AMS, 9 January 2018

Local Regression with Noisy Priors This result is for local ensemble with nearest 1/4 of ensemble and 8 increments. The small ensemble subsets lead to large sampling error. Probably worse than standard RHF. AMS, 9 January 2018

Results: Lorenz96 Standard model configuration, perfect model. Observation at location of each of the state variables. Two observation frequencies: every step, every 12 steps. Three observation types: state, square of state, log of state. Observation error variance tuned to give time mean RMSE of approximately 1.0 for basic RHF. 80 total ensemble members. For local regression: Local subsets of 60 members (pretty large). 4 increments. Tuned adaptive inflation standard deviation, localization for lowest RMSE in base RHF case. AMS, 9 January 2018

Results: Lorenz96 Observations of state: Results from base RHF and local regression statistically indistinguishable. Observations of square of state: Results from local regression are significantly better, but filter suffers occasional catastrophic divergence. Must be restarted in these cases. Observations of log of state: Results from local regression are significantly better. AMS, 9 January 2018

Low-Order Dry Dynamical Core Slide 7: GPS occultation forward operators Non-local refractivity : (Sokolovskiy et al., MWR 133, 2200-2212) NEED DETAILS OF ALGORITHM step size??? upper limits grid pictures for different latitudes Three pictures exist: ~jla/talks_and_meetings/2009/ams/cam_gps_talk/gps_lat* Expect minor differences with coarse grid, larger near poles Evolution of surface pressure field every 12 hours. Has baroclinic instability: storms move east in midlatitudes. AMS, 9 January 2018

Conclusions Sequential ensemble filters can: Apply non-Gaussian methods in observation space, Nonlinear methods for bivariate regression. Local regression with incremental update can be effective for locally smooth, continuous relations. Useful for: Nonlinear forward operators, Transformed state variables (log, anamorphosis, …). Can be expensive for ’noisy’ bivariate priors: Requires large subsets (hence large ensembles), Subsets can be found efficiently, Incremental update is a multiplicative cost. Slide 7: GPS occultation forward operators Non-local refractivity : (Sokolovskiy et al., MWR 133, 2200-2212) NEED DETAILS OF ALGORITHM step size??? upper limits grid pictures for different latitudes Three pictures exist: ~jla/talks_and_meetings/2009/ams/cam_gps_talk/gps_lat* Expect minor differences with coarse grid, larger near poles AMS, 9 January 2018

Conclusions Hard to beat standard regression for many applications. Being smart about when to apply nonlinear methods is key. Detect nonlinear bivariate priors automatically. Apply appropriate methods. Multivariate relations may still be tricky. Slide 7: GPS occultation forward operators Non-local refractivity : (Sokolovskiy et al., MWR 133, 2200-2212) NEED DETAILS OF ALGORITHM step size??? upper limits grid pictures for different latitudes Three pictures exist: ~jla/talks_and_meetings/2009/ams/cam_gps_talk/gps_lat* Expect minor differences with coarse grid, larger near poles AMS, 9 January 2018

All results here with DARTLAB tools freely available in DART. www.image.ucar.edu/DAReS/DART Anderson, J., Hoar, T., Raeder, K., Liu, H., Collins, N., Torn, R., Arellano, A., 2009: The Data Assimilation Research Testbed: A community facility. BAMS, 90, 1283—1296, doi: 10.1175/2009BAMS2618.1 AMS, 9 January 2018