1. Open Source Ensemble Kalman Filtering: the Data Assimilation Research Testbed - DART Tim Hoar, Jeffrey Anderson, Nancy Collins, Kevin Raeder, Hui Liu, Glen Romine NCAR Institute for Mathematics Applied to Geophysics

2. Sunday Afternoon (4 hours): Introductions / making teams Configure environment DART WWW-site Download DART DART_LAB – Matlab-based exercises to learn DA Monday Morning (4 hours): Recap of yesterday / questions Select chapters of the DART tutorial Moving from toy models to large models Monday Afternoon (4 hours): Diagnostics Testing Strategies … 1 observation, please. Real Observations CLM 10 years in 12 hours CAHMDA-V July 2012 pg 2

3. Introductions: I need to know where to start and what to cover. As you introduce yourself, please let me know how much experience you have with each of the following: Unix/Linux command line Shell programming (i.e. csh, ksh, sh) vi / emacs / kedit / … Matlab / NCL / IDL / NCO / R Ensemble Data Assimilation Theory I cannot give individual attention to this many people during these exercises. We will need to make teams of 3 that will need to work together during this tutorial. We need to make teams that have some experience in each of the skills listed above. When I tell you to go stand in a corner, do not take it personally! I mean no disrespect! Sunday Afternoon CAHMDA-V July 2012 pg 3

4. Your $HOME directory … Customizations … ‘dotfiles’ Compilers, Libraries, inconsistencies … Your shell … csh, sh, bash, ksh, tcsh … Your commands … $PATH, aliases … Your favorite $EDITOR Remote logins, X forwarding … Windows Batch jobs Being nice – lots of people on one machine – “top” Being graceful – orphaned processes Configuring your *nix environment on “x50” CAHMDA-V July 2012 pg 4

5. The most useful (to me) pull-down menus: Getting Started Documentation Diagnostics Miscellany: Platform-specific Notes DART “home page” http://www.image.ucar.edu/DAReS/DART CAHMDA-V July 2012 pg 5

6. Register for the DART code – really. Actually download the code we will cheat “svn” - making modifications with NO FEAR DART file tree / schematic DART documentation DART tutorial Review DART interface requirements … DART build mechanism – mkmf DART_LAB Download DART CAHMDA-V July 2012 pg 6

7. … to produce an analysis. What is Data Assimilation? + Observations combined with a Model forecast… = Overview article of DART: Anderson, Jeffrey, T. Hoar, K. Raeder, H. Liu, N. Collins, R. Torn, A. Arellano, 2009: The Data Assimilation Research Testbed: A Community Facility. Bull. Amer. Meteor. Soc., 90, 1283–1296. doi:10.1175/2009BAMS2618.1doi:10.1175/2009BAMS2618.1 CAHMDA-V July 2012 pg 7

8. Ensemble Filter for Large Geophysical Models 1. Use model to advance ensemble (3 members here) to time at which next observation becomes available. Ensemble state estimate after using previous observation (analysis) Ensemble state at time of next observation (prior) CAHMDA-V July 2012 pg 8

9. 2. Get prior ensemble sample of observation, y = h(x), by applying forward operator h to each ensemble member. Theory: observations from instruments with uncorrelated errors can be done sequentially. Ensemble Filter for Large Geophysical Models CAHMDA-V July 2012 pg 9

10. 3. Get observed value and observational error distribution from observing system. Ensemble Filter for Large Geophysical Models CAHMDA-V July 2012 pg 10

11. 4. Find the increments for the prior observation ensemble (this is a scalar problem for uncorrelated observation errors). Note: Difference between various ensemble filters is primarily in observation increment calculation. Ensemble Filter for Large Geophysical Models CAHMDA-V July 2012 pg 11

12. 5. Use ensemble samples of y and each state variable to linearly regress observation increments onto state variable increments. Theory: impact of observation increments on each state variable can be handled independently! Ensemble Filter for Large Geophysical Models CAHMDA-V July 2012 pg 12

13. 6. When all ensemble members for each state variable are updated, there is a new analysis. Integrate to time of next observation … Ensemble Filter for Large Geophysical Models CAHMDA-V July 2012 pg 13

14. DART_LAB: Matlab-based tutorial DART/DART_LAB/presentations Section 1 … the 1D perspective Section 2 … impacting an unobserved state variable Section 3 … sampling error and localization Section 4 … perturbed observations (EnKF) DART/DART_LAB/matlab Section 1 … gaussian_product, oned_ensemble, oned_model Section 2 … twod_ensemble, run_lorenz_63, run_lorenz_96 Section 3 … run_lorenz_96 Section 4 … oned_ensemble, twod_ensemble, oned_model, run_lorenz_63 and run_lorenz_96 all allow selection of EnKF. I’m going to focus on CAHMDA-V July 2012 pg 14

15. A generic ensemble filter system like DART just needs: 1. A way to make model forecasts; 2. A way to compute forward operators, h. Ensemble Filter for Large Geophysical Models CAHMDA-V July 2012 pg 15

16. That’s all for today …. CAHMDA-V July 2012 pg 16

17. Introductions and lies … which of these is true about me? “I have seen all 7 continents.” “I am a competitive square dancer.” Monday morning CAHMDA-V July 2012 pg 17

18. Questions from yesterday netCDF : ncdump, ncview Matlab customizations for DART … DART tutorial (in the interest of time, we’re skipping a lot – at your leisure go back and be complete) Sections 1,2,3 in their entirety (but quickly) Section 4: skip to pg 29 Section 5: only pg 15 Section 7: introduces lorenz_96 (L96) pg 10 Section 8: sampling error (L96) pgs 10,13 Section 9: inflation (L96) after pg 15 Section 11: building DART Section 14: observation quality control Section 18: not knowing the truth Scripting for standalone executables – large models. Monday morning CAHMDA-V July 2012 pg 18

19. Diagnostics State-space (useful if you know the truth) Observation-space (useful in general) link_obs example with dev/POP obs_diag example wth dev/WRF Testing Strategies Lorenz_96 – perfect model experiment Observation sequence file creation Ameriflux data CLM/DART CESM multi-instance facility Modifying the run script CLM variable specification – picking a state vector Adding new DART kinds/types Observation operators Monday Afternoon CAHMDA-V July 2012 pg 19

20. Reasons to NOT reinvent the wheel DART has proven methods to address the most common (and some not-so-common) issues affecting the performance of ensemble filters: Inflation Anderson, J. L., 2009 Spatially and temporally varying adaptive covariance inflation for ensemble filters. Tellus A, 61, 72-83 doi:10.1111/j.1600-0870.2008.00361.xdoi:10.1111/j.1600-0870.2008.00361.x Anderson, J. L., 2007. An adaptive covariance inflation error correction algorithm for ensemble filters. Tellus A, 59, 210-224. doi:10.1111/j.1600-0870.2006.00216.xdoi:10.1111/j.1600-0870.2006.00216.x Novel algorithms Anderson, J.L., 2010 A Non-Gaussian Ensemble Filter Update for Data Assimilation. Monthly Weather Review, 138, 4186-4198, doi:10.1175/2010MWR3253.1doi:10.1175/2010MWR3253.1 Anderson, J. L., 2007 Exploring the need for localization in ensemble data assimilation using a hierarchical ensemble filter. Physica D, 230, 99-111. doi:10.1016/j.physd.2006.02.011doi:10.1016/j.physd.2006.02.011 Parallelization Anderson, J., Collins, N., 2007. Scalable Implementations of Ensemble Filter Algorithms for Data Assimilation. Journal of Atmospheric and Oceanic Technology, 24, 1452-1463. doi:10.1175/JTECH2049.1doi:10.1175/JTECH2049.1 CAHMDA-V July 2012 pg 20

21. More reasons to NOT reinvent the wheel Diagnostics Many routines/methods are provided to explore the performance of the assimilation. Native ability to explore ‘value’ of different types of observations Immediate ability to perform ‘perfect model’ experiments Documentation Each code module has a companion HTML document to describe its use and purpose. http://www.image.ucar.edu/DAReS/DART All documentation/code available online Workshop materials Self-paced tutorials included in the download Portable, tested on many platforms Free, open source. Too many platforms/compilers to bother listing. Distributed and maintained with subversion. Can exploit, but does not need MPI. Humans! Tim, Nancy, Jeff, Kevin, Hui, Glen …all reached at dart@ucar.edu CAHMDA-V July 2012 pg 21

22. Creating the initial ensemble of CLM. model time “a long time” “spun up” Replicate what we have N times. Use a unique (and different!) realistic DATM for each. Run them forward for “a long time”. Getting a proper initial ensemble is an area of active research. We don’t know how much spread we NEED to capture the uncertainty in the system. CAHMDA-V July 2012 pg 22

23. The ensemble advantage. In a free run, the ensemble spread frequently grows. With a good assimilation: ensemble spread ultimately remains stable and small enough to be informative You can represent uncertainty. observation times CAHMDA-V July 2012 pg 23

24. Assimilation uses 80 members of 2 o FV CAM forced by a single ocean (Hadley+ NCEP-OI2) and produces a very competitive reanalysis. O(1 million) atmospheric obs are assimilated every day. 500 hPa GPH Feb 17 2003 Contours 5200m:5700m by 100 1998-2010 4x daily is free and available. Contact dart@ucar.edu 1998-2010 4x daily is free and available. Contact dart@ucar.edu Atmospheric Reanalysis CAHMDA-V July 2012 pg 24

25. http://www.image.ucar.edu/DAReS/DART dart@ucar.edu Code to implement all of the algorithms discussed is freely available from: CAHMDA-V July 2012 pg 25

27. DART is used at: Public domain software for ensemble Data Assimilation –Well-tested, portable, scalable, extensible, free! Models –Toy to HUGE Observations –Real, synthetic, novel An extensive Tutorial –With examples, exercises, explanations People: The DAReS Team 43 UCAR member universities More than 100 other sites

28. Moving towards coupled assimilation for earth system models. Tim Hoar, Nancy Collins, Kevin Raeder, Jeffrey Anderson, NCAR Institute for Math Applied to Geophysics Data Assimilation Research Section Steve Yeager, Mariana Vertenstein, Gokhan Danabasoglu, Alicia Karspeck, and Joe Tribbia NCAR/NESL/CGD/Oceanography

29. Hypothesis: Need Ensemble of Atmospheres to Force Ensemble Assimilation for Ocean 500 hPa GPH Feb 17 2003 Case 1: 23 POP members forced by a single atmosphere. Case 2: 48 POP members forced by 48 CAM/DART analyses. Case 2 Generates additional ocean spread, improved analyses.

30. Obs DART POP Coupler 2D forcing 3D restart 3D state DATM 2D forcing from CAM assimilation Current POP Assimilation from within the Climate Earth System Model - CESM

31. FLOAT_SALINITY 68200 FLOAT_TEMPERATURE395032 DRIFTER_TEMPERATURE33963 MOORING_SALINITY 27476 MOORING_TEMPERATURE 623967 BOTTLE_SALINITY 79855 BOTTLE_TEMPERATURE 81488 CTD_SALINITY 328812 CTD_TEMPERATURE 368715 STD_SALINITY 674 STD_TEMPERATURE 677 XCTD_SALINITY 3328 XCTD_TEMPERATURE 5790 MBT_TEMPERATURE 58206 XBT_TEMPERATURE 1093330 APB_TEMPERATURE 580111 World Ocean Database T,S observation counts These counts are for 1998 & 1999 and are representative. temperature observation error standard deviation == 0.5 K. salinity observation error standard deviation == 0.5 msu.

32. Ensemble Spread for Pacific 100m XBT Small spread! Twice as much! Spread of the “climatological” ensemble

33. 100m Mooring Temperature RMSE – Pacific POP/CAM as good or better RMSE

34. 23 POP 1 DATM Coupled Free Run POP forced by observed atmosphere (hindcast) 48 POP 48 CAM Physical Space: 1998/1999 SST Anomaly from HadOI-SST

35. Coupler CAM DART Obs POP CLM CICE Fully coupled assimilation will need data from all models at the same time This is a very CESM-centric view of fully coupled data assimilation.

36. Global Atmosphere models: CAMCommunity Atmosphere ModelNCAR CAM/CHEMCAM with ChemistryNCAR WACCMWhole Atmosphere CommunityNCAR Climate Model AM2 Atmosphere Model 2NOAA/GFDL NOGAPSNavy Operational Global US Navy Atmospheric Prediction System ECHAMEuropean Centre Hamburg ModelHamburg Planet WRFGlobal version of WRFJPL MPASModel for Prediction AcrossNCAR/DOE Scales (under development) DART works with many geophysical models

37. Regional Atmosphere models: WRF/ARWWeather Research and NCAR Forecast Model WRF/CHEMWRF with ChemistryNCAR NCOMMAS Collaborative Model for NOAA/NSSL Multiscale Atmospheric Simulation COAMPSCoupled Ocean/Atmosphere US Navy Mesoscale Prediction System CMAQCommunity Multi-scale Air QualityEPA COSMOConsortium for Small-Scale DWD Modeling DART works with many geophysical models

38. Ocean models: POPParallel Ocean ProgramDOE/NCAR MIT OGCMOcean General Circulation MIT Model ROMSRegional Ocean Modeling Rutgers System (under development) MPASModel for Prediction AcrossDOE/LANL Scales (under development) Land Surface models: CLMCommunity Land Model NCAR (under development) DART works with many geophysical models

39. Upper Atmosphere/Space Weather models: ROSENCAR TieGCMThermosphere IonosphereNCAR/HAO Electrodynamic GCM GITMGlobal Ionosphere Thermosphere ModelMichigan