Evaluate Earth Science Models for What They are - Cartoons of Reality John S. Irwin, NOAA Meteorologist EPA OAQPS Air Quality Modeling Group RTP, NC 27711.

Slides:



Advertisements
Similar presentations
Review bootstrap and permutation
Advertisements

Running a model's adjoint to obtain derivatives, while more efficient and accurate than other methods, such as the finite difference method, is a computationally.
MOPITT CO Louisa Emmons, David Edwards Atmospheric Chemistry Division Earth & Sun Systems Laboratory National Center for Atmospheric Research.
Analysis of 12 years of IMPROVE data in the Columbia River Gorge By Dan Jaffe University of Washington Northwest Air Quality Photo from the Wishram IMPROVE.
1 Multiple Frame Surveys Tracy Xu Kim Williamson Department of Statistical Science Southern Methodist University.
Halûk Özkaynak US EPA, Office of Research and Development National Exposure Research Laboratory, RTP, NC Presented at the CMAS Special Symposium on Air.
Comparative Analysis of Statistical Tools To Identify Recruitment-Environment Relationships and Forecast Recruitment Strength Bernard A. Megrey Yong-Woo.
Distribution of NO 2 concentrations over shooting (400 µg/m 3 per 1 hour) calculated with POLAIR dispersion model using (2004) NO 2 concentrations from.
An initial linkage of the CMAQ modeling system at neighborhood scales with a human exposure model Jason Ching/Thomas Pierce Air-Surface Processes Modeling.
Markus Amann The RAINS model: Modelling of health impacts of PM and ozone.
Marakas: Decision Support Systems, 2nd Edition © 2003, Prentice-Hall Chapter Chapter 4: Modeling Decision Processes Decision Support Systems in the.
Chapter 12 - Forecasting Forecasting is important in the business decision-making process in which a current choice or decision has future implications:
Bootstrap in Finance Esther Ruiz and Maria Rosa Nieto (A. Rodríguez, J. Romo and L. Pascual) Department of Statistics UNIVERSIDAD CARLOS III DE MADRID.
2008 Chingchun 1 Bootstrap Chingchun Huang ( 黃敬群 ) Vision Lab, NCTU.
Chapter 14 Simulation. Monte Carlo Process Statistical Analysis of Simulation Results Verification of the Simulation Model Computer Simulation with Excel.
Data Handling l Classification of Errors v Systematic v Random.
Jenny Stocker, Christina Hood, David Carruthers, Martin Seaton, Kate Johnson, Jimmy Fung The Development and Evaluation of an Automated System for Nesting.
Investigating the long- and short-term variations in meteorology in Atlanta Lucas Henneman 25 April, 2013.
Shaocai Yu *, Brian Eder* ++, Robin Dennis* ++, Shao-hang Chu**, Stephen Schwartz*** *Atmospheric Sciences Modeling Division, National Exposure Research.
Using Air Quality Models for Emissions Management Decisions
Ideas on Updating Model Evaluation Methods by John S. Irwin EPA Eight Conference on Air Quality Modeling September 22-23, 2005 US.
1 Panel Session on Uncertainty in Atmospheric Transport and Dispersion Models Moderator Steven R. Hanna Harvard School of Public Health
Ensemble-variational sea ice data assimilation Anna Shlyaeva, Mark Buehner, Alain Caya, Data Assimilation and Satellite Meteorology Research Jean-Francois.
CMAS Conference, October 16 – 18, 2006 The work presented here was performed by the New York State Department of Environmental Conservation with partial.
G O D D A R D S P A C E F L I G H T C E N T E R 1 Global Precipitation Measurement (GPM) GV Data Exchange Protocol Mathew Schwaller GPM Formulation Project.
Earth System Sciences, LLC Suggested Analyses of WRAP Drilling Rig Databases Doug Blewitt, CCM 1.
STEM Fair Graphs & Statistical Analysis. Objectives: – Today I will be able to: Construct an appropriate graph for my STEM fair data Evaluate the statistical.
Slide 1 Estimating Performance Below the National Level Applying Simulation Methods to TIMSS Fourth Annual IES Research Conference Dan Sherman, Ph.D. American.
Sensitivity of top-down correction of 2004 black carbon emissions inventory in the United States to rural-sites versus urban-sites observational networks.
CSDA Conference, Limassol, 2005 University of Medicine and Pharmacy “Gr. T. Popa” Iasi Department of Mathematics and Informatics Gabriel Dimitriu University.
Part III Gathering Data.
Performance of Resampling Variance Estimation Techniques with Imputed Survey data.
Impacts of Biomass Burning Emissions on Air Quality and Public Health in the United States Daniel Tong $, Rohit Mathur +, George Pouliot +, Kenneth Schere.
Long term and high resolution observations of climate altering halogenated gases are carried out in several research stations worldwide distributed in.
Copyright  2003 by Dr. Gallimore, Wright State University Department of Biomedical, Industrial Engineering & Human Factors Engineering Human Factors Research.
CEN st Lecture CEN 4021 Software Engineering II Instructor: Masoud Sadjadi Monitoring (POMA)
Office of Research and Development National Exposure Research Laboratory, Atmospheric Modeling and Analysis Division Office of Research and Development.
Introduction to Modeling – Part I Sarah Kelly ITEP.
1 Modeling Under PSD Air quality models (screening and refined) are used in various ways under the PSD program. Step 1: Significant Impact Analysis –Use.
Operations Research The OR Process. What is OR? It is a Process It assists Decision Makers It has a set of Tools It is applicable in many Situations.
- 1 - Overall procedure of validation Calibration Validation Figure 12.4 Validation, calibration, and prediction (Oberkampf and Barone, 2004 ). Model accuracy.
The Impact of Short-term Climate Variations on Predicted Surface Ozone Concentrations in the Eastern US 2020 and beyond Shao-Hang Chu and W.M. Cox US Environmental.
Evaluation of harvest control rules (HCRs): simple vs. complex strategies Dorothy Housholder Harvest Control Rules Workshop Bergen, Norway September 14,
First Define the Information Needed, Then Select a Course of Action John S. Irwin, NOAA EPA, OAQPS Air Quality Modeling Group RTP, NC
ICDC7, Boulder September 2005 Estimation of atmospheric CO 2 from AIRS infrared satellite radiances in the ECMWF data assimilation system Richard.
Evaluating temporal and spatial O 3 and PM 2.5 patterns simulated during an annual CMAQ application over the continental U.S. Evaluating temporal and spatial.
Types of Models Marti Blad Northern Arizona University College of Engineering & Technology.
Unscented Kalman Filter (UKF) CSCE 774 – Guest Lecture Dr. Gabriel Terejanu Fall 2015.
Designing Factorial Experiments with Binary Response Tel-Aviv University Faculty of Exact Sciences Department of Statistics and Operations Research Hovav.
- 1 - Calibration with discrepancy Major references –Calibration lecture is not in the book. –Kennedy, Marc C., and Anthony O'Hagan. "Bayesian calibration.
Operations Research Models and Methods Advanced Operations Research Reference: Operations Research Models and Methods, Operations Research Models and Methods,
Diagnostic verification and extremes: 1 st Breakout Discussed the need for toolkit to build beyond current capabilities (e.g., NCEP) Identified (and began.
Impact of Temporal Fluctuations in Power Plant Emissions on Air Quality Forecasts Prakash Doraiswamy 1, Christian Hogrefe 1,2, Eric Zalewsky 2, Winston.
11/04/091 Some Topics Concerning The COSYSMOR Model/Tool John E. Gaffney, Jr Center For Process Improvement Excellence.
Ideas on Selection and Approval of Models for EPA Regulatory Use by John S. Irwin EPA Eighth Conference on Air Quality Modeling September.
Jonathan Pleim NOAA/ARL* RTP, NC A NEW COMBINED LOCAL AND NON-LOCAL PBL MODEL FOR METEOROLOGY AND AIR QUALITY MODELING * In Partnership with the U.S. Environmental.
Introduction to Modeling – Part I Sarah Kelly ITEP Sarah Kelly ITEP.
1 THE AERMOD MODELING SYSTEM AN OVERVIEW FOR THE 8 TH MODELING CONFERENCE SEPTEMBER 22, 2005.
EPA’s 8 th Conference on Air Quality Modeling Comments on Model Evaluation By Bob Paine, ENSR (Peer reviewed by the A&WMA AB-3 Committee)
Daiwen Kang 1, Rohit Mathur 2, S. Trivikrama Rao 2 1 Science and Technology Corporation 2 Atmospheric Sciences Modeling Division ARL/NOAA NERL/U.S. EPA.
Upper Rio Grande R Basin
Types of Models Marti Blad PhD PE
Global Impacts and Consequences of Climate Change
Process Capability and Capability Index
Statistical Methods for Model Evaluation – Moving Beyond the Comparison of Matched Observations and Output for Model Grid Cells Kristen M. Foley1, Jenise.
Two Decades of WRF/CMAQ simulations over the continental U. S
Predicting Future-Year Ozone Concentrations: Integrated Observational-Modeling Approach for Probabilistic Evaluation of the Efficacy of Emission Control.
STEM Fair Graphs & Statistical Analysis
Suggested Analyses of WRAP Drilling Rig Databases
Business Statistics - QBM117
Presentation transcript:

Evaluate Earth Science Models for What They are - Cartoons of Reality John S. Irwin, NOAA Meteorologist EPA OAQPS Air Quality Modeling Group RTP, NC 27711

Model Evaluation Background Linear regressions (Clarke, 1964), Marin (1971). EPA Guideline on Air Quality Models, EPA-450/ , (OAQPS, 1978), Revised (1980, 1994, proposed 2001). National Commission on Air Quality Panel examines uses and limitations of air quality models, (Fox and Fairobent, 1981) BAMS(62): September, 1980: Woods Hole Workshop. Judging air quality model performance, (Fox, 1981) BAMS (62): Framework for evaluating air quality models, (Venkatram, 1982) BLM (24):

Model Evaluation Background(cont.) 1984 –Uncertainty in air quality modeling, (Fox, 1984) BAMS (65): –Review of the attributes and performance of 10 rural diffusion models, (Smith, 1984) BAMS (65): –Potentially useful additions to the rural model performance evaluation, (Irwin and Smith) BAMS (65): : Air quality model evaluation and uncertainty (Hanna, 1988) JAPCA (38): : Confidence limits for air quality model evaluations as estimated by bootstrap and Jackknife resampling methods. (Hanna, 1989) AE(23):

Model Evaluation Background(cont.) 1990: A statistical procedure for determination of the best performing air quality simulation model, (Cox and Tikvart) AE(24A): : Verification, validation, and confirmation of numerical models in the earth sciences. (Oreskes et. al., 1994): Science (263): : Standard Guide for statistical evaluation of atmospheric dispersion model performance, ASTM, D , 16 pages.

Examples Examples: –Gaussian Plume (Irwin et al., 1987) –Grid Model (Hanna et al., 1998, 2001) –Transport Direction (Weil et al., 1992)

Computed correlation coefficints (r) for Run 26 Green Glow, PST, August 28, 1959.

Unexplained Variability & Simple versus Complex Modeling

Evaluation

My answer: We need an evaluation procedure, that uses statistical evaluation test methods to: Determine which of several models in performing best (for the data available) the physics within the model: –This will be defined as the “best performer.” Determine whether the differences seen in the performance of the other models is statistically significant (in light of stochastic variations present in the data comparisons). –This will identify other models that may be performing as well (a set of ‘best’ performers). I call this the Olympics “High Bar”Analogy. This is an open (hopefully fair) competition, in which the rules are known and the conclusions reached are objectively determined.

My answer: (cont.) Once the ‘set of best performers’ has been defined, then a new set of statistical evaluation test methods would be used to determine, which of these models best performs the user-defined tasks. –Again, define the ‘best’ performer. –Then, test to see if differences in performance are statistically significant. This sequence recognizes that models are used for situations for which they do not have the requisite physics. If you test ONLY for the user defined tasks, you likely will end up with perverted models whose results are “tuned” to the data at hand, that may well provide erroneous results when used operationally. Example: Pasquill dispersion sigmas have a 3-minute averaging time. They are tested in their ability to replicate 1-hr, 3-hr concentration extremes, and then applied to produce annual averages.

Promising Test methods: Grouped Data: –(Irwin and Smith, 1984), ASTM (2002). Decomposed Time Series: –Eskridge, R.E., Ku, J.Y., Rao, S.T., Porter,P.S., Zurbenko, I.G., (1997)BAMS (78): –Rao, S.T., Zurbenko, I.G., Neagu, R., Porter, P.S., Ku, J.Y., Henry, R.F., (1997), BAMS, (78): Process Analysis: –Dennis R.L. (1986), Air Pollution Modeling and its Application V, Plenum Press, pp –Dennis, R.L., Arnold, J.R., Tonnesen, G.S., Y. Li (1999): Computer Physics Communications, 117:

All Models of Physical Processes are Cartoons of Reality Models Simulate Only a Portion of the Natural Variability. They Do Not Simulate What Is Directly Seen. FIRST: Test a Model to Accurately Perform the Physics Within It THEN: Test a Model to Perform Some User-defined Task (Which More Often Then Not Is Beyond the Capabilities of the Physics Within the Model). All “test methods” should provide a test of whether differences between several models are statistically significant. All “test methods” and test data sets should be peer reviewed and public domain.