Atmospheric Data Analysis on the Grid Kevin Hodges ESSC Co-workers: Brian Hoskins, Lennart Bengtsson Lizzie Froude.

Slides:



Advertisements
Similar presentations
Sharing Experiences in Operational Consensus Track Forecasting Rapporteur: Andrew Burton Team members: Philippe Caroff, James Franklin, Ed Fukada, T.C.
Advertisements

ECMWF June 2006Slide 1 Access to ECMWF data for Research Manuel Fuentes Data and Services Section, ECMWF ECMWF Forecast Products User Meeting.
Jennifer Catto Supervisors: Len Shaffrey – NCAS Climate and Kevin Hodges - ESSC The Representation of Extratropical Cyclones in HiGEM.
The Reading e-Science Centre Jon Blower Reading e-Science Centre Environmental Systems Science Centre University of Reading United Kingdom.
Numerical Weather Prediction Models
A Unified Data Model and Programming Interface for Working with Scientific Data Doug Lindholm Laboratory for Atmospheric and Space Physics University of.
Setting up of condor scheduler on computing cluster Raman Sehgal NPD-BARC.
Extra-tropical Cyclones in Recent Re-Analyses Kevin Hodges Lennart Bengtsson and Robert Lee 1.
Part 1a: Overview of the UM system
NCAS Unified Model Introduction Part 1a: Overview of the UM system University of Reading, 3-5 December 2014.
16 January 2005 Lennart Bengtsson Celsius lecture 2005 Uppsala Unversity The modelling of the climate system Professor Lennart Bengtsson ESSC, University.
22 April 2005 Oslo Met. Institutt Storm tracks and Climate change Lennart Bengtsson Changes in extra-tropical and tropical storms in the 21st century Professor.
28 August 2006Steinhausen meeting Hamburg On the integration of weather and climate prediction Lennart Bengtsson.
The International Surface Pressure Databank (ISPD) and Twentieth Century Reanalysis at NCAR Thomas Cram - NCAR, Boulder, CO Gilbert Compo & Chesley McColl.
‘Dynamically simulated tropic storms in a changing climate and their impact on the assessment of future climate risk’ - PhD project Ray Bell Supervisors.
9 July 2008, COLA, Washington ETC in a warmer climate? Lennart Bengtsson Extra-tropical cyclones in a warmer climate. Will they be more intense? Professor.
Introduction Downloading and sifting through large volumes of data stored in differing formats can be a time-consuming and sometimes frustrating process.
Grid for Coupled Ensemble Prediction (GCEP) Keith Haines, William Connolley, Rowan Sutton, Alan Iwi University of Reading, British Antarctic Survey, CCLRC.
Larry Marx and the Project Athena Team. Outline Project Athena Resources Models and Machine Usage Experiments Running Models Initial and Boundary Data.
Climate Forecasting Unit Prediction of climate extreme events at seasonal and decadal time scale Aida Pintó Biescas.
High Resolution Climate Modelling in NERC (and the Met Office) Len Shaffrey, University of Reading Thanks to: Pier Luigi Vidale, Jane Strachan, Dave Stevens,
03/27/2003CHEP20031 Remote Operation of a Monte Carlo Production Farm Using Globus Dirk Hufnagel, Teela Pulliam, Thomas Allmendinger, Klaus Honscheid (Ohio.
Pursuing Faster I/O in COSMO POMPA Workshop May 3rd 2010.
Update on Storm Surge at NCEP Dr. Rick Knabb, Director, National Hurricane Center and representing numerous partners 21 January 2014.
What is a Climate Model?.
10-14 Aug 2009, ICTP, Trieste Workshop on "High-Resolution Climate Modeling" Lennart Bengtsson Tropical and extra-tropical cyclones in high resolution.
Barcelona, 06 May 2015 s2dverification Seasonal to decadal forecast verification in R Overview Nicolau Manubens.
Recent Advances in Climate Extremes Science AVOID 2 FCO-Roshydromet workshop, Moscow, 19 th March 2015 Simon Brown, Met Office Hadley Centre.
© Crown copyright Met Office Plans for Met Office contribution to SMOS+STORM Evolution James Cotton & Pete Francis, Satellite Applications, Met Office,
NHC Activities, Plans, and Needs HFIP Diagnostics Workshop August 10, 2012 NHC Team: David Zelinsky, James Franklin, Wallace Hogsett, Ed Rappaport, Richard.
The New Zealand Institute for Plant & Food Research Limited Use of Cloud computing in impact assessment of climate change Kwang Soo Kim and Doug MacKenzie.
Data Discovery and Access to The International Surface Pressure Databank (ISPD) 1 Thomas Cram Gilbert P. Compo* Doug Schuster Chesley McColl* Steven Worley.
Course Evaluation Closes June 8th.
30 April 2008 MPI, Hamburg ETC in a warmer climate? Lennart Bengtsson Extra-tropical cyclones in a warmer climate. Will they be more intense? Professor.
Peter Knippertz et al. – Uncertainties of climate projections of severe European windstorms European windstorms Knippertz, Marsham, Parker, Haywood, Forster.
Lennart Bengtsson ESSC, Uni. Reading THORPEX Conference December 2004 Predictability and predictive skill of weather systems and atmospheric flow patterns.
London 2 May 2008 Extreme (European) Windstorms and Expected Changes in a Warmer Climate Lennart Bengtsson Professor ESSC, University of Reading Max Planck.
Meteorology 485 Long Range Forecasting Friday, January 16, 2004.
00/XXXX 1 Data Processing in PRISM Introduction. COCO (CDMS Overloaded for CF Objects) What is it. Why is COCO written in Python. Implementation Data Operations.
1 Adventures in Web Services for Large Geophysical Datasets Joe Sirott PMEL/NOAA.
Exploring Multi-Model Ensemble Performance in Extratropical Cyclones over Eastern North America and the Western Atlantic Ocean Nathan Korfe and Brian A.
Tropical cyclone activity in the Minerva T1279 seasonal forecasts. Preliminary analysis Julia Manganello 1, Kevin Hodges 2 1 COLA, USA 2 NERC Centre for.
Lan Xia (Yunnan University) cooperate with Prof. Hans von Storch and Dr. Frauke Feser A study of Quasi-millennial Extratropical Cyclone Activity using.
TIGGE Archive Access at NCAR Steven Worley Doug Schuster Dave Stepaniak Hannah Wilcox.
Data Discovery and Access to The International Surface Pressure Databank (ISPD) 1 Thomas Cram Gilbert P. Compo* Doug Schuster Chesley McColl* Steven Worley.
Judith Curry James Belanger Mark Jelinek Violeta Toma Peter Webster 1
Analysis of Typhoon Tropical Cyclogenesis in an Atmospheric General Circulation Model Suzana J. Camargo and Adam H. Sobel.
Climate-SDM (1) Climate analysis use case –Described by: Marcia Branstetter Use case description –Data obtained from ESG –Using a sequence steps in analysis,
Jennifer Catto Supervisors: Len Shaffrey and Kevin Hodges Extra-tropical cyclones and Storm Tracks.
Global Weather Prediction -Possible developments in the next decades- Professor Lennart Bengtsson ESSC, University of Reading MPI for Meteorology, Hamburg.
Dust modelling in HiGAM Presentation for HiGEM meeting, Reading 31 st Jan 2008 Margaret Woodage Environmental Systems Science Centre University of Reading,
© Crown copyright 2007 Forecasting weeks to months ahead Dr. Alberto Arribas Monthly-to-Decadal area, Met Office Hadley Centre Exeter, April 2014.
1 Application of MET for the Verification of the NWP Cloud and Precipitation Products using A-Train Satellite Observations Paul A. Kucera, Courtney Weeks,
Extreme Precipitation from Extra-Tropical Cyclones: A Limited Area Model Climate Change Analysis Adrian Champion, Kevin Hodges, Lennart Bengtsson NCEO.
Jim Kinter David Straus, Erik Swenson, Richard Cirone
A Guide to Tropical Cyclone Guidance
Course Evaluation Now online You should have gotten an with link.
Course Evaluation Now online You should have gotten an with link.
TIGGE Data Archive and Access System at NCAR
The cf-python software library
Communicating Uncertainty via Probabilistic Forecasts for the January 2016 Blizzard in Southern New England Frank M Nocera, Stephanie L. Dunten & Kevin.
Previous and Current Work
PyStormTracker: A Parallel Object-Oriented Cyclone Tracker in Python
Course Evaluation Now online You should have gotten an with link.
A Comparison of Extratropical Cyclones in Recent Reanalyses
Seasonal Prediction Activities at the South African Weather Service
(WCRP Seasonal Prediction Workshop) Applied Meteorology Group
Barcelona, 23 September 2015 Impact of resolution and initialisation in climate seasonal predictions F.J. Doblas Reyes.
Verification of Tropical Cyclone Forecasts
MOGREPS developments and TIGGE
Presentation transcript:

Atmospheric Data Analysis on the Grid Kevin Hodges ESSC Co-workers: Brian Hoskins, Lennart Bengtsson Lizzie Froude

TRACK Software  Objectively identify and track weather systems such as extra-tropical and tropical cyclones.  Derive statistics for distribution and properties of weather systems.  Apply to data from climate models, reanalyses and numerical weather prediction.  Verify climate models, study properties of storms and the impact of climate change, explore how well storms are predicted in numerical weather prediction both deterministic and ensemble NWP.

Example’s NH, DJF, 2002/2003,  850 SH, JJA, 1999,  850

2005

Statistics 5.0x10 -5 NH, DJF, Trd.+Int., Int. c.i. 0.5 x10 -5 SH, JJA, Trd.+Int., Int. c.i x10 -5

Ensemble Weather Prediction

Why is the Grid Useful?  Data sets are large and becoming larger as resolution increases. IPCC models ~250Km, latest models, e.g. ECHAM5, HIGEM, new ECMWF reanalysis ~60Km, ECMWF deterministic forecasts ~30Km.  Data can extend over multi-year periods, e.g. IPCC  Climate models and NWP forecasts are also run in ensemble mode, ECHAM5 IPCC 3 member ensemble, ECMWF EPS 50 members (10-14 day) twice daily.  High temporal resolution required for tracking, 6hr or better.  Example: processing all the winters in a single 30 year period can take several days on a single machine depending on the machine and data resolution.  Faster analysis if data is processed in parallel. Data is organised in individual files, e.g. for each year, each season or each ensemble member.  For confidence intervals and significance tests on the statistics requires Monte-Carlo methods – resampling.

TRACK on the Grid  CONDOR to manage jobs on the campus grid.  Vanilla universe – no linking to condor libraries – no need for checkpointing.  Each job is a script which condor submits to each machine. Script copies data from ESSC (scp) and runs the code using the options supplied and copies the results back when finished.  Turnaround limited by the number of available machines and the time each job takes to run on a single machine, example, of 30 winters takes ~1-2hrs on 30 machines, ECMWF EPS (50 members, 14 days) ~30mins.  Main problems – individual machine resources, disk, memory, the higher resolutions require sufficient space in /tmp and >1GB memory – limits available machines. Machine reboots – need to resubmit jobs.

Statistics  Due to the nature of the tracks and the way the statistics are computed need to use resampling methods to determine confidence intervals (Bootstrap) and significance tests for differences (Permutation). Bootstrap – single sample, sample with replacement. Permutation – two samples, sample pooled data without replacement, new pairs of samples.  Samples ~2000, impossible on a single machine.  Using CONDOR, 2000 samples can be done in ~1day for a pool of ~100 machines. Depends on the sampling grid size.  Compute sampling distributions and p-values.

Significance Example Track Density Mean Intensity

Remote Data Access  Data often stored remotely.  Combine CONDOR with Opendap, data on an Opendap/DODS server can be accessed via URL directly by the application without having to download data to disk.  For data in netcdf format re-link application with curl enabled netcdf libraries. Only need to read data required, on the fly subsetting.  URL replaces filename, e.g. dods/dods- ncep2/pressure/hgt/hgt.2002.nc?time,level[0:1:0],lat,lon,hgt[100:1:105][0:1:0][ 0:1:72][0:1:143]  Can aggregate data into a single data set, useful for reading across files.

.