Integrated Grid workflow for mesoscale weather modeling and visualization Zhizhin, M., A. Polyakov, D. Medvedev, A. Poyda, S. Berezin Space Research Institute.

Slides:



Advertisements
Similar presentations
1 NASA CEOP Status & Demo CEOS WGISS-25 Sanya, China February 27, 2008 Yonsook Enloe.
Advertisements

Introduction to the BinX Library eDIKT project team Ted Wen Robert Carroll
Metadata at ICPSR Sanda Ionescu, ICPSR.
DataTools Models Data, models and tools: Dealing with any complex hydraulic engineering problem invariable use is made of: data, models and tools.
NetCDF Weather Data from Unidata By David Maidment, Tim Whiteaker and Cedric David Center for Research in Water Resources University of Texas at Austin.
IWay Service Manager 6.1 Product Update Scott Hathaway iWay Software Copyright 2010, Information Builders. Slide 1.
McIDAS-V McIDAS-V The 5 th Generation of McIDAS by Tom Whittaker Space Science and Engineering Center University of Wisconsin-Madison USA with contributions.
Air and Water Quality Information Scientist Contributed Measurements Government Sponsored Community Input and Ratings Educational Localized in 26.
2014 ESIP Summer Meeting July 8–11, 2014 | Frisco, Colorado Advancing Scientific Data Support in ArcGIS Nawajish Noman.
Computational Physics Kepler Dr. Guy Tel-Zur. This presentations follows “The Getting Started with Kepler” guide. A tutorial style manual for scientists.
Development of a Community Hydrologic Information System Jeffery S. Horsburgh Utah State University David G. Tarboton Utah State University.
Integrating Historical and Realtime Monitoring Data into an Internet Based Watershed Information System for the Bear River Basin Jeff Horsburgh David Stevens,
Stanford University EH&S A Service Oriented Architecture For Rich Internet Applications Sheldon M. Heitz.
The International Surface Pressure Databank (ISPD) and Twentieth Century Reanalysis at NCAR Thomas Cram - NCAR, Boulder, CO Gilbert Compo & Chesley McColl.
Christine White, Esri Growing OPeNDAP Support: Current ArcGIS Workflows and Future Directions Christine White, Esri
Активное распределенное хранилище для многомерных массивов Дмитрий Медведев ИКИ РАН.
TPAC Digital Library Talk Overview Presenter:Glenn Hyland Tasmanian Partnership for Advanced Computing & Australian Antarctic Division Outline: TPAC Overview.
Overview of the ODP Data Provider Sergey Sukhonosov National Oceanographic Data Centre, Russia Expert training on the Ocean Data Portal technology, Buenos.
M. Taimoor Khan * Java Server Pages (JSP) is a server-side programming technology that enables the creation of dynamic,
Discussion and conclusion The OGC SOS describes a global standard for storing and recalling sensor data and the associated metadata. The standard covers.
EARTH SCIENCE MARKUP LANGUAGE “Define Once Use Anywhere” INFORMATION TECHNOLOGY AND SYSTEMS CENTER UNIVERSITY OF ALABAMA IN HUNTSVILLE.
DISTRIBUTED DATA FLOW WEB-SERVICES FOR ACCESSING AND PROCESSING OF BIG DATA SETS IN EARTH SCIENCES A.A. Poyda 1, M.N. Zhizhin 1, D.P. Medvedev 2, D.Y.
ESSE Environmental Scenario Search Engine for the Data Services Grid Mikhail Zhizhin, Geophysical Center Russian Academy of Sciences Eric Kihn,
Unidata’s TDS Workshop TDS Overview – Part II October 2012.
HDF5 A new file format & software for high performance scientific data management.
DM_PPT_NP_v01 SESIP_0715_AJ HDF Product Designer Aleksandar Jelenak, H. Joe Lee, Ted Habermann Gerd Heber, John Readey, Joel Plutchak The HDF Group HDF.
Ohio State University Department of Computer Science and Engineering Automatic Data Virtualization - Supporting XML based abstractions on HDF5 Datasets.
RDFS Rapid Deployment Forecast System Visit at: Registration required.
A Metadata Based Approach For Supporting Subsetting Queries Over Parallel HDF5 Datasets Vignesh Santhanagopalan Graduate Student Department Of CSE.
Unidata TDS Workshop TDS Overview – Part I XX-XX October 2014.
Pipelines and Scientific Workflows with Ptolemy II Deana Pennington University of New Mexico LTER Network Office Shawn Bowers UCSD San Diego Supercomputer.
Introduction to Apache OODT Yang Li Mar 9, What is OODT Object Oriented Data Technology Science data management Archiving Systems that span scientific.
Mid-Course Review: NetCDF in the Current Proposal Period Russ Rew
CUAHSI Hydrologic Information Systems. HIS Project Team Yao Liang John Helly Project co-PI Collaborator.
EARTH SCIENCE MARKUP LANGUAGE Why do you need it? How can it help you? INFORMATION TECHNOLOGY AND SYSTEMS CENTER UNIVERSITY OF ALABAMA IN HUNTSVILLE.
Accomplishments and Remaining Challenges: THREDDS Data Server and Common Data Model Ethan Davis Unidata Policy Committee Meeting May 2011.
The netCDF-4 data model and format Russ Rew, UCAR Unidata NetCDF Workshop 25 October 2012.
Integrated Model Data Management S.Hankin ESMF July ‘04 Integrated data management in the ESMF (ESME) Steve Hankin (NOAA/PMEL & IOOS/DMAC) ESMF Team meeting.
1 Dapper and Argo Joe Sirott PMEL/NOAA. 2 What is Dapper? Web server that provides distributed access to in-situ data via OPeNDAP protocol Clients include.
The european ITM Task Force data structure F. Imbeaux.
Research Design for Collaborative Computational Approaches and Scientific Workflows Deana Pennington January 8, 2007.
1 Web-Based Data Archive, Monitoring, and Mining Tool Jeffrey A. Melby, PhD USACE ERDC Coastal and Hydraulics Lab
CUAHSI Hydrologic Information Systems David R. Maidment Center for Research in Water Resources University of Texas at Austin and Ilya Zaslavsky, David.
A/WWW Enterprises 28 Sept 1995 AstroBrowse: Survey of Current Technology A. Warnock A/WWW Enterprises
, Key Components of a Successful Earth Science Subsetter Architecture ASDC Introduction The Atmospheric Science Data Center (ASDC) at NASA Langley Research.
_______________________________________________________________CMAQ Libraries and Utilities ___________________________________________________Community.
Unidata’s Common Data Model and the THREDDS Data Server John Caron Unidata/UCAR, Boulder CO Jan 6, 2006 ESIP Winter 2006.
IODE Ocean Data Portal - ODP  The objective of the IODE Ocean Data Portal (ODP) is to facilitate and promote the exchange and dissemination of marine.
May 2003National Coastal Data Development Center Brief Introduction Two components Data Exchange Infrastructure (DEI) Spatial Data Model (SDM) Together,
CE 394K.2 Surface Water Hydrology Lecture 1 – Introduction to the course Readings for today –Applied Hydrology, Chapter 1 –“Integrated Observatories to.
NQuery: A Network-enabled Data-based Query Tool for Multi-disciplinary Earth-science Datasets John R. Osborne.
Information Technology: GrADS INTEGRATED USER INTERFACE Maps, Charts, Animations Expressions, Functions of Original Variables General slices of { 4D Grids.
SEEK Science Environment for Ecological Knowledge l EcoGrid l Ecological, biodiversity and environmental data l Computational access l Standardized, open.
GIS for Atmospheric Sciences and Hydrology By David R. Maidment University of Texas at Austin National Center for Atmospheric Research, 6 July 2005.
AHM04: Sep 2004 Nottingham CCLRC e-Science Centre eMinerals: Environment from the Molecular Level Managing simulation data Lisa Blanshard e- Science Data.
Data Stewardship at the NOAA Data Centers Sub Topic - Value Added Products ESIP Federation Meeting, Washington, DC January 6-8, 2009.
TIGGE Archive Access at NCAR Steven Worley Doug Schuster Dave Stepaniak Hannah Wilcox.
The HDF Group January 8, ESIP Winter Meeting Data Container Study: HDF5 in a POSIX File System or HDF5 C 3 : Compression, Chunking,
1 CLASS – Simple NOAA Archive Access Portal SNAAP Eric Kihn and Rob Prentice NGDC CLASS Developers Meeting July 14th, 2008 Simple NOAA Archive Access Portal.
ESIP Air Quality Jan Air Quality Cluster Air Quality Cluster Technology Track Earth Science Information Partners Partners NASA NOAA EPA (?) USGS.
Copyright 2007, Information Builders. Slide 1 iWay Web Services and WebFOCUS Consumption Michael Florkowski Information Builders.
OGC Web Services with complex data Stephen Pascoe How OGC Web Services relate to GML Application Schema.
XML and Distributed Applications By Quddus Chong Presentation for CS551 – Fall 2001.
The CUAHSI Hydrologic Information System Spatial Data Publication Platform David Tarboton, Jeff Horsburgh, David Maidment, Dan Ames, Jon Goodall, Richard.
Andrew White, Brian Freitag, Udaysankar Nair, and Arastoo Pour Biazar
Downloading Weather Observations
Step 1 Create Database Info activity in Adeptia Server specifying the driver, URL and user credentials information for the database in which stored.
Lecture 8 Database Implementation
CEE 6440 GIS in Water Resources Fall 2004 Term Paper Presentation
Computational Physics Kepler
Presentation transcript:

Integrated Grid workflow for mesoscale weather modeling and visualization Zhizhin, M., A. Polyakov, D. Medvedev, A. Poyda, S. Berezin Space Research Institute of the Russian Academy of Sciences

Abstract For the model input and output we use a scalable parallel storage and data mining system called ActiveStorage. It can store different types of weather data, provided they are in the same Command Data Model (UNIDATA CDM): NCEP reanalysis, NCDC stations weather data, MM5 model output. The MM5 is a mesoscale weather forecast model. For the input boundary conditions the model takes basic parameters such as elevation, air pressure and temperature, etc. It can ingest reanalysis and direct observation data. As the output the model provides high-resolution regional weather grids. To make the MM5 input data and the modeling results accessible on the Grid to the Earth Science community, we have developed a set of grid services (resources and activities) inside the OGSA-DAI (both ver. 2 and 3) grid service container. To visualize the weather data we have developed a special plugin for the NASA World Wind which can read the data directly from the OGSA-DAI resources and plot it over the 3D globe in different ways, such as contour lines, filled areas and vector fields.

Active Storage, Modeling, Data Mining and Visualization Services

ActiveStorage ActiveStorage is a generic storage for arrays of primitive data types. Its data model is based on the Unidata’s Common Data Model, used in netCDF, HDF5 and OpenDAP. Basically, ActiveStorage is a SQL Server database with CLR stored procedures and a client library. The stored procedures and the client library provide an abstraction layer for data access. Large arrays are split into chunks and can be spread across several parallel database servers for better performance.

SQL Server 2005/2008 DB ActiveStorage components Metadata tables Data and directory tables Stored procedures Client library

Common Data Model This is the Common Data Model (CDM) used in the recent versions of OpenDAP, netCDF and HDF5. Its purpose is the representation of multidimensional scientific data.

How it works SQL Server DB Client library 2. Issue commands to the database server 3. Select the requested data from several chunks 3. Return the data parts to the client library 4. Assemble the data parts into one multi-dimensional array 1. Pass multi-dimensional data request to the client library Application

Parallel query processing SQL Server DB 1 Client library Application SQL Server DB 2

Parallel query performance 1 database server 4 parallel database servers

NCEP/NCAR Weather Reanalysis Continually updating gridded data set Incorporates observations and global climate model output 74 weather parameters 5000 netCDF files, 30 – 500 MB each Time coverage: 1948 – hourly values Grids: Regular grid, 2.5 x 2.5 degrees T62 Gaussian grid, 192 x 94 points.

NCDC Integrated Surface Database 1901 – 2008 time coverage. 30 million sensors. 1.7 billion observations. Fixed ground stationsShipsMobile stationsBuoys FM V N N N ADDGA KA1120N datetimelatlon Mandatory data sectionAdditional data sectionSection marker Group marker Parameter group Control data section ASCII files packed with gzip. 50 GB packed; 400 GB unpacked. When you’ve downloaded and unpacked the data...

import ru.wdcb.mdb.NcConnector import com.microsoft.sqlserver.jdbc.SQLServerDriver s = 'jdbc:sqlserver://localhost:1433;databaseName=NCEP_01;user=g uest;password=guest'; connector = NcConnector(); ncid = connector.nc_open(s,0); varid = connector.nc_inq_varid(ncid,'air'); origin = [ ]; size = [ ]; stride = [ ]; A = connector.nc_get_vars_short(ncid,varid,origin,size,stride); plot(A, 'DisplayName', 'A', 'YDataSource', 'A'); figure origin = [ ]; size = [ ]; stride = [ ]; B = connector.nc_get_vars_shortm(ncid,varid,origin,size,stride); B = reshape(B,[73 144]); imagesc (B); figure(gcf); MATLAB script using ActiveStorage library

Environmental Data Service: OGSA-DAI plugin

Activities for data export XML output stream – We have plugin for NASA World Wind to visualize XML- formatted data – Can easily be transformed using XSLT to web page or another XML document, e.g. MS Excel – Can be used as input for ESSE fuzzy logic search engine NetCDF binary data file – Standard for scientific data storage in files – There are several visualization programs for NetCDF – Compatible with Unidata Common Data Model standard

Data flow management by OGSA-DAI OGSA-DAI query from single data source OGSA-DAI query from distributed data sources

Parallel mesoscale weather model MM5 Same Source Parallel MM5 Source code for the parallel MPI and the single process MM5 model are the same Automated parallel code generation from MM5 sources by ANL: –FLIC compiler –RSL library for model domain segmentation and message exchange We have ported MM5 code to the MS Windows Server 2008 HPC platform

MM5 model as a grid client

Visualizing data from ActiveStorage with NASA WorldWind A NASA WorldWind plugin, developed at the Moscow State University allows to retrieve data from ActiveStorage via an OGSA-DAI service. Several kinds of visualization are available: - isolines - color map - vector field OGSA-DAI services can be used by other applications to retrieve data from ActiveStorage

NASA World Wind as a grid client Using OGSA-DAI services and a special API plugin, the NASA World Wind can visualize both the MM5 input and output datasets