HDF5 Performance Enhancements with the Elimination of Unlimited Dimension Debbie Mao, Daniel Ziskin, Merritt Deeter, Sara Martinez-alonso MOPITT is an.

Slides:

Advertisements

Similar presentations

1 NASA CEOP Status & Demo CEOS WGISS-25 Sanya, China February 27, 2008 Yonsook Enloe.

Advertisements

The HDF Group ESIP Summer Meeting Easy access HDF files via Hyrax Kent Yang The HDF Group 1 July 8 – 11, 2014.

National Aeronautics and Space Administration Jet Propulsion Laboratory California Institute of Technology MLS Level 3 Products Paul Wagner 1, Yibo Jiang.

The HDF Group HDF/HDF-EOS Workshop XIV1 Easy Remote Access via OPeNDAP Kent Yang and Joe Lee The HDF Group The 14 th HDF/HDF-EOS Workshop.

The HDF Group HDF Group Support for NPP/JPSS Mike Folk, Elena Pourmal, Larry Knox, Albert Cheng The HDF Group The 15 th HDF and HDF-EOS.

AIRS and MOPITT CO ARSET Applied Remote SEnsing Training CO plumes at an altitude of roughly 2 km (700 millibars) over the Houston area and extending out.

The HDF Group Apr , 2012HDF/HDF-EOS Workshop XV1 Interoperability with netCDF-4 Kent Yang, Larry Knox, Elena Pourmal The HDF Group.

NASA Trace Gas Products for Air Quality Applications NASA Remote Sensing Training September 2014 ARSET Applied Remote SEnsing Training A project of NASA.

University of Illinois at Urbana-ChampaignHDF Mike Folk HDF-EOS Workshop IV Sept , 2000 HDF Update HDF.

University of Illinois at Urbana-ChampaignHDF 1McGrath/Yang 2/27/02 Transitioning from HDF4 to HDF5 Robert E. McGrath Kent Yang.

Support EOS: Review and Discussions Kent Yang and Joe Lee The HDF Group October 16, 2012 Oct. 16, 2012Annual HDF Briefing to ESDIS1.

The HDF Group HDF/HDF-EOS Workshop XIV1 Easy Access of NASA HDF data via OPeNDAP Kent Yang and Joe Lee The HDF Group September 28,2010.

What is HDF-EOS? Information compiled from HDF-EOS Workshop II HDF-EOS Workshop III, 1999 ESDIS Project, Code 423 NASA/Goddard Space Flight Center Greenbelt.

A Matlab tutorial in working with HDF files

1 HDF-EOS and Related Tools Status Update. 2 Overview.

Introduction to MISR Data Analysis and Tools Linda Hunt Science Systems & Applications Inc., NASA LaRC Brian E. Rheingans Jet Propulsion Laboratory, California.

1 HDF-EOS APIs, tools, etc. Larry Klein, Abe Taaheri, and Cid Praderas L-3 Communications Government Services, Inc. November 30, 2005.

1 High level view of HDF5 Data structures and library HDF Summit Boeing Seattle September 19, 2006.

HDF5 A new file format & software for high performance scientific data management.

Important ESDIS 2009 tasks review Kent Yang, Mike Folk The HDF Group April 1st, /1/20151Annual briefing to ESDIS.

DM_PPT_NP_v01 SESIP_0715_AJ HDF Product Designer Aleksandar Jelenak, H. Joe Lee, Ted Habermann Gerd Heber, John Readey, Joel Plutchak The HDF Group HDF.

The HDF Group September 28, 2010HDF/HDF-EOS Workshop XIV1 Easy Access of HDF data via NCL/IDL/MATLAB Kent Yang, Tong Qi, Ziying Li, Yi.

Page 1 HDF-EOS Tools Abe Taaheri, Raytheon IIS ESIP Meeting Chapel Hill, NC July 9, 2013.

ArcGIS and NASA EOS Data NASA GES DISC Team: Aijun Chen, Wenli Yang, Peisheng Zhao, Long Pham, Chris Lynnes, Gail Wade NASA ESDIS: Ross Bagwell, Francis.

Improving the usability of HDF-EOS2 data Kent Yang, Joe Lee, Choonghwan Lee The HDF Group March 31 st, /26/2016Annual briefing to ESDIS1.

ATMOSPHERIC SCIENCE DATA CENTER ‘Best’ Practices for Aggregating Subset Results from Archived Datasets Walter E. Baskin 1, Jennifer Perez 2 (1) Science.

1 Status of HDF-EOS, Related Software and Tools. 2 TOOLKIT / HDF-EOS Support.

Why do I want to know about HDF and HDF- EOS? Hierarchical Data Format for the Earth Observing System (HDF-EOS) is NASA's primary format for standard data.

Tools for Interoperability between HDF and NetCDF Mike Folk and MuQun Yang The HDF Group The HDF Group provides the following tools for the NASA HDF and.

The Namibia Flood Dashboard Satellite Acquisition and Data Availability through the Namibia Flood Dashboard Matt Handy NASA Goddard Space Flight Center.

1/14/200925th IIPS Conference 1 Challenges to Archive and Access NASA HDF-EOS Data in the long Term MuQun Yang (The HDF Group) Choonghwan Lee (The HDF.

The netCDF-4 data model and format Russ Rew, UCAR Unidata NetCDF Workshop 25 October 2012.

1 HDF-EOS Development Current Status and Schedule Larry Klein, Shen Zhao, Abe Taaheri and Ray Milburn L-3 Communications Government Services, Inc. September.

The HDF Group November 3-5, 2009 HDF-OPeNDAP Project Update HDF/HDF-EOS Workshop XIII1 Joe Lee and Kent Yang The HDF Group James Gallagher.

Towards Long-Term Archiving of NASA HDF-EOS and HDF Data Data Maps and the Use of Mark-Up Language Ruth Duerr, Mike Folk, Muqun Yang, Chris Lynnes, Peter.

Ensuring Long Term Access to Remotely Sensed HDF4 Data with Layout Maps Ruth Duerr, NSIDC Christopher Lynnes, GES DISC The HDF Group Oct HDF and.

HDF & HDF-EOS Workshop VIII 2004 October Aurora, CO Bruce Beaumont, Matt Smith, Helen Conover, Sara Graves Subsetting at UAH.

1 HDF-EOS Status, Related Tools and Issues. 2 Overview.

HDF OPeNDAP Project Update MuQun Yang and Hyo-Kyung Lee The HDF Group March 31, Annual briefing to ESDIS10/31/2015.

The HDF Group HDF/HDF-EOS Workshop XV1 Tools to Improve the Usability of NASA HDF Data Kent Yang and Joe Lee The HDF Group April 17, 2012.

NetCDF file generated from ASDC CERES SSF Subsetter ATMOSPHERIC SCIENCE DATA CENTER Conversion of Archived HDF Satellite Level 2 Swath Data Products to.

HDF4 OPeNDAP Project Progress Report MuQun Yang and Hyo-Kyung Lee 1 HDF Developers' Meeting11/24/2015.

HDF5 OPeNDAP Project Update and Demo MuQun Yang and Hyo-Kyung Lee (The HDF Group) James Gallagher (OPeNDAP, Inc.) 1HDF and HDF-EOS Workshop XII, Aurora,

The HDF Group Data Interoperability The HDF Group Staff Sep , 2010HDF/HDF-EOS Workshop XIV1.

The HDF Group Introduction to netCDF-4 Elena Pourmal The HDF Group 110/17/2015.

1 Status of HDF-EOS, Related Software and Tools. 2 TOOLKIT / HDF-EOS Support.

Comments from User Services C. Boquist/Code 423 The HDF Group Meeting 1 April 2009.

CESD 1 SAGES Scottish Alliance for Geoscience, Environment & Society The challenges of geo-simulation data Centre For Earth System Dynamics

Research & Development Building a science foundation for sound environmental decisions Remote Sensing Information Gateway (RSIG)

Aura HDF-EOS File Format Guidelines: Overview and Status Cheryl Craig.

11/8/2007HDF and HDF-EOS Workshop XI, Landover, MD1 Software to access HDF5 Datasets via OPeNDAP MuQun Yang, Hyo-Kyung Lee The HDF Group.

HDF-EOS Aura File Format Guidelines Cheryl Craig HDF-EOS Workshop IV Sept 20, HDF-EOS Aura File Format Guidelines Cheryl Craig September 20, 2000.

The HDF Group November 3-5, 2009HDF/HDF-EOS Workshop XIII1 The New HDF-EOS Web Site - How it can help you Kent Yang, Joe Lee The HDF Group.

Goddard Earth Sciences Data and Information Services Center, NASA/GSFC, Code 902, Greenbelt, Maryland 20771, USA INTRODUCTION  NASA Goddard Earth Sciences.

Introduction to MISR Data Analysis and Tools Brian E. Rheingans Jet Propulsion Laboratory, California Institute of Technology Exploring and Using MISR.

Update on Unidata Technologies for Data Access Russ Rew

HDF5 OPeNDAP Project Update and Demo MuQun Yang and Hyo-Kyung Lee (The HDF Group) James Gallagher (OPeNDAP, Inc.) 1HDF and HDF-EOS Workshop XII, Aurora,

DataGrid France 12 Feb – WP9 – n° 1 WP9 Earth Observation Applications.

Making Satellite Datasets Accessible for Everyone A look into my NASA Internship – summer 2015 Aaron Scott University of North Dakota.

Data Are from Mars, Tools Are from Venus

Transition from HDF4 to HDF5: Issues

Easy Access of HDF data via NCL/IDL/MATLAB

Moving from HDF4 to HDF5/netCDF-4

MERRA Data Access and Services

HDF-EOS to GeoTIFF Conversion Tool & HDF-EOS Plug-in for HDFView

Efficiently serving HDF5 via OPeNDAP

NetCDF and Scientific Data Standard

HDF-EOS to GeoTIFF Conversion Tool & HDF-EOS Plug-in for HDFView

HDF-EOS Aura File Format Guidelines

HDF-EOS Workshop XXI / The 2018 ESIP Summer Meeting

Presentation transcript:

HDF5 Performance Enhancements with the Elimination of Unlimited Dimension Debbie Mao, Daniel Ziskin, Merritt Deeter, Sara Martinez-alonso MOPITT is an instrument flying on NASA's Earth Observing System Terra spacecraft, which was launched in 1999, and has been operational since March 3rd, 2000. MOPITT measures tropospheric carbon monoxide (CO) on the global scale, ~3 days data will almost cover the whole earth.

MOPITT Product Versions Operational Period Data Format MOP02 Size V3 * 2002 - 2009 HDF-EOS2 100 MB V4 * 2009 - 2012 140 MB V5 * 2011 - 2016 240 MB V6 2013 - present HDF-EOS5 250 MB V7 2016 - present 500 MB * no longer available Level 1 swath radiance Level 2 swath retrieved CO Level 3 grid avg retrieved CO, daily & monthly

MOPITT V5 HDF-EOS2 Simple

MOPITT V6 HDF-EOS5 structure met

MOPITT V7 HDF-EOS5 Dimen List More fields

Access to MOPITT Products NASA Data Archives via Reverb ASDC Data Pool ASDC MOPITT Subsetter Tools / Software for MOPITT Products hdfview IDV panoply MOPITT web viewer IDL NCL, …...

MOPITT Data Dimensions Level 1: track(Unlim), stare, pixel,... 1 track = 29 stares 1 stare = 4 pixels Level 2: time(Unlim), pressure,…... Level 3: xDim, yDim, pressure,…...

MOPITT hdf-eos2 to hdf-eos5 Migration Experience Unlimited dimension File open / close Idl: netCDF ~ hdfeos5 Augmentation tool Up left corner → low left corner NCL IDV Chunk size ~ idl reading time Strict data type

Before: AprioriCOMixingRatioProfile Dim: nTime,nPrs,nTwo List: 4 RetrievedCOMixingRatioProfile List: 3

After Elimination of Unlimited Dimension Use Dan's IDL to read and write all info but dimension list Run Augmentation tool to add dimension list After Elimination of Unlimited Dimension

HDF Problem Our QA Scientist noticed something very peculiar: our V7 products was much faster to analyze than V6. V6 % Time elapsed: 1541.6740 seconds. V7 % Time elapsed: 1120.7156 seconds. V7 file size is almost double of V6, but 27.3% quicker to read WHY ???

Experimental Design Compare V7 access time with and without unlimited dimension Experiment #1: use 5 datasets read multiple times Experiment #2: use 5 datasets for multiple days to measure access time. Action: Open file Read data Close file Data Fields: co_mx[nTime][nPrs][nTwo] co_tot[nTime][nTwo] lat, long, time [nTime]

MOPITT HDF5 Reading Time Comparison – Experiment #1

MOPITT HDF5 Reading Time Comparison – Experiment #2

Summary MOPITT has over 17 years data There are more CF attributes in Version 7 products Transition from HDF-EOS2 to HDF-EOS5 is a big job After eliminating Unlimited dimension: Experiment #1: small difference, about 2% Experiment #2: large difference at the first, about 40% the difference becomes smaller over time Conclusion: eliminating the unlimited dimension has an effect on access times in some applications.