1 HDF-EOS APIs, tools, etc. Larry Klein, Abe Taaheri, and Cid Praderas L-3 Communications Government Services, Inc. November 30, 2005.

Slides:



Advertisements
Similar presentations
1 HDF-EOS Status, Related Tools and Issues. 2 TOOLKIT / HDF-EOS Support.
Advertisements

JTX Overview Overview of Job Tracking for ArcGIS (JTX)
MODIS Data at NSIDC MODIS Collection 5/Long Term Data Record Workshop Molly McAllister & Terry Haran January
1 An Overview of HDF-EOS (Part I) Doug Ilg Raytheon STX (301)
Operating-System Structures
LP DAAC User Services Provide support to users on ASTER data related problems; Resources on ASTER: EROS Data Center Land Processes DAAC URL:
Development of a Prototype Web GIS Server for HDF-EOS Data based on OGC Web Map Specifications Liping Di, Zhangshi Yin, Wenli Yang, Meixia Deng, and R.
1 Generalized Conversion of HDF-EOS Products to GIS Compatible Formats Larry Klein, Ray Milburn, Cid Praderas, and Abe Taaheri Emergent Information Technologies,
1 HDF-EOS Vector Data Eugene Chi, Liping Di, R.Suresh NASA/RITSS
University of Illinois at Urbana-ChampaignHDF Mike Folk HDF-EOS Workshop IV Sept , 2000 HDF Update HDF.
The HDF-EOS5 Tutorial Ray Milburn L3 Communciations, EER Systems Inc McCormick Drive, 170 Largo, MD USA
R A D I A T I O N B U D G E T C L O U D S A E R O S O L S T R O P O S P H E R I C C H E M I S T R Y L a n g l e y A t m o s p h e r i c S c i e n c e s.
What is HDF-EOS? Information compiled from HDF-EOS Workshop II HDF-EOS Workshop III, 1999 ESDIS Project, Code 423 NASA/Goddard Space Flight Center Greenbelt.
EARTH SCIENCE MARKUP LANGUAGE “Define Once Use Anywhere” INFORMATION TECHNOLOGY AND SYSTEMS CENTER UNIVERSITY OF ALABAMA IN HUNTSVILLE.
05 December, 2002HDF & HDF-EOS Workhop VI1 ESDIS Status Richard Ullman ESDIS Project
1ESDIS HDF-EOS Workshop IV Landover, Maryland, September 20, 2000 The Landsat 7 Processing System ( LPS ) Level Zero-R Science Products Michael R. Reid.
1 HDF-EOS and Related Tools Status Update. 2 Overview.
Introduction to MISR Data Analysis and Tools Linda Hunt Science Systems & Applications Inc., NASA LaRC Brian E. Rheingans Jet Propulsion Laboratory, California.
1 HDF-EOS Status, Related Tools and Issues. 2 Overview.
1 High level view of HDF5 Data structures and library HDF Summit Boeing Seattle September 19, 2006.
Important ESDIS 2009 tasks review Kent Yang, Mike Folk The HDF Group April 1st, /1/20151Annual briefing to ESDIS.
Page 1 HDF-EOS Tools Abe Taaheri, Raytheon IIS ESIP Meeting Chapel Hill, NC July 9, 2013.
U.S. Department of the Interior U.S. Geological Survey Diving into the Data Pool with DAAC2Disk Kelly Lemig ERT, Inc., contractor to the U.S. Geological.
December 5, 2002HDF-EOS Workshop VI1 The Goddard Earth Sciences Distributed Active Archive Center The Goddard Earth Sciences Distributed Active Archive.
1 Status of HDF-EOS, Related Software and Tools. 2 TOOLKIT / HDF-EOS Support.
December 1, 2005HDF & HDF-EOS Workshop IX P eter Cao, NCSA December 1, 2005 Sponsored by NLADR, NFS PACI Project in Support of NCSA-SDSC Collaboration.
1 HDF-EOS Status and Development Larry Klein, Abe Taaheri, and Cid Praderas L-3 Communications Government Services, Inc. November 30, 2005.
Why do I want to know about HDF and HDF- EOS? Hierarchical Data Format for the Earth Observing System (HDF-EOS) is NASA's primary format for standard data.
UAH The University of Alabama in Huntsville SUBSETTING Matt Smith Information Technology and Systems Center (ITSC) University of Alabama in Huntsville.
Introduction to MISR Data Analysis and Tools
Page 1 Status of HDF-EOS, Related Software, and Tools Abe Taaheri, Raytheon IIS HDF & HDF-EOS Workshp XIII Riverdale, MD November 4, 2009.
“OnEarth” WMS Server WMS Global Mosaic Lucian Plesea Jet Propulsion Laboratory California Institute of Technology.
U.S. Department of the Interior U.S. Geological Survey Access to MODIS Land Data Products Through the Land Processes DAAC John Dwyer and Carolyn Gacke,
1 HDF-EOS Development Current Status and Schedule Larry Klein, Shen Zhao, Abe Taaheri and Ray Milburn L-3 Communications Government Services, Inc. September.
Page 1 Current Status of HDF-EOS, Maintenance, Developments, and Tools Abe Taaheri, Raytheon IIS HDF & HDF-EOS Workshop XI November 2007.
HDF & HDF-EOS Workshop VIII 2004 October Aurora, CO Bruce Beaumont, Matt Smith, Helen Conover, Sara Graves Subsetting at UAH.
1 HDF-EOS Status, Related Tools and Issues. 2 Overview.
WSL Workshop The Long and Winding Road to MODIS Data.
HDF and HDF-EOS Workshop VIII, October 26-28, /12 Peter Cao, National Center for Supercomputing Applications This work is supported in part by a.
September 4, 2003MODIS Ocean Data Products Workshop, Oregon State University1 Goddard Earth Sciences (GES) Distributed Active Archive Center (DAAC) MODIS.
Page 1 TOOLKIT / HDF-EOS Status and Development Abe Taaheri, Raytheon IIS Aura DSWG meeting October 2007.
GES DISC DAAC February 28, 2002HDF-EOS Workshop V1 The Goddard DAAC The Goddard DAAC Presented by:
NetCDF file generated from ASDC CERES SSF Subsetter ATMOSPHERIC SCIENCE DATA CENTER Conversion of Archived HDF Satellite Level 2 Swath Data Products to.
HDF-EOS Workshop IV September 19, 2000 MODIS Reprojection Tool presented by John M. Weiss, Ph.D. Department of Mathematics and Computer Science South Dakota.
12/2/2015Fall 2002 AGU Meeting1 Generalized EOS Data Converter: Making Data Products Accessible to GIS Tools Larry Klein, Ray Milburn, Cid Praderas and.
Facilitating Access to EOS Data at the NSIDC DAAC Siri Jodha Singh Khalsa ECS Science Coordinator for the National Snow and Ice Data Center, Distributed.
1 HDF Vendors/Software Developers Workshop HDF And HDF-EOS Tools R.Suresh NASA/GSFC/HSTX Ph: FAX:
s Donna J. Scott, Marilyn Kaminski, Jason Wolfe, Terry Haran NSIDC's MODIS Snow and Sea Ice Products NSIDC provides a suite.
Page 1 HDF-EOS Status, Related Tools and Issues Abe Taaheri, Raytheon IIS 2009 THG Briefing to ESDIS.
HDF and HDF-EOS Workshop VIII, October 26-28, /12 Peter Cao, National Center for Supercomputing Applications Ray Milnurn, Dave Buto, L-3 Communications.
1 Status of HDF-EOS, Related Software and Tools. 2 TOOLKIT / HDF-EOS Support.
Robert Wolfe NASA Goddard Space Flight Center Code 614.5, Greenbelt, MD Robert Wolfe NASA Goddard Space Flight Center Code 614.5,
Comments from User Services C. Boquist/Code 423 The HDF Group Meeting 1 April 2009.
1 The Polar HDF-EOS Data Imaging and Subsetting (PHDIS) Tool Siri Jodha Singh Khalsa Emergent Information Technologies, Inc. National Snow and Ice Data.
Aura HDF-EOS File Format Guidelines: Overview and Status Cheryl Craig.
Summary of HDF-EOS5 Files, Data Model and File Format Abe Taaheri, Raytheon IIS HDF & HDF-EOS Workshop XI November 2007.
U.S. Department of the Interior U.S. Geological Survey Discover MODIS Land at your Desk January 17, 2007 Calli B. Jenkerson* MODIS Science Data Specialist.
September 23-25, 2003HDF-EOS Workshop VII ECS-HSA the HEW Subsetting Appliance HDF-EOS Workshop VII Silver Spring, MD – September 23-25, 2003 Dr. Sara.
MODIS Cryosphere Science Data Product Metrics Prepared by the ESDIS SOO Metrics Team for the Cryosphere Science Data Review January 11-12, 2006.
Introduction to MISR Data Analysis and Tools Brian E. Rheingans Jet Propulsion Laboratory, California Institute of Technology Exploring and Using MISR.
MODIS Data at NSIDC MODIS Science Team Meeting - Nov. 2, 2006.
ORNL DAAC SPATIAL DATA ACCESS TOOL Open Geospatial Consortium (OGC) Services Bruce E. Wilson Suresh K. Santhana Vannan Yaxing Wei Tammy W. Beaty National.
MODIS Enhanced-V 1.MODIS IR covering Enhanced-V event 0430z May 25, 2004 Located MODIS L1B Granule at LAADs (MODIS archive and distribution web site) that.
Introduction to MISR Data Analysis and Tools Brian E. Rheingans Jet Propulsion Laboratory, California Institute of Technology Exploring and Using MISR.
AIRS Meeting GSFC, February 1, 2002 ECS Data Pool Gregory Leptoukh.
Emergent Information Technologies, Inc.
NASA HDF and HDF-EOS Status Use in EOSDIS
HDF-EOS to GeoTIFF Conversion Tool & HDF-EOS Plug-in for HDFView
HDF and HDF-EOS Tools George Schwenke Raytheon STX
HDF-EOS to GeoTIFF Conversion Tool & HDF-EOS Plug-in for HDFView
Presentation transcript:

1 HDF-EOS APIs, tools, etc. Larry Klein, Abe Taaheri, and Cid Praderas L-3 Communications Government Services, Inc. November 30, 2005

2 What is HDF-EOS ? HDF (Hierarchical Data Format) is the format chosen by NASA as the standard for the EOS program HDF-EOS is an extension to HDF which standardizes storage of and access to common Earth Sciences data structures –By providing a single interface to data structures common to earth science –By providing a container for EOS inventory, archive and product specific metadata –By establishing a consistent relationship between geolocation and science data.

3 HDF-EOS Grid Structure Usage - Data which is organized by regular geographic spacing, specified by projection parameters. Structure –Any number of 2-D to 8-D data arrays per structure, one per data type (e.g. temperature) –Geolocation information contained in projection formula, coupled by structural metadata. –Any number of Grid structures per file allowed.

4 HDF-EOS Swath Structure Data which is organized by time, or other track parameter. Spacing can be irregular. Structure –Geolocation information stored explicitly in Geolocation Field (2-D array) –Data stored in 2-D or 3-D arrays –Time stored in 1-D or 2-D array, TAI is standard. –Geolocation/science data connected by structural metadata

5 Swath Data Example

6 HDF-EOS Point Structure Data is specified temporally and/or spatially, but with no particular organization Structure –Tables used to store science data at a particular Lat/Long/Height –Up to eight levels of data allowed. Structural metadata specifies relationship between levels.

7 Zonal Average Structure Generalized array structure with no geolocation linkage Data can be organized by time or track parameter Data spacing can be irregular Structure –Data stored in multidimensional arrays –Time stored in 1-D or 2-D array

8 HDF-EOS 2 HDF4 - based, storage format for EOS standard products. Used operationally by MODIS, MISR, ASTER, Landsat, AIRS and other EOS instruments Support for Grid/Point/Swath structures HDF4 and HDF-EOS 2 will be supported by NCSA/NASA for the fore-seeable future.

9 HDF-EOS 5 Based on HDF5, a complete rewrite of HDF4 with a different user interface. –First released in –Format used by EOS Aura instruments Designed to ‘resemble’ HDF-EOS 2 to the maximum extent possible. –Supports same data structures –Added prefix ‘HE5_’ to HDF-EOS 2 functions. –Doesn’t preclude HDF5 functionality.

10 Top Level of HDF-EOS 5 File STRUC. METADATA HDFEOS SWATH GRID POINT Root -- “/” ADDITIONAL Global (File) Attributes Global (File) Attributes HDFEOS INFORMATION HDFEOS INFORMATION ARRAY ECS METADATA

11 Basic HDF-EOS Functions Access - Open, create, close objects/files Definition - Set key parameters of data sets Input/Output - Read/Write data Inquiry - Return information about data Subset - Read data from specified region

12 File/Object Access GRID: HE5_GDopen/he5_gdopen HE5_GDcreate/he5_gdcreate HE5_GDattach/he5_gdattach HE5_GDdetach/he5_gddetach HE5_GDclose/he5_gdclose

13 File/Object Access POINT: HE5_PTopen/he5_ptopen HE5_PTcreate/he5_ptcreate HE5_PTattach/he5_ptattach HE5_PTdetach/he5_ptdetach HE5_PTclose/he5_ptclose

14 File/Object Access SWATH: HE5_SWopen/he5_swopen HE5_SWcreate/he5_swcreate HE5_SWattach/he5_swattach HE5_SWdetach/he5_swdetach HE5_SWclose/he5_swclose

15 File/Object Access ZA: HE5_ZAopen/he5_zaopen HE5_ZAcreate/he5_zacreate HE5_ZAattach/he5_zaattach HE5_ZAdetach/he5_zadetach HE5_ZAclose/he5_zaclose

16 Definition GRID: HE5_GDdefdim/he5_gddefdim HE5_GDdefproj/he5_gddefproj HE5_GDdeforigin/he5_gddeforigin HE5_GDdefpixreg/he5_gd_defpreg

17 Definition POINT: HE5_PTdeflevel/he5_ptdeflevel HE5_PTdeflinkage/he5_ptdeflinkage

18 Definition SWATH: HE5_SWdefdim/he5_swdefdim HE5_SWdefdimmap/he5_swdefmap HE5_SWdefidxmap/he5_swdefimap HE5_SWdefgeofield/he5_swdefgfld HE5_SWdefdatafield/he5_swdefdfld

19 Definition ZA: HE5_ZAdefdim/he5_zadefdim HE5_ZAdefine/he5_zadefine HE5_ZAdefcomp/he5_zadefcomp HE5_ZAdefchunk/he5_zadefchunk

20 Input/Output GRID: HE5_GDwritefield/he5_gdwrfld HE5_GDreadfield/he5_gdrdfld HE5_GDwriteattr/he5_gdwrattr HE5_GDreadattr/he5_gdrdattr

21 Input/Output POINT: HE5_PTwritelevel/he5_ptwritelevel HE5_PTreadlevel/he5_ptreadlevel HE5_PTwriteattr/he5_ptwrattr HE5_PTreadattr/he5_ptrdattr

22 Input/Output SWATH: HE5_SWwritefield/he5_swwrfld HE5_SWreadfield/he5_swrdfld HE5_SWwriteattr/he5_swwrattr HE5_SWreadattr/he5_swrdattr

23 Input/Output ZA: HE5_ZAwrite/he5_zawrite HE5_ZAread/he5_zaread HE5_ZAwriteattr/he5_zawrattr HE5_ZAreadattr/he5_zardattr

24 Inquiry GRID: HE5_GDinqdims/he5_gdinqdims HE5_GDdiminfo/he5_gddiminfo HE5_GDprojinfo/he5_gdprojinfo HE5_GDorigininfo/he5_gdorigininfo HE5_GDpixreginfo/he5_gdpreginfo

25 Inquiry POINT: HE5_PTlevelinfo/he5_ptlevelinfo HE5_PTinqattrs/he5_ptinqattrs HE5_PTattrinfo/he5_ptattrinfo

26 Inquiry SWATH: HE5_SWinqdatafields/he5_swinwdflds HE5_SWinqgeofields/he5_swinqgflds HE5_SWinqattrs/he5_swinqattrs HE5_SWattrinfo/he5_swattrinfo

27 Inquiry ZA: HE5_ZAinquire/he5_zainquire HE5_ZAinfo/he5_zainfo HE5_ZAinqattrs/he5_zainqattrs HE5_ZAattrinfo/he5_zaattrinfo

28 Programming Model Open the HDF-EOS5 file Create or Attach to a specified Grid/Swath/Point/ZA object Define key object features and/or perform necessary operations (with the data sets, attributes, etc) Detach from the Grid/Swath/Point/ZA object Close the HDF-EOS5 file

29 HDF-EOS Functionality Basic File I/O Swath Interface Grid Interface Point Interface Profile Interface Generalized Array Interface (Zonal Avg. ( HDF5 )) Subsetting Fill Values Compression (gzip, szip, shuffling) Chunking/Tiling Aliases Global (File), Group & Local Attributes Threadsafe Version

30 Applications HE5View (HDF5 - based browser) EOSView (HDF4 - based browser) HDFView (with HDF-EOS Plug-in), Java EOS Browser (HDF-EOS 2 and 5 access) heconvert (converts HDF4 - based Grid/Point/Swath structures to HDF-EOS 5 equivalents) –Compliments HDF4 -> HDF5 conversion tool –Not tested on all products HEG (HDF-EOS to GeoTIFF Converter/Subsetter)

31 HDFView HDFView: A Java-based browser from NCSA for HDF4 and HDF5 – based data. HDF-EOS plug-in extends HDFView to browse HDF-EOS data in both versions. –One reason: EOSView is becoming obsolete The extension is a plug-in module and does not change NCSA HDFView functionality. All copy, save, editing, image display, etc. for HDF4 and 5 objects remains. Identical look and feel when displaying HDF and HDF-EOS objects. Users can access a complete breakdown of HDF- EOS objects. e.g. grid information, projection information, dimensions, datafields, and attributes are displayed.

32 HDFView (Cont.) Release 2.2 October A separate HDF-EOS plug-in: Users install HDFView 2.2 and integrates HDF-EOS plug-in into it. - Ability to do "Save as" for HDF-EOS5: modify individual values in grid or swath data. Delete entire grids or swaths. Save the result. - Separate help window for HDF-EOS objects Release 2.3 January Point Object viewing capability - Geolocating pixels for swath and grid images  Future Plans: - Ability to modify input HDF-EOS2 granules. - Ability to cut/paste objects, modify/delete attributes. - Ability to create new grids/swaths

33 HDFView showing ASTER L1B swath

34 A utility that converts EOSDIS data from HDF-EOS format to common Geographical Information System (GIS)-compatible formats. Two versions available: –A downloadable desktop version –Access through NASA archive online storage (Data Pools) –Implemented also in prototypes for Stand-alone Data Pool, OGC (Open Geospatial Consortium) for Web Services Chaining, and DOWS (Deploy OGC Web Services) on the DAACs Details of internal file structures are transparent to users. HDF-EOS to GeoTIFF Converter (HEG)

35 HEG Functionality Subsetting ( spatial, field, band ) Performs conversion to a single-band or multi- band GeoTIFF’s HDF-EOS Swath to HDF-EOS Grid and Swath HDF-EOS Swath or Grid to GeoTIFF HDF-EOS Swath or Grid to generic Binary ( with metadata file ) Stitching ( mosaicing ) + subsetting + Reprojecting Running from Command Line (batch jobs) or with Java-based GUI Reprojection ( to UTM, PS, TM, STP, LCC, LAMAZ, GEO, SIN ) Subsampling ( Subsample stacks )

36 HEG Functionality Metadata preservation/creation. Currently supports MODIS, MISR, ASTER, AIRS and AMSR-E products on TERRA and AQUA (>110 products), AURA/HDF-EOS5 support in progress Allows file selection from users local storage Operable on Sun, SGI, Win, Linux (64-bit soon), MAC Integrated into ECS Data Pool –Reduces the transfer time of HDF-EOS data sets (if subset is requested) –Provides the end-user with the exact file required by their application User Interface –Portable, written in Java. –Not dependent on COTS (eg. IDL).

37 Availability Access to libraries and applications:

38 Mt. Etna Eruption : MISR L1B Swaths: Stitched

39 Mt. Etna : Stitched, Subsetted ASTER L1B

40 MODIS L1B: Reprojected, Co-Registered with ASTER, La Plata Tornado

41 La Plata Tornado Damage: ASTER L1B reprojected/stitched/subsetted

42 MLS L2GP-H2O Product (2004, day 243), Geographic Projection, Full Globe

43 MLS L2GP-H2O Product, PS Projection, Northern Hemisphere

Hurricane Francis, MODIS (MOD021KM, Multi-Band GeoTIFF, 3 bands: EV_1KM_RefSB, Bands 1,2,5)

45 AST_L1B, 2003 California Fires Geo. projection, Multi-Band GeoTIFF, (3 bands: Visibile Near Infrared (VNIR)) Two Swaths Stitched ….. ….. and Subsetted