The HDF Group Update on NASA Wish List Mike Folk, Elena Pourmal, Kent Yang, Joe Lee The HDF Group Briefing to ESDIS April 4, /4/2013HDF Briefing to NASA 1

Overview NASA Wish List is a set of tasks requested during last briefing. Total 48 items in the list. Focus: 36 High Priority Items Two Divisions G eneral Maintenance, QA, and Support (GMQS) ESDIS Support, Operation Users Support, and Studies (EOS) 4/4/2013HDF Briefing to NASA 2

GMQS Division Accountable: Elena Pourmal Categories: HDF4 and HDF5 Libraries General Tools (HDFView, h5dump, etc.) General Support (Releases) 4/4/2013HDF Briefing to NASA 3

EOS Division Accountable: Kent Yang Categories: ESDIS Software (including OPeNDAP) Promotion Studies 4/4/2013HDF Briefing to NASA 4

GMQS > Libraries HDF4 Library Item 12: Modify a compressed HDF4 data array. HDF5 Library Item 34: Add “parallel” gzip. 4/4/2013HDF Briefing to NASA5

GMQS > Libraries > Item 12 ItemModify a compressed HDF4 data array ActionReviewed and confirmed that HDF4 library cannot overwrite a part of the dataset (all or nothing). Created an example code that HDFView can use to resolve user’s request. Original comments HDF Java user reported this problem for his quality screen application. StatusIn Progress Comments Update HDFView (version 2.10). Do more investigation in HDF4. Fix bugs. Provide recommendations to HDF4 library users on how modify compressed datasets. 4/4/2013HDF Briefing to NASA 6

GMQS > Libraries > Item 34 ItemAdd parallel gzip ActionReviewed parallel gzip code. It cannot be used with HDF5 as is. Talked to other stakeholders interested in the feature. Original comments GES DISC (?) looked at parallel gzip to improve performance. StatusIn Progress Comments Work on the design document for the feature. Come up with the estimate to do the work. Talk to all stakeholders to raise funds. 4/4/2013HDF Briefing to NASA 7

GMQS >Tools h5dump Item 9: Clean ASCII output from h5dump HDFView Item15: HDFView and netCDF-4 files with VL attributes Item 4,18, 35: HDF-EOS plug-in and HDFView Item 17, 22-27: New features in HDFView 4/4/2013HDF Briefing to NASA8

GMQS > Tools > Item 9 ItemClean ASCII output for h5dump ActionRFC is sent to James Johnson and Chris Lynnes. The fix is made and ready for release. Original comments Goal is to easily provide clean ASCII output, without attribute info., etc. StatusDone Comments 4/4/2013HDF Briefing to NASA 9

GMQS > Tools > Item 15 ItemNetCDF-4 files with string type attributes give variable length of size of one. ActionFixed and changes committed to SVN repository. Original comments There doesn’t seem to be a way to create variable length strings. StatusDone Comments It’ll be in HDFView 2.10, which is available upon request. 4/4/2013HDF Briefing to NASA 10

GMQS > Tools > Item 4 ItemObtain permission to redistribute HDF- EOS plugin with HDFView ActionSent a letter to Dan and confirmed with Lou. Original comments Lou will look into getting permission. Lou may need to ask Dan for a letter of approval. StatusDone Comments We can release HDF-EOS plug-in with HDFView in our website. 4/4/2013HDF Briefing to NASA 11

GMQS > Tools > Item 18 ItemSynchronize EOS plugin with HDFView ActionWe plan to come up with the RFC document that discusses the 4 solutions and we will solicit recommendations. Original comments StatusPlanned Comments See the next slide for 4 solutions. 4/4/2013HDF Briefing to NASA 12

GMQS > Tools > Item 18 4/4/2013HDF Briefing to NASA 13 1.Add the plug-in to the HDFView installer but have an option to install the plug-in or no. 2.Have a separate HDFView installer that packed with the plug-in and will install the plug-in automatically. 3.Have a Add plug-in in HDFView so that users can add the plug-in anytime. 4.Ask Abe to distribute HDFView with the plug-in (like #2) and we will provide a link to his release of HDFView.

GMQS > Tools > Item 35 ItemHDF-EOS Plug-in Functionality ActionDiscuss the identified features with James Johnson and other ESDIS stakeholders (RFC) Original comments James asked if HDFView can have some functionality like Panoply. StatusGathered requirements and identified list of features Comments The common feature between HDF-EOS Plug-in and Panoply is the ability to display data on map. 4/4/2013HDF Briefing to NASA 14

GMQS > Tools > Item 25 ItemNew Features in HDFView - OPeNDAP ActionWe’re working on RFC. Original comments You’ll probably get more requests for this in the future as far as MODIS goes. StatusIn Progress Comments Joe had discussion with Peter on OPeNDAP. 4/4/2013HDF Briefing to NASA 15

GMQS > Tools > Item 17, 24, 27, 23 ItemNew Features in HDFView: Dimension Scales, DOI, Graphics, GeoTIFF ActionDimension Scales and DOI: April Graphics (fill values, etc.): May GeoTIFF: June Original comments StatusScheduled RFC Comments GeoTIFF work depends on Item 8, which will be discussed in EOS division later. 4/4/2013HDF Briefing to NASA 16

GMQS > Tools > Item 22, 26 ItemNew Features in HDFView: CF Conventions, New web service queries ActionThe survey report was completed and published. Original comments Peter will do this as part of writing up results of survey. DAACS are starting to provide services for WCS, OpenSearch, etc. but there are very few clients that support them. StatusNo plan Comments html/hdfview/hdfview_surevey_report_2013.pdf 4/4/2013HDF Briefing to NASA 17

GMQS: General Support Issues Item 1: Supported OS and compilers Item 3: Cygwin support is needed by HDF-EOS Item 20: Tools of value to users 4/4/2013HDF Briefing to NASA18

GMQS > General Support > Item 1 ItemNeed to know what OS and compilers are supported. ActionOS and compilers are planned for each maintenance release. An HDF developer is assigned at the time of release planning for support issues on a particular platform. Original comments StatusIn Progress Comments See our wiki page in the next slide. 4/4/2013HDF Briefing to NASA 19

GMQS > General Support > Item 1 4/4/2013HDF Briefing to NASA20

GMQS > General Support > Item 3 ItemCygwin support is needed by HDF-EOS. ActionCygwin is tested daily for HDF4 and HDF5. Original comments StatusDone Comments 4/4/2013HDF Briefing to NASA 21

GMQS > General Support > Item 20 ItemTools of value to users ActionWe updated and reorganized tools pages* to help users to find the right tool. We’ve been working with DAACs to raise awareness of tools through telecons and direct contacts. Original comments Earthdata.nasa.gov describes tools available to DAACs. What we do here should be integrated with that and inform them. StatusIn Progress Comments * olsbycat.html 4/4/2013HDF Briefing to NASA 22

10 Minute Break 4/4/2013HDF Briefing to NASA 23

EOS > ESDIS Software OPeNDAP HDF5 Handler Item 19:Support NetCDF-4 model in HDF5 handler (Done) Item 32:HDF5 handler maintenance OPeNDAP HDF4 Handler Item 28:HDF4 handler maintenance OPeNDAP fileout_netcdf Handler Item 5: Fix fileout_netcdf Handler. (Done) Item 42:Dump as NetCDF-4 for fileout_netcdf 4/4/2013HDF Briefing to NASA 24

Briefing to NASA 25 EOS > ESDIS Software > Item 19 ItemMake the HDF5 handler support the netCDF-4 data model ActionKent added the support to HDF5 OPeNDAP handler in December Original comments GES DISC had netCDF-4 files that would like to be served by Hyrax StatusDone CommentsA RPM was created and was sent to GES DISC in Jan. Everything works fine.

Briefing to NASA 26 EOS > ESDIS Software > Item 32 ItemHDF5 OPeNDAP handler maintenance ActionImprove code quality and tests for HDF5 OPeNDAP handler (default option only) Original comments StatusWill get started on June 1 st, 2013 Comments

Briefing to NASA 27 EOS > ESDIS Software > Item 28 ItemHDF4 OPeNDAP handler maintenance ActionImprove the code quality and other maintenance Original comments Need to improve the code readability, error handling etc. StatusIn Progress CommentsWe replaced the new[ ] with vector according to the suggestions of OPeNDAP team. We added support in handling MEaSUREs VIP product and the version 6 of MODIS geo-location files.

EOS > ESDIS Software > Item 5 ItemOPeNDAP’s file netCDF out module is not working in Hyrax 1.8 ActionInvestigate the problem: Then fix the bug or make recommendation. Original commenats Chris Lynnes and James Johnson: This is the first priority. StatusDone CommentsGES DISC, LP DAAC and MODAPS all installed this version and they reported the file netCDF dumper works fine. 4/4/2013HDF Briefing to NASA 28

Briefing to NASA 29 EOS > ESDIS Software > Item 42 ItemnetCDF-4 (as opposed to netCDF-3) as a file dumper output possibility for OPeNDAP. ActionMake this item a high priority based on the conversation with GES DISC (Previous priority: medium high) Original comments StatusWill work on this in July CommentsUsing netCDF-4 classic(with the compression) as the file dumper output becomes more important.

EOS > ESDIS Software H4CF Conversion Toolkit Item 21:H4CF Conversion Toolkit HDF-EOS5 to NetCDF-4 Tool Item 29:Retire HDF-EOS5 to NetCDF-4 conversion tool (Done) 4/4/2013HDF Briefing to NASA 30

Briefing to NASA 31 EOS > ESDIS Software > Item 21 ItemImprove the H4CF conversion toolkit ActionUpgrade the H4CF conversion toolkit to make it consistent with the way how the HDF4 OPeNDAP handler handled the NASA HDF-EOS2 and HDF4 data. Original comments Will update naming conventions, fix critical bugs, improve features for popular products such as MODIS. Will implement a test suite and update the configuration process. StatusIn Progress. ( Release plan: June 15) CommentsGood progress upgrading the toolkit to follow CF conventions as the HDF4 OPeNDAP handler. Implemented comprehensive test suite Updated configuration and conversion to netCDF3 and netCDF4-classic. MISR data & coding style still need to be improved.

Briefing to NASA 32 EOS > ESDIS Software > Item 29 ItemRetire "HDF-EOS 5 to netCDF-4" conversion tool ActionMade an announcement and waited for replies Original comments HDF-EOS5 augmentation tool is a superset of this tool. StatusDone CommentsAnnounced on Dec. 5 th, No response by Feb. 15 th, 2013, so we announced tool’s official retirement on Feb. 15 th, 2013.

EOS > Operation Users Support Guide Data Producers with HDF Best Practices Item 11: Document issues for NASA HDF4/HDF- EOS2 products Make HDF-EOS Easy to Build Item 30:CMake for HDF-EOS Item 31:Autoconfiguration for the EOS Toolkit 4/4/2013HDF Briefing to NASA 33

Briefing to NASA 34 EOS > Op. Users Support > Item 11 ItemPublish the document describing issues for NASA HDF4 (EOS2) products ActionWrite up the issues in a white paper. Make available on website. Push to all relevant stakeholders. Original comments Needs to be a clear and concise document. "Guide to good use of HDF." "Best practices," something like that. Perhaps create a "best practices" section on the web site. StatusIn Progress CommentsWe are working on the outline of the white paper now.

Briefing to NASA 35 EOS > Op. Users Support > Item 30 ItemCMake for HDF-EOS Action Original comments Clarify with Abe what it means. StatusNot started yet, assigned the person to work on this CommentsCommunicated with Abe several times.

Briefing to NASA 36 EOS > Op. Users Support > Item 31 ItemAutoconfiguration for the ECS toolkit. ActionMeet with Abe and look at all of his tools in the toolkit (including libraries) and create a roadmap for implementing autoconfiguration. Original comments StatusNot started yet, assigned the person to work on this CommentsCommunicated with Abe several times.

EOS > Promotion HDF-EOS website Item 16: Hard to find zoo link (Done) Item 2: Promote example pages on HDF-EOS web site Item13: Publicize HDF-EOS website 4/4/2013HDF Briefing to NASA 37

Briefing to NASA 38 EOS > Promotion > Item 16 ItemHard to find zoo link at hdfeos.org ActionImprove the ability to find example pages Original comments StatusDone CommentsPut the zoo link at the top section in examples page with a description. Put the zoo link in the front page’s DID YOU KNOW tip section.

EOS > Promotion > Item 2 4/4/2013HDF Briefing to NASA 39 ItemPromote example pages on HDF-EOS web site ActionWith Lou's help, contact Earth data pages about putting pointers to these pages. Contact DAAC user services working groups. Work with ESDIS to submit an article to Earth Observer. Original comments StatusIn Progress Comments1)A link* to the comprehensive page at hdfeos.org has been added to earthdata.nasa.gov. 2) We worked with ESDIS to submit an article about the comprehensive example to Earth Observer. * and-examples

EOS > Promotion > Item 13 4/4/2013HDF Briefing to NASA 40 ItemPublicizing HDF-EOS website ActionWrite brochure about website. Get help distributing brochure at conferences such as AGU. Contact ESDIS staff about this. Original comments StatusIn Progress CommentsWe contacted with ESDIS staff and we are planning to create a NASA brochure about the HDF-EOS comprehensive example page.

EOS > Studies Joe Item 6: Open Source Apache Incubator pilot project Kent Item 8: GeoTIFF investigation Item 33: A specification for CF metadata for HDF5 and HDF5 is needed Mike Item10: DOI support in HDF4 and HDF5 4/4/2013HDF Briefing to NASA 41

EOS > Studies > Item 6 4/4/2013HDF Briefing to NASA 42 ItemOpen Source Apache Incubator Pilot Project ActionLegal issues like NOSA license were reviewed. A new project proposal* for HDF to GeoTIFF converter is being written. Original comments StatusIn Progress Comments We are recruiting team members and sponsors! Champion: Paul Ramirez (NASA JPL) Mentor: Chris Mattman (NASA JPL) *

EOS > Studies > Item 8 4/4/2013HDF Briefing to NASA 43 ItemGeoTIFF investigations ActionWrite white paper, jointly with Abe, evaluating GeoTIFF support needs and opportunities. Original comments Need to determine which products are important for GeoTIFF support. StatusIn Progress CommentsCollected GeoTiFF information from several data centers and Abe. Held a phone discussion with Abe about the paper. Started on outline of white paper.

EOS > Studies > Item 33 4/4/2013HDF Briefing to NASA 44 ItemA specification for CF metadata for HDF4 and HDF5 ActionWrite a paper on the current status of work on CF conventions, then decide what is best. Original comments We need to better understand who is doing what on this topic. Ted Habermann is working on it. There will be a discussion on it at the ESDSWG meeting. Would like the spec to be made available. Would be good for ESDIS to publish as a tech note/standard. StatusIn Progress CommentsWe participated in the telecon of ESDS HDF5 and provided material and suggestions about CF and interoperability to the WG chairs.

EOS > Studies > Item 10 4/4/2013HDF Briefing to NASA 45 ItemWrite paper on DOI support in HDF5 ActionWhite paper completed, ready for review. Original comments StatusDone CommentsHighlights: In EOS, DOIs assigned at program/mission/collection/product level DOI API and BNF grammar proposed Implementation estimated at 100 hours Next steps?

Thank You! Questions? Comments? 4/4/2013HDF Briefing to NASA 46

Acknowledgements 4/4/2013HDF Briefing to NASA This work was supported by Subcontract number under Raytheon Contract number NNG10HP02C, funded by the National Aeronautics and Space Administration (NASA). Any opinions, findings, conclusions, or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of Raytheon or the National Aeronautics and Space Administration. 47