Guided Tour of Pythonian Museum H. Joe Lee (hyoklee@hdfgroup.org) The HDF Group If you feel weird about the title. Don’t worry! This talk is not about petrified snakes. My talk is mostly about Earth data and Python. This tour shows how Giant Python swallows Big Earthdata in HDF. This work was supported by NASA/GSFC under Raytheon Co. contract number NNG15HZ39C

Welcome to Pythonian Museum! One of my favorite places on Earth is Smithonian Museum because it has a large collection of amazing things around the world. The best part is, it is free. For last 10 years, we’ve collected many NASA HDF data product samples from different NASA data centers for various testing purposes. At first, we called these collection as Zoo because of data product’s diversity. Like biological creatures, NASA HDF data products have evolved over time. Some old data products are no longer available from NASA data center and they are petrified in our FTP site. So I think it’s legitimate to call it as Museum. With NASA HDF data products from many different data centers, we’ve provided MTALAB/IDL/NCL examples. Python examples are recently added for last few years. Based on our website log, Python’s popularity surged significantly. https://commons.wikimedia.org/wiki/File:Smithsonian_Building_NR.jpg

Your tour guide is the maintainer of hdfeos.org website. loves diverse NASA HDF products. has created 170+ Python examples so far. When you hear the word Python and your brain came up with computer language, you’re a nerd. Has anyone visited Smithonian museum? Python zoo tour. “Research as art” Museum curator Zoo keeper

http://hdfeos.org/zoo Let’s Go! Where is the museum? It’s a confession time. Public use graphics https://www.autodraw.com/artists

The Main Entrance List of NASA Data Centers List of NASA HDF Products Click on any product!

Pythonian tour has 4 packages pyhdf – for HDF4 h5py – for HDF5 netCDF4 – for HDF4/HDF5 gdal – for HDF-EOS2/HDF-EOS5

Why 4 packages? Ideally, the best product is one that can be accessed by all packages.

How about other packages?

What to focus on during tour Does product have lat/lon dataset? If not, are they stored in different product or metadata or document? Bit packing / scale & offset handling Visualization tips on world map with different projections Memory / performance issue

GES DISC AIRS Grid HDF-EOS2 – 1D Lat/Lon variables pyhdf / netCDF4 Equidistant Cylindrical plot # Read geolocation dataset. lat = hdf.select(‘Latitude’)

GES DISC AIRS Swath HDF-EOS2 – 2D Lat/Lon variables pyhdf / netCDF4 Polar Stereographic plot # Read geolocation dataset. lat = hdf.select('Latitude') latitude = lat[:,:] # Polar stereo m = Basemap(projection='npstere', resolution='l' boundinglat=65, lon_0 = 180) Let’s start from A. Polar Stereographic projection is ideal for swath.

GES DISC TRMM Swath v7 HDF4 – pyhdf / netCDF4 Lat/Lon in one 3D dataset - subsetting Longitude shifting for pcolormesh() # There is a wrap-around effect to deal with. longitude[longitude < -165] += 360

GES DISC TRMM Grid Lat/Lon are calculated from documentation. # http://disc.sci.gsfc.nasa.gov/additional/faq/precipitation_faq.shtml latitude = np.arange(-49.875, 49.875, 0.249375) longitude = np.arange(-179.875, 179.876, 0.25)

GES DISC OMI Grid HDF-EOS5 - h5py (Python3) Lat/lon are calculated from StructMetadata # There is no geolocation data, so construct it ourselves. longitude = np.arange(0., 1440.0) * 0.25 - 180 + 0.125 latitude = np.arange(0., 720.0) * 0.25 - 90 + 0.125

GES DISC MLS Swath HDF-EOS5 / h5py netCDF-4 will not work due to object reference Vertical profile line plot –plot() Toolkit Internal Time (TAI) handling # Handle TAI. timebase = datetime.datetime(1993, 1, 1, 0, 0, 0) + datetime.timedelta(seconds=time[399])

GES DISC HIRDLS ZA HDF-EOS5 - h5py / netCDF4 Zonal Average plot - contourf() Log scale in Y-axis # Apply log scale along the y-axis to get a better image. lev = np.log10(lev)

GES DISC GOSAT/ACOS HDF5 – h5py 1D lat/lon swath Trajectory - scatter() Multiple plots Orthographic projection map

LAADS MOD06 Swath HDF-EOS2 – pyhdf / netCDF4 Lat/lon from MOD03 product Scale / offset / valid range S. Pole Stereographic projection # Read geolocation dataset from MOD03 product. hdf = SD(FILE_NAME, SDC.READ) hdf_geo = SD(GEO_FILE_NAME, SDC.READ) lat = hdf_geo.select('Latitude')

LP DAAC MOD09GA Grid HDF-EOS2 – pyhdf / gdal Lat/lon in Sinusoidal Projection mpl_toolkits.basemap.pyproj # In basemap, the sinusoidal projection is global, so we won't use it. # Instead we'll convert the grid back to lat/lons. sinu = pyproj.Proj("+proj=sinu +R=6371007.181 +nadgrids=@null +wktext") wgs84 = pyproj.Proj("+init=EPSG:4326") lon, lat= pyproj.transform(sinu, wgs84, xv, yv)

MEaSURES VIP GRID HDF-EOS2 – pyhdf / netCDF4 / gdal StructMetadata parsing Huge dataset – sub sampling for visualization # Scale down the data by a factor of 6 so that low-memory machines can handle it. data = data[::6, ::6]

ASDC CALIPSO HDF4 – pyhdf / netCDF4 Bitmask / Complex subsetting Discrete colorbar # You can visualize other blocks by changing subset parameters. data2d = data[3500:3999, 0:164] # 20.2km to 30.1km data2d = data[3500:3999, 165:1164] # 8.2km to 20.2km data2d = data[3500:4000, 1165:] # -0.5km to 8.2km This is the beast like Giant Squid at Smithonian museum.

More Examples? NASA Developers Portal – PyDAP + PyCMR We started porting our examples into NASA Developer portal. They are mostly related to OPeNDAP. https://developer.earthdata.nasa.gov/opendap/cms-ch4-flx-na

This work was supported by NASA/GSFC under Raytheon Co This work was supported by NASA/GSFC under Raytheon Co. contract number NNG15HZ39C