1. Guided Tour of Pythonian Museum
H. Joe Lee
The HDF Group
If you feel weird about the title. Don’t worry!
This talk is not about petrified snakes.
My talk is mostly about Earth data and Python.
This tour shows how Giant Python swallows Big Earthdata in HDF.
This work was supported by NASA/GSFC under Raytheon Co. contract number NNG15HZ39C

2. Welcome to Pythonian Museum!
One of my favorite places on Earth is Smithonian Museum because it has a large collection of amazing things around the world.
The best part is, it is free.
For last 10 years, we’ve collected many NASA HDF data product samples from different NASA data centers for various testing purposes.
At first, we called these collection as Zoo because of data product’s diversity.
Like biological creatures, NASA HDF data products have evolved over time.
Some old data products are no longer available from NASA data center and they are petrified in our FTP site.
So I think it’s legitimate to call it as Museum.
With NASA HDF data products from many different data centers, we’ve provided MTALAB/IDL/NCL examples.
Python examples are recently added for last few years.
Based on our website log, Python’s popularity surged significantly.

3. Your tour guide is the maintainer of hdfeos.org website.
loves diverse NASA HDF products.
has created 170+ Python examples so far.
When you hear the word Python and your brain came up with computer language, you’re a nerd.
Has anyone visited Smithonian museum?
Python zoo tour.
“Research as art”
Museum curator
Zoo keeper

4. http://hdfeos.org/zoo Let’s Go! Where is the museum?
It’s a confession time.
Public use graphics

5. The Main Entrance List of NASA Data Centers List of NASA HDF Products
Click on any product!

6. Pythonian tour has 4 packages
pyhdf – for HDF4
h5py – for HDF5
netCDF4 – for HDF4/HDF5
gdal – for HDF-EOS2/HDF-EOS5

7. Why 4 packages?
Ideally, the best product is one that can be accessed by all packages.

8. How about other packages?

9. What to focus on during tour
Does product have lat/lon dataset?
If not, are they stored in different product or metadata or document?
Bit packing / scale & offset handling
Visualization tips on world map with different projections
Memory / performance issue

10. GES DISC AIRS Grid HDF-EOS2 – 1D Lat/Lon variables pyhdf / netCDF4
Equidistant Cylindrical plot
# Read geolocation dataset.
lat = hdf.select(‘Latitude’)

11. GES DISC AIRS Swath HDF-EOS2 – 2D Lat/Lon variables pyhdf / netCDF4
Polar Stereographic plot
# Read geolocation dataset. lat = hdf.select('Latitude') latitude = lat[:,:]
# Polar stereo
m = Basemap(projection='npstere', resolution='l' boundinglat=65, lon_0 = 180)
Let’s start from A. Polar Stereographic projection is ideal for swath.

12. GES DISC TRMM Swath v7 HDF4 – pyhdf / netCDF4
Lat/Lon in one 3D dataset - subsetting
Longitude shifting for pcolormesh()
# There is a wrap-around effect to deal with.
longitude[longitude < -165] += 360

13. GES DISC TRMM Grid Lat/Lon are calculated from documentation.#
latitude = np.arange( , , )
longitude = np.arange( , , 0.25)

14. GES DISC OMI Grid HDF-EOS5 - h5py (Python3)
Lat/lon are calculated from StructMetadata
# There is no geolocation data, so construct it ourselves. longitude = np.arange(0., ) * latitude = np.arange(0., 720.0) *

15. GES DISC MLS Swath HDF-EOS5 / h5py
netCDF-4 will not work due to object reference
Vertical profile line plot –plot()
Toolkit Internal Time (TAI) handling
# Handle TAI. timebase = datetime.datetime(1993, 1, 1, 0, 0, 0) + datetime.timedelta(seconds=time[399])

16. GES DISC HIRDLS ZA HDF-EOS5 - h5py / netCDF4
Zonal Average plot - contourf()
Log scale in Y-axis
# Apply log scale along the y-axis to get a better image.
lev = np.log10(lev)

17. GES DISC GOSAT/ACOS HDF5 – h5py 1D lat/lon swath
Trajectory - scatter()
Multiple plots
Orthographic projection map

18. LAADS MOD06 Swath HDF-EOS2 – pyhdf / netCDF4
Lat/lon from MOD03 product
Scale / offset / valid range
S. Pole Stereographic projection
# Read geolocation dataset from MOD03 product.
hdf = SD(FILE_NAME, SDC.READ)
hdf_geo = SD(GEO_FILE_NAME, SDC.READ)
lat = hdf_geo.select('Latitude')

19. LP DAAC MOD09GA Grid HDF-EOS2 – pyhdf / gdal
Lat/lon in Sinusoidal Projection
mpl_toolkits.basemap.pyproj
# In basemap, the sinusoidal projection is global, so we won't use it.
# Instead we'll convert the grid back to lat/lons.
sinu = pyproj.Proj("+proj=sinu +R= wktext")
wgs84 = pyproj.Proj("+init=EPSG:4326")
lon, lat= pyproj.transform(sinu, wgs84, xv, yv)

20. MEaSURES VIP GRID HDF-EOS2 – pyhdf / netCDF4 / gdal
StructMetadata parsing
Huge dataset – sub sampling for visualization
# Scale down the data by a factor of 6 so that low-memory machines can handle it.
data = data[::6, ::6]

21. ASDC CALIPSO HDF4 – pyhdf / netCDF4 Bitmask / Complex subsetting
Discrete colorbar
# You can visualize other blocks by changing subset parameters. data2d = data[3500:3999, 0:164] # 20.2km to 30.1km
data2d = data[3500:3999, 165:1164] # 8.2km to 20.2km
data2d = data[3500:4000, 1165:] # -0.5km to 8.2km
This is the beast like Giant Squid at Smithonian museum.

22. More Examples? NASA Developers Portal – PyDAP + PyCMR
We started porting our examples into NASA Developer portal.
They are mostly related to OPeNDAP.

23. This work was supported by NASA/GSFC under Raytheon Co
This work was supported by NASA/GSFC under Raytheon Co. contract number NNG15HZ39C