1. ESIP Summer Meeting – 2017 July27
NOAA Big Data Project and
Preliminary GOES-16 Data
Dr. Jeff de La Beaujardière
Data Management Architect
National Oceanic and Atmospheric Administration
ESIP Summer Meeting – 2017 July27

2. Acknowledgements Many thanks to:
BDP Core Team: Ed Kearns, Andy Bailey, Shane Glass, Jeff de La Beaujardière, Tony LaVoi, Jay Morris, Derek Parks
NOAA: Brian Eiler*, Zach Goldstein, Dave Michaud, Glenn Tallia, Derek Hanson, Kate Abbott, Amy Gaskins*, Alan Steremberg*, Maia Hansen*, Steve Ansari, Steve Del Greco*, Brian Nelson, Carlos Rivero*, Ken Casey, Rich Baldwin, Ed Clark, Brian Cosgrove, Steve Volz, Mark Paese, Donna McNamara, Chris Sisko, Nathan Wilson, Mark Brady*, Renata Lana
NC State University / CICS-NC: Otis Brown, Scott Wilkins, Jon Brannock, Lou Vazquez, Scott Stevens, Paula Hennon*, Andrew Buddenberg, Angel Li
NOAA’s Big Data Collaborators and their partners (not an all-inclusive list)
Amazon: Jed Sundwall, Ariel Gold*, Jeff Layton, Joe Flasher
Microsoft: Sam Khoury, Sid Krishna, Shannon Murphy
Google: Will Curran, Matt Hancher, Eli Bixby, Tino Tereshko, Amy Unruh, Tanya Shastri, Ossama Alami, Valliappa “Lak” Lakshmanan^, Mike Hamberg
Open Commons Consortium: Walt Wells, Maria Patterson, Zac Flamig
Unidata: Mohan Ramamurthy, Jeff Weber
IBM: James Stevenson, Stefani Jones, Mary Glackin, Peter Neilley, John Aviles
The Climate Corporation: Adam Pasch

3. NOAA has "Big Data" (Volume, Variety, Velocity, ...)
Backup slides for AFCEA Big Data Panel
Satellites
Weather radars
Multibeam bathymetry
Buoy networks
Tide gauges
Human observers
Animal telemetry
Ships
Aircraft
Numerical models
NOAA data are unique, valuable, irreplaceable, and collected at public expense

4. Archive Projections for NOAA data
160 PB
2010
2020
2030
Courtesy Steve Del Greco & Ken Casey, NOAA/ NCEI

5. Most users want answers, not big data
Data to Decisions:
Distill huge & complex data to ~1 bit: take umbrella? permit building? mitigate flooding? build seawall?
Support non-expert data users

6. Notional Cloud Deployment Scenario
Backup slides for AFCEA Big Data Panel
Notional Cloud Deployment Scenario
Commercial Cloud
Information
Products
Access services
Public
users
Discovery services
Utility services
Jeff's opinion: We should leverage Cloud as much as possible
Simplify agency approval process for Cloud services
Working copy of NOAA datasets in the Cloud
Steer public-facing data usage to Cloud
Encourage compute-in-place rather than downloads
Provide basic tools (single-function, composable)
Favor serverless approach, not renting compute instances
Probably must use on-premises infrastructure for some things...and least for now
official archival copy of data
operational data processing
numerical weather forecasting
NOAA security boundary
One-way
push
On-premises
Computing
Master copy of NOAA Data
Operational
customers
Operational Processing
Forecast Models
Derived from NOAA EDM Framework (2013), figure 8

7. NOAA Big Data Project Concept (R&D with 5 collaborators)
Backup slides for AFCEA Big Data Panel
NOAA Big Data Project Concept (R&D with 5 collaborators)
User
User
User
User
User
User
use data remotely
User
Cloud-hosted data analysis, visualization, integration
copy huge
datasets
Agency Service Tier
NOAA Data
Briefing to OSTP PARR meeting

8. NOAA Data at BDP Collaborators
Amazon Web Services (AWS) - NOAA BDP Page
NOAA Next-Generation Radar (NEXRAD)
NOAA Geostationary Satellite (GOES-16) (provisional data)
Google Cloud Platform - Public Data in BigQuery
NOAA Global Historical Climatology Network (GHCN)
NOAA Global Surface Weather Summary of the Day (GSOD)
NOAA Intl Comprehensive Ocean-Atmosphere Data Set (ICOADS)
IBM - NOAA Earth Systems Data Portal
NOAA Rapid Refresh weather model (RAP)
NOAA Fisheries Data (NMFS)
Open Commons Consortium (OCC) Environmental Data Commons
Microsoft Azure
Public services TBD

9. Example BDP Success Story
NEXRAD Radar Data : Present
Entire NWS NEXRAD Level 2 Archive (300 TB) was transferred from NCEI to AWS, OCC ( ), Microsoft, and Google

10. Example BDP Success Story
NEXRAD Level 2 Radar Data on AWS
Increased 2.3X
Data Usage
Decreased 50%
Archive Server Load
Ansari et al., Unlocking the potential of NEXRAD data through NOAA’s Big Data Partnership

11. Google Cloud Platform Example
1.2 PBs of climate and weather data accessed through Google BigQuery, from Jan-Apr 2017
Without “trying” - not advertised yet
Joins, joins, joins
30-100x of NOAA deliveries in that time
Images in Google Earth Engine
GOES-16 (June 2017)
National Water Model data
Weather and Climate model output
Climate data records

12. GOES-R: NOAA's next generation of geostationary weather satellites
Geostationary Operational Environmental Satellite
GOES-R series: 4-satellite program (GOES-R/S/T/U) to extend operational GOES system through 2036.
Increased spatial resolution, faster coverage, real-time mapping of lightning activity, improved monitoring of solar activity.
GOES-R renamed GOES-16 after launch (2016 Nov 19)

13. GOES-16 full-disk composite 2017-01-15 (13:30 EDT)

14. Crop of GOES-16 full-disk composite 2017-01-15 (13:30 EDT)

15. GOES-16 Advanced Baseline Imager – 16 channels (2017-01-15)

16. Notes regarding GOES-16 and BDP
NOAA's GOES-16 satellite has not been declared operational and its data are preliminary and undergoing testing.
The Big Data Project (BDP) is a demonstration effort and business experiment and is not an operational function.
We wish to learn from the BDP experiment to help inform future NOAA and NESDIS decisions on open data distribution to users.

17. Traditional Satellite Data Internet Access Strategy
One-to-One Model
Consumer
Ground System Data Distribution
Consumer
Consumer
Consumer
Consumer

18. Big Data Project Satellite Data Access Demo Activity
One-to-Many Model

19. Initial Distribution Statistics
GOES-16 BDP Demo Live as
of July 12, 2017:
Initial Distribution Statistics
Cooperative Institute for Climate and Satellites - North Carolina (CICS-NC) is helping NOAA by providing feeds of the GOES-16 data from the NOAA Ground System (as an authorized user) to the BDP CRADA Collaborators.
BDP is offering 5 validated feeds to the CRADA Collaborators
timing - as fast as they appear at NOAA distribution point
single bounce of data through CICS-NC systems, w/checksums
minimizes load on NOAA’s operational systems and networks
Observed additional latencies from CICS-NC transfer mechanism
From NOAA Ground System to BDP Collaborator platforms
Maximum additional latency: 2 to 3 min (full disk ABI, Band 2)
Typical Range of additional latency: 30 sec - 3 min

20. https://aws.amazon.com/public-datasets/goes/
AWS GOES-16

21. https://aws.amazon.com/public-datasets/goes/
AWS GOES-16

22. OCC’s Environmental Data Commons

23. OCC GOES-16 Resources

24. OCC GOES-16 Resources

25. Open Discussion
Would you find availability of GOES-16 data in the Cloud useful?
What Cloud functions or tools would be helpful for GOES-16 data?
Do you have existing Cloud-compatible or Cloud-native tools that would be relevant to GOES-16 data?
Should ESIP convene a hackathon or challenge activity (with modest funding) to help prototype/develop tools?