1. UPCOMING DATA WORK FOR NOAA
Presented by Dr. Ed Kearns, Big Data Project Director
NOAA Big Data Project Progress
Apr 2015 Rollout
Oct 2015 CDAC briefing
Some Challenges and Lessons Learned
“Portal versus Platforms”
Discussion
OPEN COMMONS CONSORTIUM

2. NOAA BDP Update Usage Pattern from NEXRAD Weather Radar Data on AWS
TB accessed
AWS
NOAA
start BDP

3. NEXRAD on AWS: By the Numbers
Since Oct 2015, as compared to past data access patterns at NOAA/NCEI:
+130% in NEXRAD weather radar usage by volume
80% of NEXRAD archive orders are now fulfilled by AWS
Single access point for both archived and realtime data
64% of the NEXRAD data stayed on the AWS platform
Faster: job that took 3+ years now take only a few days
Cheaper: loads on NOAA archive access systems are down over 50%
New Applications/Science: Bird Migration, Insect studies
…at no net cost to the US taxpayer
S. Ansari et al, 2016 (submitted)

4. NOAA BDP Next Steps NOAA Big Data Project now collaborating on
GOES/GOES-R data access
Fisheries catch data
Numerical weather prediction model output
Climate data
Other products across NOAA
Need to…
Bring other projects to maturity
Assess project metrics
Document lessons learned
Develop the plan for the post-CRADA future

5. NOAA BDP Lessons Learned
Almost all of NOAA’s data are available, now – but for them to move, their value must be understood.
Need a “problem owner” – 3rd party?
To engage, the business case must be understood.
Hard to develop for true “Big Data” applications?
Weather and climate data are easier to build business cases about than ocean data.
Maturity of markets? Research versus operations?

6. NOAA BDP Challenges How can new users/markets be identified?
NOAA best support Data Quality throughout?
How can NOAA’s expertise be effectively leveraged?
Will platforms truly enable wider use of NOAA data?

7. Acknowledgements Many thanks to:
NOAA: Brian Eiler, Zach Goldstein, Dave Michaud, Glen Talia, Amy Gaskins*, Alan Steremberg*, Maia Hansen*, Steve Ansari, Steve Del Greco*,  Jeff de la Beaujardiere, Brian Nelson, Tony LaVoi, Jay Morris, Carlos Rivero*, Ken Casey, Shane Glass, Zoe Black
NC State University / CICS-NC: Otis Brown, Scott Wilkins, Jonathon Brannock, Lou Vazquez, Scott Stevens, Paula Hennon, Andrew Buddenberg
NOAA’s Big Data Collaborators and their partners (not an all inclusive list)
Amazon: Jed Sundwall, Arial Gold Jeff Layton
Microsoft: Sam Khoury, Sid Krishna
Google: Will Curran, Matt Hancher, Eli Bixby, Tino Tereshko, Amy Unruh, Tanya Shastri, Ossama Alami, Valliappa “Lak” Lakshmanan
Open Commons Consortium: Walt Wells, Maria Patterson,  Zac Flammig
Unidata: Mohan Ramamurthy, Jeff Weber
IBM: James Stevenson, Stefani Jones, Mary Glackin, Peter Neilley
The Climate Corporation: Adam Pasch