UPCOMING DATA WORK FOR NOAA Presented by Dr. Ed Kearns, Big Data Project Director NOAA Big Data Project Progress Apr 2015 Rollout Oct 2015 CDAC briefing Some Challenges and Lessons Learned “Portal versus Platforms” Discussion OPEN COMMONS CONSORTIUM

NOAA BDP Update Usage Pattern from NEXRAD Weather Radar Data on AWS TB accessed AWS NOAA start BDP

NEXRAD on AWS: By the Numbers Since Oct 2015, as compared to past data access patterns at NOAA/NCEI: +130% in NEXRAD weather radar usage by volume 80% of NEXRAD archive orders are now fulfilled by AWS Single access point for both archived and realtime data 64% of the NEXRAD data stayed on the AWS platform Faster: job that took 3+ years now take only a few days Cheaper: loads on NOAA archive access systems are down over 50% New Applications/Science: Bird Migration, Insect studies …at no net cost to the US taxpayer S. Ansari et al, 2016 (submitted)

NOAA BDP Next Steps NOAA Big Data Project now collaborating on GOES/GOES-R data access Fisheries catch data Numerical weather prediction model output Climate data Other products across NOAA Need to… Bring other projects to maturity Assess project metrics Document lessons learned Develop the plan for the post-CRADA future

NOAA BDP Lessons Learned Almost all of NOAA’s data are available, now – but for them to move, their value must be understood. Need a “problem owner” – 3rd party? To engage, the business case must be understood. Hard to develop for true “Big Data” applications? Weather and climate data are easier to build business cases about than ocean data. Maturity of markets? Research versus operations?

NOAA BDP Challenges How can new users/markets be identified? NOAA best support Data Quality throughout? How can NOAA’s expertise be effectively leveraged? Will platforms truly enable wider use of NOAA data?

Acknowledgements Many thanks to: NOAA: Brian Eiler, Zach Goldstein, Dave Michaud, Glen Talia, Amy Gaskins*, Alan Steremberg*, Maia Hansen*, Steve Ansari, Steve Del Greco*,  Jeff de la Beaujardiere, Brian Nelson, Tony LaVoi, Jay Morris, Carlos Rivero*, Ken Casey, Shane Glass, Zoe Black NC State University / CICS-NC: Otis Brown, Scott Wilkins, Jonathon Brannock, Lou Vazquez, Scott Stevens, Paula Hennon, Andrew Buddenberg NOAA’s Big Data Collaborators and their partners (not an all inclusive list) Amazon: Jed Sundwall, Arial Gold (now @DOT), Jeff Layton Microsoft: Sam Khoury, Sid Krishna Google: Will Curran, Matt Hancher, Eli Bixby, Tino Tereshko, Amy Unruh, Tanya Shastri, Ossama Alami, Valliappa “Lak” Lakshmanan (formerly @TCC) Open Commons Consortium: Walt Wells, Maria Patterson,  Zac Flammig Unidata: Mohan Ramamurthy, Jeff Weber IBM: James Stevenson, Stefani Jones, Mary Glackin, Peter Neilley The Climate Corporation: Adam Pasch