Big Data and Cloud Computing Expert Panel

Slides:

Advertisements

Similar presentations

Kathy Fontaine CEOS WGISS 24, DLR Oberpfaffenhofen Munich October NASA Agency Report.

Advertisements

Brian Killough NASA NASA Future Missions Summary CGMS-38 Meeting Delhi, India November 8-12, 2010.

Space-based Architecture for Climate Mary Kicza NOAA’s Assistant Administrator for Satellite and Information Services May 19, 2011.

BEDI -Big Earth Data Initiative

USGS Core Data Stream Report Session 7: Baseline Global Observation Scenario SDCG-7 Sydney, Australia March 4 th – 6 th 2015.

1 NASA Weather Applications Program 2010 Program Overview John A. Haynes Program Manager, Weather Applied Sciences Program Earth Science Division Science.

NASA AGENCY REPORT Dr. Eric J. Lindstrom Physical Oceanography Program Scientist Earth Science Division Science Mission Directorate MARCH 12, 2007.

1 January 17, 2007 NASA IPY Interest NASA has a strong interest in IPY themes Present environmental status of the polar regions Quantify and understand.

1 Air Quality Applied Sciences Team (AQAST) Team Meeting – June 2015 John A. Haynes, MS Program Manager, Health and Air Quality Applied Sciences Program.

Next Generation Application Platform (NGAP) Andrew Mitchell WGISS-39 Tsukuba, Japan Monday, May 11,

Ensuring Credibility of NASA’s Earth Science Data Session Title: IN11D. Data Curation, Credibility, Preservation Implementation, and Data Rescue to Enable.

NASA, CGMS-40, November 2012 Coordination Group for Meteorological Satellites - CGMS Status report on the current and future satellite systems by NASA.

NASA’s Earth Science Division Flight Overview Bradley D. Doorn Program Manager Earth Science Division, Science Mission Directorate PECORA 18.

NOAA Satellite Conference April 8-12, 2013 Mary Kicza Assistant Administrator for Satellite & Information Services.

EUM/SIR/VWG/12/0375, v1, 10 July 2012 Coordination Group for Meteorological Satellites - CGMS EUM/SIR/VWG/12/0375, v1, 10 July 2012 Coordination Group.

Michael H. Freilich 21 April 2009 Earth Science Division Overview and Status.

NASA, CGMS-41, July 2013 Coordination Group for Meteorological Satellites - CGMS Status report on the current and future satellite systems by NASA Presented.

NASA Earth Science Technology Update Presented to Doppler Wind Lidar Working Group April 28, 2015 George J. Komar Associate Director/Program Manager Earth.

1 GEO Energy Expert Meeting: Perspectives on Work Plan Tasks August 28-29, 2006 Richard Eckman NASA Applied Sciences Program Science Mission Directorate.

Center for Satellite Applications and Research (STAR) Review 09 – 11 March 2010 Image: MODIS Land Group, NASA GSFC March 2000 Mission and Science Support.

MODIS Workshop An Introduction to NASA’s Earth Observing System (EOS), Terra, and the MODIS Instrument Michele Thornton

Second GPM Applications Workshop 9-10.June Partners in the GPM Constellation Second GPM Applications Workshop 9-10.June.2015.

Overview of CEOS Virtual Constellations Andrew Mitchell NASA CEOS SIT Team / WGISS NASA ESRIN – Frascati, Italy September 20, 2013 GEOSS Vision and Architecture.

Breakout Session IV: Applying Remote Sensing Observations to Impacts Assessment IPCC WG 2 Report (2008) Climate Change Impacts, Adaptation and Vulnerability.

Diane E. Wickland NPP Program Scientist NPP Science: HQ Perspective on VIIRS May 18, 2011.

Joint Workshop Applied Sciences Objectives Ed Sheffner Applied Sciences Program NASA HQ.

Educator Resources Lauren Ritter, NASA Education Pathways Intern Hurricane and Severe Storm Sentinel (HS3) Global Precipitation Measurement (GPM) Soil.

Session 2: Improved Situational Awareness Sixth Meeting of the Science Advisory Committee 28 February – 1 March 2012 Future Activities National Space Science.

Gary Jedlovec Roadmap to Success transitioning unique NASA data and research technologies to operations.

Update on NASA’s Earth Observing Data and Information System (EOSDIS) Presented to the CEOS WGISS September 29, 2015 Dawn Lowe, ESDIS Project Goddard Space.

Evolving a Legacy System Evolution of the Earth Observing Data and Information System M. Esfandiari 1, H. Ramapriyan 1, J. Behnke 1, E. Sofinowski 2 1.

ESA UNCLASSIFIED – For Official Use Data Stewardship Interest Group WGISS-40 Meeting Preservation of SW & Documents at CEOS Agencies Approaches and Lessons.

Transitioning unique NASA data and research technologies to operations Short-term Prediction Research and Transition (SPoRT) Project Future Directions.

AMS Washington Forum Federal Panel Michael H. Freilich 4 April 2013 Suomi NPP VIIRS Visible Composite NASA LDCM First-Light OLI False-Color Fort Collins.

Review of the 2012 Suomi NPP Applications Workshop: Outcomes, Results, Progress Forrest Melton November 18, 2014 Suomi NPP Applications.

Session 2 Conclusion: Future Product Development Activities Science Advisory Committee Meeting 26 – 28 August, 2014 National Space Science and Technology.

LIVE INTERACTIVE YOUR DESKTOP January 23, 2012 NES: Weather and Climate: Satellite Meteorology Presented by: Rudo Kashiri.

1 Before we begin the Tech Infusion process discussions, a few questions On the the Context of SEEDS Tech Infusion R. Husar, The questions.

NASA Earth Exchange (NEX) A collaborative supercomputing environment for global change science Earth Science Division/NASA Advanced Supercomputing (NAS)

1 Current Plans for Long Term Archiving of MODIS Data Martha Maiden Program Executive Earth Science Data Systems NASA Headquarters MODIS Meeting November.

ESRIN – Frascati, Italy September 20, 2013 NASA Report GEOSS Vision and Architecture Meeting ESRIN – Frascati, Italy September 20, 2013 Andrew Mitchell.

LP DAAC Overview – Land Processes Distributed Active Archive Center Chris Doescher LP DAAC Project Manager (605) Chris Torbert.

Report from NASA 17th GSICS Executive Panel, Biot, 2-3 June 2016 James J. Butler NASA Goddard Space Flight Center Code 618 Biospheric Sciences Laboratory.

FY16-17 Overview/Summary MODIS Science Team Meeting June

NASA, CGMS-43, May 2015 Coordination Group for Meteorological Satellites - CGMS Status report on the current and future satellite systems by NASA Presented.

NASA Earth Science Data Stewardship

NASA’s Earth Science Data Systems Update

NASA Flight Program Update

VEGA-GEOGLAM Web-based GIS for crop monitoring and decision support in agriculture Evgeniya Elkina, Russian Space Research Institute The GEO-XIII Plenary.

NASA/US Ocean Satellite Missions

Presented to the CEOS WGISS

Big Data in Earth Observation

KNOW YOUR EARTH PROJECT 2014 QUIZ COLLECTION

Improving Data Access, Discovery, and Usability

Presentation on Copernicus Dissemination

NASA Overview NASA Earth Science Division CEOS Plenary 2017

Copernicus Programme European Commission CEOS Plenary 2017

NASA Agency Report Andrew Mitchell Deputy Project Manager / Technical

NASA Report on Cal/Val Activities

NASA Report on Cal/Val Activities

Land Imagery Data Architectures

Future Data Architectures Big Data Workshop – April 2018

Andrew Mitchell São Paulo, Brazil 12th of April 2018

NASA Earthdata Login Andrew Mitchell - NASA WGISS-44 Beijing, China

FDA Objectives and Implementation Planning

WGISS Connected Data Assets April 9, 2018 Yonsook Enloe

Precipitation Virtual Constellation (P-VC)

WIS Strategy – WIS 2.0 Submitted by: Matteo Dell’Acqua(CBS) (Doc 5b)

WGISS Connected Data Assets Oct 24, 2018 Yonsook Enloe

Agency Reports – USGS Jenn Lacey LSI-VC-5 Agenda Item #2 February 2018

Presentation transcript:

Big Data and Cloud Computing Expert Panel Assessing Cloud-Native Architectures for EOSDIS Applications Dan Pilone - dan@element84.com The material is based upon work supported by the National Aeronautics and Space Administration under Contract Number NNG15HZ39C

Earth Science Instruments on ISS: Formulation Implementation Primary Ops Extended Ops Landsat 9 PACE NI-SAR SWOT TEMPO JPSS-2 (NOAA) RBI, OMPS-Limb Sentinel-6A/B GRACE-FO (2) ICESat-2 Earth Science Instruments on ISS: RapidScat, CATS, LIS, SAGE III (on ISS), TSIS-1, OCO-3, ECOSTRESS, GEDI, CLARREO-PF CYGNSS ISS SORCE, TCTE (NOAA) NISTAR, EPIC (NOAA’S DSCOVR) QuikSCAT EO-1 Landsat 7 (USGS) SMAP Terra Fleet - Earth Science Division - Missions (with color key; editable) Dec. 15, 2015 Credit: NASA/J. Mottar ------------------------------ SLIDE INFO UDATE EJD PER FREILICH DECISIONS FEB 4, 2016 Changed “Altimetry FO (Formulation in FY16) to “Sentinal-6A/B” On Landsat 9, removed comma and TIRFF Change I was going to make – “NISTAR, EPIC (NOAA’s DSCOVR) to green primary ops was already corrected Dec. 15, 2015 – Updated: TEMPO changed to Implementation phase; NISTAR, EPIC changed to Primary Ops phase; ; OCO-3 and ECOSTRESS changed to Implementation phase; Aquarius removed; slide now fully editable. JMottar May 18, 2015 – Added to SMD Multimedia Library. May 18, 2015 – Updated the outdated 2014 version: TRMM was removed; NISTAR, EPIC (NOAA's DSCOVR) were added; SMAP icon was updated; moon at top left was removed; color key moved to top left.-JMottar ID: I-EA-15-0040 (image) ID: SL-EA-15-0068 (slide) Suomi NPP (NOAA) Aqua CloudSat Landsat 8 (USGS) CALIPSO GPM Aura GRACE (2) OSTM/Jason 2 (NOAA) OCO-2

Major Components of the Earth Science Data Systems Program Earth Observing System Data and Information System (EOSDIS) Core systems for processing, ingesting and archiving data for the Earth Science Division Competitively Selected Programs Making Earth System Data Records for Use in Research Environments (MEaSUREs) 27 projects selected in 2012 Advancing Collaborative Connections for Earth System Science (ACCESS) 12 projects selected in 2013 International and Interagency Coordination and Development CEOS WGISS OSTP Climate Data Initiative NASA-ESA Bilateral GEO/USGEO Tjf 2/11/16: OSTM = Ocean Surface Topography Mission NPP = National Polar-orbiting Partnership ISS = International Space Station SMAP = Soil Moisture Active Passive SAC-D =OCO-2 = Orbiting Carbon Observatory-2 1/3/2019

12 Discipline Oriented DAACs To do this, we distribute that data over 12 discipline oriented DAACs. Cyrosphere, Land processes, snow and ice, socio-economic, etc. This sets the stage for our problem…

15+ years of Earth Science Data Deletion of MODIS Collection 4 Products Landsat-7 Data Migrated from LPDAAC to USGS Addition of AQUARIUS and ICEBRIDGE products ALOS PALSAR & UAVSAR products ICESat MEASURES 2006 MEASURES 2012 TERRA AQUA AURA Addition of SNPP CERES&OMPS Total EOSDIS Accumulated Data Archive Volume (Petabytes) A petabyte is a lot of data: 1 PB is 20 million four-drawer filing cabinets full of text 1 PB is 13.3 years of HD-TV video 1.5 PB is the size of 10 billion photos on Facebook

EOSDIS Archive Growth Estimate (Prime + Extended) Lots of assumptions in this chart. Subject to change...

EOSDIS Vision 2020 (cloud relevant) Data Analysis at Scale Data and Processing Mobility Dataset Upgrading Virtual Collections Combining Data, Combining Tools

Evolution Review E&E Recommendation: The ESDIS Project should develop, implement and report on the outcome of prototypes to explore the advantages, risks, and costs of using commercial cloud environments for: Storage and data transfer […] Processing […] Improved data access […]

Existing prototyping efforts were restructured to investigate and test commercial cloud environments for essential EOSDIS and DAAC operations.

What does these consider?

Infrastructure, Scaling, Monitoring, Security, …

Development Model, Compliant Deployment, Operations, …

Application Architecture…

Systems Architecture

Outstanding Questions Can operational NASA science data systems function effectively in a commercial cloud? Do commercial clouds offer increased efficiency within a NASA environment and/or provide additional functionality? How do costs compare to opportunities and risks? How can NASA efficiently partner with commercial cloud providers to improve services and usage of data and services?

Looking forward Need broader knowledge of cloud architectures and patterns (e.g. NGAP, serverless, 12 factor applications, etc.) Rapid pace of change Potentially dramatic change in data usage patterns What about Cloud Native data formats? What about Cloud Native services?

Dan Pilone // dan@element84.com Thank you! Dan Pilone // dan@element84.com

This material is based upon work supported by the National Aeronautics and Space Administration under Contract Number NNG15HZ39C.