Transition of Science Algorithms into Enterprise Product Generation Operations 2016 AMS Annual Meeting Dylan Powell, Ph.D. Lockheed Martin ESPDS Science Engineer Rich Baker, Solers Inc. Peter MacHarrie, Solers Inc. Ron Niemann, Solers Inc. Dan Beall, Solers Inc.

2 2 Svalbard S-NPP, GCOM-W1 JPSS CGS IDPS JPSS CGS IDPS ESPDS NDE 1.0 Ingest Product Generation Product Management SDRs & EDRs (xDRs) Product Distribution: - S-NPP xDRs - NESDIS Unique Products (NUPs) - Tailored Products - GCOM-W L1/L2 Products Ancillary Data NASA VIIRS Image ESPC NDE 1.0 Storage Product and Ancillary Data Files Product Distribution ESPC Data Distribution Services (DDS) External/Internal Users Archive – CLASS (NUPs only) External/Internal Users Archive – CLASS (NUPs only)

3 3 Svalbard/Fairbanks/ McMurdo S-NPP, JPSS-1/2, GCOM-W1 JPSS CGS IDPS JPSS CGS IDPS NOAA Product Generation Services (NDE 2.0) NOAA Product Generation Services (NDE 2.0) Ingest Product Generation Product Management SDRs & EDRs (xDRs) Products File Availability Notifications: - S-NPP/JPSS xDRs - NESDIS Unique Products (NUPs) - Tailored Products - GCOM-W L1/L2 Products Ancillary Data File Availability Notifications NASA VIIRS Image ESPDS Common Infrastructure Platform PDA (PD) Services Product Distribution to Consumers Ancillary Data Acquisition Enterprise Shared Storage Product and Ancillary Data Files External/Internal Users Archive – CLASS (NUPs only) External/Internal Users Archive – CLASS (NUPs only)

 NOAA Data Exploitation (NDE) 1.0 in operations since July 2014 and has been distributing critical S-NPP data since soon after launch (November 2011)  NDE 2.0 provides an Enterprise Product Generation (PG) framework implemented in a SOA within the ESPDS common infrastructure platform  Established and mature process for efficiently transitioning new and updated science algorithms from the NOAA Center for Satellites Applications and Research (STAR) - in place since 2011  Well defined, documented, and simple interface for “plug-and-play” algorithm integration allowing for a repeatable process and small integration team  Data agnostic, execute algorithms for any platform/instrument (“algorithm as a service” concept)  Established solution for product generation for S-NPP, JPSS-1/2, GCOM- W1, and can be leveraged for other NOAA or non-NOAA missions 4

5 Ingest Dec 2014 Dec 2015 Ingested Data103 TB111 TB Product Generation Number of Production Jobs1,510,9971,591,465 Generated Data25 TB27 TB Production Job Success Rate99.992%99.990% Product Distribution Distributed Data29 TB34 TB Subscription Based Services Number of Users2934 Number of Subscriptions (active)

66 Develop Algorithm Preliminary Design Review Critical Design Review Unit Test Readiness Review System (Algorithm) Readiness Review Prepare Delivered Algorithm Package (DAP) for each Science Algorithm for delivery to NDE Algorithm Delivery Standards, Integration, and Test (DAP document acts as ICD) Configuration management of DAP Develop Algorithm Enhancements, Bug Fixes, and Updates Redeliver DAP Develop/test in NDE DEV Environment NDE operating system/compilers Science libraries VPN access Initial integration activities Integrate DAP into NDE DEV Environment Configuration management Production rules Unit testing System Test Algorithm in NDE TEST Environment Full contention, live data flow Performance testing Generate Operational Product in NDE Production Environment Operations 24/7 Monitoring Satellite Products and Services Review Board (SPSRB) approves product for operations STAR NDE KEY Calibration/ Validation Office of Satellite and Product Operations (OSPO) performs software code review 4 Define Requirements (JPSS Level 1) 8

7 ESPDS Common Infrastructure Platform NOAA Data Exploitation (NDE) PG Enterprise Shared Storage Science Algorithm - Single Driver Script - Read Process Control File (PCF) - Write output files to Process Status File (PSF) Input Data Science Algorithm - Log warnings and errors to master log file - Return code PCF PSF Logs Return Code Working Directory - Temporary directory for execution of production job - Dynamic input data - Write the PCF Input/Output Data PCF PSF Logs Return Code Working Directory - Destroyed after successful execution - Failed jobs cause directory to be stored for later analysis Input/Output Data

AlgorithmPlatform/Primary Instrument NDE 1.0 Operations Status ESPDS PG (NDE 2.0) Baseline Microwave Integrated Retrieval System (MiRS) S-NPP/ATMSOperational 2014QST-1 Advanced Clear-Sky Processor for Oceans (ACSPO) S-NPP/VIIRSOperational 2014QST-1 NOAA Unique CrIS/ATMS Processing System (NUCAPS) S-NPP/CrISOperational 2014QST-1 VIIRS Polar Winds (VPW)S-NPP/VIIRSOperational 2014QST-1 Green Vegetation Fraction (GVF)S-NPP/VIIRSOperational 2014QST-1 S-NPP Tropical CycloneS-NPP/ATMSOperational 2015QST-1 Vegetation Health (VH)S-NPP/VIIRSOperational 2015QST-1 GCOM-W1 AMSR2 Algorithm Software Processor (GAASP) GCOM-W1/AMSR2Operational 2015QST-1 Active Fires (AF)S-NPP/VIIRSOperational 2016QST-2 OMPS-Limb Profiler SDRS-NPP/OMPS-LPOperational 2016ORR 8

AlgorithmPlatform/Primary Instrument NDE 1.0 Operations Status ESPDS PG (NDE 2.0) Baseline OMPS-Limb Profiler EDRS-NPP/OMPS-LPOperational 2016ORR JPSS Risk Reduction (Clouds, Aerosols, Cryosphere) S-NPP, JPSS/VIIRSN/AQST-1 GCOM-W1 AMSR2 SDRGCOM-W1/AMSR2N/AQST-1 9

Microwave Integrated Retrieval System (MiRS) Advanced Clear-Sky Processor for Oceans (ACSPO) Sea Surface Temperature Total Precipitable Water

11 NOAA Unique CrIS/ATMS Processing System (NUCAPS) VIIRS IR Image AWIPS2 NUCAPS Sounding

12 Green Vegetation Fraction (GVF) VIIRS Polar Winds (VPW)

13 Vegetation Health (VH) Tropical Cyclones (TC)

14 Aggregation Resample NDE VIIRS Image Filtering Remapping NDE VIIRS RGB NDE VIIRS RGB NDE VIIRS RGB 375 m 3 km NDE ACSPO SST

15 Funny River Fire, Alaska VIIRS I1 (Visible) VIIRS I3 (Fire)

16 JPSS Risk Reduction Products Clouds Snow/Ice Aerosols GCOM AMSR2 Products Winds Clouds Soil Moisture