1. Space Science and Engineering Center University of Wisconsin-Madison Space Science and Engineering Center University of Wisconsin-Madison 1 NPP Atmosphere PEATE Peer Review Space Science and Engineering Center University of Wisconsin-Madison August 2007 APSPS: Atmosphere PEATE Science Processing System Liam E. Gumley, Hank Revercomb, Scott Mindock, Steve Dutcher, Robert, Andy Heidinger, Mike Pavolonis, Richard Frey, Bryan Baum, Paolo Antonelli, Bruce Flynn, Rick Jensen

2. Space Science and Engineering Center University of Wisconsin-Madison Space Science and Engineering Center University of Wisconsin-Madison 2 What is the Atmosphere PEATE?

3. Space Science and Engineering Center University of Wisconsin-Madison Space Science and Engineering Center University of Wisconsin-Madison 3 Part of NASA NPOES SDS program. One of five. Ocean NASA Goddard Land NASA Goddard Ozone NASA Goddard Sounder NASA JPL Atmosphere UW-Madison SSEC The PEATE is

4. Space Science and Engineering Center University of Wisconsin-Madison Space Science and Engineering Center University of Wisconsin-Madison 4 The NPP Mission for NASA: Continue the Climate Record NPP Sensor Payload Includes: –Visible Infrared Imager Radiometer Suite (VIIRS) –Cross-Track Infrared Sounder (CrIS) –Advanced Technology Microwave Sounder (ATMS) –Ozone Mapping & Profiling Sensor (OMPS) Launch is expected in 2010 Land EDRs Surface Albedo Land Surface Temperature Snow Cover and Depth Surface Type Active Fires Ice Surface Temperature Vegetation Index Aerosol Optical Thickness Aerosol Particle Size Ocean EDRs Chlorophyll Sea Surface Temperature Ozone EDRs Ozone Total Column/Profile Atmosphere EDRs Cloud Mask Suspended Matter Cloud Cover/Layers Cloud Effective Particle Size Cloud Top Height Cloud Top Pressure Cloud Top Temperature Cloud Base Height Cloud Optical Thickness Sounder EDRs Vertical Moisture Profile Vertical Temperature Profile Vertical Pressure Profile NPP is intended to continue the climate record established by Terra and Aqua. NPP Science Products are known as “Environmental Data Records” or EDRs. NPP EDR Algorithms were developed by industry, not by the NPP Science Team.

5. Space Science and Engineering Center University of Wisconsin-Madison Space Science and Engineering Center University of Wisconsin-Madison 5 Atmosphere PEATE Science Processing System (APSPS)

6. Space Science and Engineering Center University of Wisconsin-Madison Space Science and Engineering Center University of Wisconsin-Madison 6 Algorithm can come from anywhere. Once qualified, the algorithm can be applied from ARM. Algorithm Lifecycle

7. Space Science and Engineering Center University of Wisconsin-Madison Space Science and Engineering Center University of Wisconsin-Madison 7 Continuous, automated, testing of software systems. Continuous, automated, science product generation. Continuous, automated, global science product evaluation. Rapid retrospective global product generation (100 > real-time). (e.g., to create CDRs, or to evaluate alternative EDR algorithms). Simple Algorithm Integration Few Algorithm Implementation Restrictions. No science to operations transformation required. Rule based process triggers. Truly scalable system SOA - Service Orientated Architecture Why and How is the PEATE different from other systems at SSEC?

8. Space Science and Engineering Center University of Wisconsin-Madison Space Science and Engineering Center University of Wisconsin-Madison 8 We examined NASA Ocean SDPS and MODIS Land Processing Systems. Lessons learned: Recipe-based approach to running science algorithms (system doesn’t care what the algorithm is, as long as it knows how to assemble the ingredients to make the recipe) Cluster of compute resources (no need for a large shared memory computer) Decouple the components of the processing system (store, compute, distribute) Use commodity hardware/software (e.g., Rackmount Intel/AMD servers, Linux) Key Design Decisions: Create a system where individual components have loose dependencies on each other. Leverage existing cluster processing hardware infrastructure and knowledge base. Use established software technologies where possible to speed development time (e.g, Eclipse, SOAP) Create a system which is scalable, efficient, and cost effective. Science Processing System Trade Studies and Key Decisions

9. Space Science and Engineering Center University of Wisconsin-Madison Space Science and Engineering Center University of Wisconsin-Madison 9 Use Commodity Hardware - x86 Use Open Source Software - Linux, Unix like(OSX) Scaleable: Laptops Desktops Clusters Maintainability: System lifetime spans years. Replace/Add hardware, software, people. Manage Dependencies. Automated Testing Reusability: Software Patterns. Development processes. Interfaces. (Public and Internal) Atmosphere PEATE Design Goals

10. Space Science and Engineering Center University of Wisconsin-Madison Space Science and Engineering Center University of Wisconsin-Madison 10 Subversion : Revision controlled system code. Revision controlled science application code. Revision controlled qualified binaries. Bugzilla : Bug tracking of system and applications. To do work scheduling and prioritization. Issue tracking What technologies are being used?

11. Space Science and Engineering Center University of Wisconsin-Madison Space Science and Engineering Center University of Wisconsin-Madison 11 Checks out code. Builds code. Tests code with direct interfaces. Tests code with service interfaces. Tests all packages. Runs system use case tests. If everything passes, checks into distribution repository. Nightly Builds

12. Space Science and Engineering Center University of Wisconsin-Madison Space Science and Engineering Center University of Wisconsin-Madison 12 Java: Established high level language. Strict type checking. Good tool support. (Eclipse,Ant,junit,log4j) Good integration with Web Technologies. (Web Services,Axis2) UML: Software Industry Standard. (Standard Diagram Examples) Good tool support. (Eclipse plugins) Provides standard methods of describing/designing architectures. XML: Established Cross platform Technology Implementation Technologies

13. Space Science and Engineering Center University of Wisconsin-Madison Space Science and Engineering Center University of Wisconsin-Madison 13 Eclipse : UML, code framing, reverse engineering. Web Tools, web service exploring C++, contractor algorithms, LEOCAT Fortran, contractor algorithms, LEOCAT Subversion Ant + junit Development Builds Night Builds, Automated Tests and Distribution Ant works inside and out of Eclipse. Development Environment

14. Space Science and Engineering Center University of Wisconsin-Madison Space Science and Engineering Center University of Wisconsin-Madison 14 ING - Ingest System Brings external data into system DMS - Data Management System Holds Data Files Requires Unique Names Preserves Unique Names CRG - Computation Resource Grid Implements simple scheduling ARM - Algorithm Rule Manager Loosely Coupled Web Services Maintainable Scalable PEATE Software Architecture

15. Space Science and Engineering Center University of Wisconsin-Madison Space Science and Engineering Center University of Wisconsin-Madison 15 Sample Activity Diagram Like Flowchart Ovals = Activity, Rectangle = Data Action A and B Decoupled Dot = start, Circle = end Swim Lanes = system boundaries

16. Space Science and Engineering Center University of Wisconsin-Madison Space Science and Engineering Center University of Wisconsin-Madison 16 PEATE Service Activities

17. Space Science and Engineering Center University of Wisconsin-Madison Space Science and Engineering Center University of Wisconsin-Madison 17 Allows rule based algorithm execution. Match rules to incoming data. Uses regular expressions. Provides rule status. ARM - Algorithm Rule Manager

18. Space Science and Engineering Center University of Wisconsin-Madison Space Science and Engineering Center University of Wisconsin-Madison 18 All files stored in files system with original names. Database compliments file system. Web Services provide distributed access. Data Migration for maintenance. DMS - Data Management System

19. Space Science and Engineering Center University of Wisconsin-Madison Space Science and Engineering Center University of Wisconsin-Madison 19 Provides job list of algorithms and data ready to be processed. Provides status of jobs Implements simple scheduling. CRG - Computation Resource Grid

20. Space Science and Engineering Center University of Wisconsin-Madison Space Science and Engineering Center University of Wisconsin-Madison 20 Consumes web services. Jobs from CRG. Uses DMS for inputs and outputs. Runs on nodes or available CPUs. Obtains Binaries from subversion. ALG - Algorithm host

21. Space Science and Engineering Center University of Wisconsin-Madison Space Science and Engineering Center University of Wisconsin-Madison 21 ALG - Algorithm host

22. Space Science and Engineering Center University of Wisconsin-Madison Space Science and Engineering Center University of Wisconsin-Madison 22 PEATE Deployment - 9/5/2007

23. Space Science and Engineering Center University of Wisconsin-Madison Space Science and Engineering Center University of Wisconsin-Madison 23 Short term: Scaling test. ING / DMS Deployment. DMS Recovery Techniques. DMS Status Server. CRG/ARM database integration. 1.5 VIIRS cloud mask deployment. Long Term: Multiple DMS Services - Service Chaining GOES Archive Discovery Services: Advertise Data and Computational Resources. WSIL UDDI PEATE Roadmap

24. Space Science and Engineering Center University of Wisconsin-Madison Space Science and Engineering Center University of Wisconsin-Madison 24 Single Computer: System can run on single computer with or without Web Services Good for debugging System is well suited for running binaries on multi-core systems. Work Group: APSPS can be run on group of PC. DMS can be used to share data. Cluster: APSPS (Alghost) can run on cluster nodes providing extra CPUs. Web Service Variations: Different implementations of web services can exist. Specialized ARM for McIDAS V integration. Specialized CRG for unique job scheduling. APSPS Variations