NPP Atmosphere PEATE Development APSPS: Atmosphere PEATE Science Processing System Scott Mindock Sample body slide from the second section of the presentation’s middle. For the first body slide of this second section, consider repeating the corresponding image from the mapping slide. Use the headline (no more than two lines) to say something about this topic. In the body of the slide, support that headline with images and parallel points (no more than four). See CSP, pages 116-117 and 147-151. Space Science and Engineering Center University of Wisconsin-Madison August 2007

Development Tools Used Subversion Eclipse Ant Bugzilla Java Sample body slide from the second section of the presentation’s middle. For the first body slide of this second section, consider repeating the corresponding image from the mapping slide. Use the headline (no more than two lines) to say something about this topic. In the body of the slide, support that headline with images and parallel points (no more than four). See CSP, pages 116-117 and 147-151.

Do these things Build and test before starting. Test during development Integrate test code. Check code in. Get fresh copy, build and test after check in. If functionality is added, code that tests functionality must also be added. Sample body slide from the second section of the presentation’s middle. For the first body slide of this second section, consider repeating the corresponding image from the mapping slide. Use the headline (no more than two lines) to say something about this topic. In the body of the slide, support that headline with images and parallel points (no more than four). See CSP, pages 116-117 and 147-151.

Development Process Sample body slide from the second section of the presentation’s middle. For the first body slide of this second section, consider repeating the corresponding image from the mapping slide. Use the headline (no more than two lines) to say something about this topic. In the body of the slide, support that headline with images and parallel points (no more than four). See CSP, pages 116-117 and 147-151.

Information http://whirlwind.ssec.wisc.edu/bugzilla/ Check out the system code svn co http://whirlwind/svn/APSPS APSPS Look at bugzilla http://whirlwind.ssec.wisc.edu/bugzilla/ Qualified algorithms http://whirlwind.ssec.wisc.edu/svn/qualifiedalgorithm/ Sample body slide from the second section of the presentation’s middle. For the first body slide of this second section, consider repeating the corresponding image from the mapping slide. Use the headline (no more than two lines) to say something about this topic. In the body of the slide, support that headline with images and parallel points (no more than four). See CSP, pages 116-117 and 147-151.

Development Process - Getting started Get the code from subversion svn co http://whirlwind/svn/APSPS APSPS ant clean dev ant clean dist - half hour When tests past you can start changing things Sample body slide from the second section of the presentation’s middle. For the first body slide of this second section, consider repeating the corresponding image from the mapping slide. Use the headline (no more than two lines) to say something about this topic. In the body of the slide, support that headline with images and parallel points (no more than four). See CSP, pages 116-117 and 147-151.

Configuration Management Subversion : Revision controlled system code. Revision controlled science application code. Revision controlled qualified binaries. Bugzilla : Bug tracking of system and applications. To do work scheduling and prioritization. Issue tracking Sample body slide from the second section of the presentation’s middle. For the first body slide of this second section, consider repeating the corresponding image from the mapping slide. Use the headline (no more than two lines) to say something about this topic. In the body of the slide, support that headline with images and parallel points (no more than four). See CSP, pages 116-117 and 147-151.

Nightly Builds Checks out code. Builds code. Tests code with direct interfaces. Tests code with service interfaces. Tests all packages. Runs system use case tests. If everything passes, checks into distribution repository. Sample body slide from the second section of the presentation’s middle. For the first body slide of this second section, consider repeating the corresponding image from the mapping slide. Use the headline (no more than two lines) to say something about this topic. In the body of the slide, support that headline with images and parallel points (no more than four). See CSP, pages 116-117 and 147-151.

Atmosphere PEATE Design Goals Use Commodity Hardware - x86 Use Open Source Software - Linux, Unix like(OSX) Scaleable: Laptops Desktops Clusters Maintainability: System lifetime spans years. Replace/Add hardware, software, people. Manage Dependencies. Automated Testing Reusability: Software Patterns. Development processes. Interfaces. (Public and Internal) Use of commodity software components - open source Trade off of use of scheduling vs pull driven. IDPS - Ocean - Standard cluster

The PEATE is Part of NASA NPOES SDS program. One of five. Ocean NASA Goddard Land NASA Goddard Ozone NASA Goddard Sounder NASA JPL Atmosphere UW-Madison SSEC Sample body slide from the second section of the presentation’s middle. For the first body slide of this second section, consider repeating the corresponding image from the mapping slide. Use the headline (no more than two lines) to say something about this topic. In the body of the slide, support that headline with images and parallel points (no more than four). See CSP, pages 116-117 and 147-151.

The NPP Mission for NASA: Continue the Climate Record NPP Sensor Payload Includes: Visible Infrared Imager Radiometer Suite (VIIRS) Cross-Track Infrared Sounder (CrIS) Advanced Technology Microwave Sounder (ATMS) Ozone Mapping & Profiling Sensor (OMPS) Launch is expected in 2010 NPP is intended to continue the climate record established by Terra and Aqua. NPP Science Products are known as “Environmental Data Records” or EDRs. NPP EDR Algorithms were developed by industry, not by the NPP Science Team. Land EDRs Surface Albedo Land Surface Temperature Snow Cover and Depth Surface Type Active Fires Ice Surface Temperature Vegetation Index Aerosol Optical Thickness Aerosol Particle Size Ocean EDRs Chlorophyll Sea Surface Temperature Ozone EDRs Ozone Total Column/Profile Atmosphere EDRs Cloud Mask Suspended Matter Cloud Cover/Layers Cloud Effective Particle Size Cloud Top Height Cloud Top Pressure Cloud Top Temperature Cloud Base Height Cloud Optical Thickness Sounder EDRs Vertical Moisture Profile Vertical Temperature Profile Vertical Pressure Profile

Why and How is the PEATE different from other systems at SSEC? Continuous, automated, testing of software systems. Continuous, automated, science product generation. Continuous, automated, global science product evaluation. Rapid retrospective global product generation (100 > real-time). (e.g., to create CDRs, or to evaluate alternative EDR algorithms). Simple Algorithm Integration Few Algorithm Implementation Restrictions. No science to operations transformation required. Rule based process triggers. Truly scalable system SOA - Service Orientated Architecture Sample body slide from the second section of the presentation’s middle. For the first body slide of this second section, consider repeating the corresponding image from the mapping slide. Use the headline (no more than two lines) to say something about this topic. In the body of the slide, support that headline with images and parallel points (no more than four). See CSP, pages 116-117 and 147-151.

Atmosphere PEATE Science Processing System (APSPS)

Algorithm Lifecycle Algorithm can come from anywhere. Once qualified, the algorithm can be applied from ARM.

Science Processing System Trade Studies and Key Decisions We examined NASA Ocean SDPS and MODIS Land Processing Systems. Lessons learned: Recipe-based approach to running science algorithms (system doesn’t care what the algorithm is, as long as it knows how to assemble the ingredients to make the recipe) Cluster of compute resources (no need for a large shared memory computer) Decouple the components of the processing system (store, compute, distribute) Use commodity hardware/software (e.g., Rackmount Intel/AMD servers, Linux) Key Design Decisions: Create a system where individual components have loose dependencies on each other. Leverage existing cluster processing hardware infrastructure and knowledge base. Use established software technologies where possible to speed development time (e.g, Eclipse, SOAP) Create a system which is scalable, efficient, and cost effective.

Implementation Technologies Java: Established high level language. Strict type checking. Good tool support. (Eclipse,Ant,junit,log4j) Good integration with Web Technologies. (Web Services,Axis2) UML: Software Industry Standard. (Standard Diagram Examples) Good tool support. (Eclipse plugins) Provides standard methods of describing/designing architectures. XML: Established Cross platform Technology Sample body slide from the second section of the presentation’s middle. For the first body slide of this second section, consider repeating the corresponding image from the mapping slide. Use the headline (no more than two lines) to say something about this topic. In the body of the slide, support that headline with images and parallel points (no more than four). See CSP, pages 116-117 and 147-151.

Development Environment Eclipse : UML, code framing, reverse engineering. Web Tools, web service exploring C++, contractor algorithms, LEOCAT Fortran, contractor algorithms, LEOCAT Subversion Ant + junit Development Builds Night Builds, Automated Tests and Distribution Ant works inside and out of Eclipse. Sample body slide from the second section of the presentation’s middle. For the first body slide of this second section, consider repeating the corresponding image from the mapping slide. Use the headline (no more than two lines) to say something about this topic. In the body of the slide, support that headline with images and parallel points (no more than four). See CSP, pages 116-117 and 147-151.

PEATE Software Architecture ING - Ingest System Brings external data into system DMS - Data Management System Holds Data Files Requires Unique Names Preserves Unique Names CRG - Computation Resource Grid Implements simple scheduling ARM - Algorithm Rule Manager Loosely Coupled Web Services Maintainable Scalable Sample body slide from the second section of the presentation’s middle. For the first body slide of this second section, consider repeating the corresponding image from the mapping slide. Use the headline (no more than two lines) to say something about this topic. In the body of the slide, support that headline with images and parallel points (no more than four). See CSP, pages 116-117 and 147-151.

PEATE Service Activities Sample body slide from the second section of the presentation’s middle. For the first body slide of this second section, consider repeating the corresponding image from the mapping slide. Use the headline (no more than two lines) to say something about this topic. In the body of the slide, support that headline with images and parallel points (no more than four). See CSP, pages 116-117 and 147-151.

ARM - Algorithm Rule Manager Allows rule based algorithm execution. Match rules to incoming data. Uses regular expressions. Provides rule status. Sample body slide from the second section of the presentation’s middle. For the first body slide of this second section, consider repeating the corresponding image from the mapping slide. Use the headline (no more than two lines) to say something about this topic. In the body of the slide, support that headline with images and parallel points (no more than four). See CSP, pages 116-117 and 147-151.

DMS - Data Management System All files stored in files system with original names. Database compliments file system. Web Services provide distributed access. Data Migration for maintenance. Sample body slide from the second section of the presentation’s middle. For the first body slide of this second section, consider repeating the corresponding image from the mapping slide. Use the headline (no more than two lines) to say something about this topic. In the body of the slide, support that headline with images and parallel points (no more than four). See CSP, pages 116-117 and 147-151.

CRG - Computation Resource Grid Provides job list of algorithms and data ready to be processed. Provides status of jobs Implements simple scheduling. Sample body slide from the second section of the presentation’s middle. For the first body slide of this second section, consider repeating the corresponding image from the mapping slide. Use the headline (no more than two lines) to say something about this topic. In the body of the slide, support that headline with images and parallel points (no more than four). See CSP, pages 116-117 and 147-151.

ALG - Algorithm host Consumes web services. Jobs from CRG. Uses DMS for inputs and outputs. Runs on nodes or available CPUs. Obtains Binaries from subversion. Sample body slide from the second section of the presentation’s middle. For the first body slide of this second section, consider repeating the corresponding image from the mapping slide. Use the headline (no more than two lines) to say something about this topic. In the body of the slide, support that headline with images and parallel points (no more than four). See CSP, pages 116-117 and 147-151.

ALG - Algorithm host Sample body slide from the second section of the presentation’s middle. For the first body slide of this second section, consider repeating the corresponding image from the mapping slide. Use the headline (no more than two lines) to say something about this topic. In the body of the slide, support that headline with images and parallel points (no more than four). See CSP, pages 116-117 and 147-151.

PEATE Deployment - 9/5/2007 Sample body slide from the second section of the presentation’s middle. For the first body slide of this second section, consider repeating the corresponding image from the mapping slide. Use the headline (no more than two lines) to say something about this topic. In the body of the slide, support that headline with images and parallel points (no more than four). See CSP, pages 116-117 and 147-151.

PEATE Roadmap Short term: Scaling test. ING / DMS Deployment. DMS Recovery Techniques. DMS Status Server. CRG/ARM database integration. 1.5 VIIRS cloud mask deployment. Long Term: Multiple DMS Services - Service Chaining GOES Archive Discovery Services: Advertise Data and Computational Resources. WSIL UDDI Sample body slide from the second section of the presentation’s middle. For the first body slide of this second section, consider repeating the corresponding image from the mapping slide. Use the headline (no more than two lines) to say something about this topic. In the body of the slide, support that headline with images and parallel points (no more than four). See CSP, pages 116-117 and 147-151.

APSPS Variations Single Computer: System can run on single computer with or without Web Services Good for debugging System is well suited for running binaries on multi-core systems. Work Group: APSPS can be run on group of PC. DMS can be used to share data. Cluster: APSPS (Alghost) can run on cluster nodes providing extra CPUs. Web Service Variations: Different implementations of web services can exist. Specialized ARM for McIDAS V integration. Specialized CRG for unique job scheduling. Sample body slide from the second section of the presentation’s middle. For the first body slide of this second section, consider repeating the corresponding image from the mapping slide. Use the headline (no more than two lines) to say something about this topic. In the body of the slide, support that headline with images and parallel points (no more than four). See CSP, pages 116-117 and 147-151.