1. Space Science and Engineering Center University of Wisconsin-Madison Space Science and Engineering Center University of Wisconsin-Madison 1 NPP Atmosphere PEATE Critical Design Review Presented by the Atmosphere PEATE Team Space Science and Engineering Center University of Wisconsin-Madison 17 April 2008 Part 2: Science Processing System Scott Mindock

2. Space Science and Engineering Center University of Wisconsin-Madison Space Science and Engineering Center University of Wisconsin-Madison 2 Design System Subsystems Software Development and Verification Science Investigation Support Logistics Agenda

3. Space Science and Engineering Center University of Wisconsin-Madison Space Science and Engineering Center University of Wisconsin-Madison 3 UML - Unified Modeling Language Use Case - Captures requirements at functional level Activity - Describes the steps of a use case Package - Show software structure and dependencies Class - Show software structure and dependencies Web Services - SOAP XML based communications Definitions

4. Space Science and Engineering Center University of Wisconsin-Madison Space Science and Engineering Center University of Wisconsin-Madison 4 Sample Activity Diagram Like Flowchart Ovals = Activity Rectangle = Data Action A and B Decoupled Dot = start Circle = end

5. Space Science and Engineering Center University of Wisconsin-Madison Space Science and Engineering Center University of Wisconsin-Madison 5 Maintainability - System lifetime spans years Reusability - Subsystems and design patterns Testability - Subsystem dependencies are managed Scalability - Design must scale to larger and smaller systems Design: Goals (1 of 3)

6. Space Science and Engineering Center University of Wisconsin-Madison Space Science and Engineering Center University of Wisconsin-Madison 6 Learn from others - IDPS, Clusters, Ocean PEATE, Disney Parkwide Leverage existing proven technologies and standards Java Web Services - Provide communications between subsystems Apache Tomcat - Web Application Server Apache Axis2 - Web Application providing web services JAXB - Java XML Binding ANT - XML Structured make Eclipse - Java IDE Subversion - Revision control Prototype key features - Demonstration Projects Loosely Coupled System - Manage dependencies, Define interfaces Design: Methodology (2 of 3)

7. Space Science and Engineering Center University of Wisconsin-Madison Space Science and Engineering Center University of Wisconsin-Madison 7 All systems use XML configuration. All configurations have a XML schema. System persistence leverages schema based code generation. All systems can persist to DB or XML (scalable) Major components are deployed web services. (decouple, scalable) Test patterns used for scenario based testing / system verification Design: Patterns ( 3 of 3 )

8. Space Science and Engineering Center University of Wisconsin-Madison Space Science and Engineering Center University of Wisconsin-Madison 8 External Relationships Subsystems System Activity System

9. Space Science and Engineering Center University of Wisconsin-Madison Space Science and Engineering Center University of Wisconsin-Madison 9 System: External Relationships

10. Space Science and Engineering Center University of Wisconsin-Madison Space Science and Engineering Center University of Wisconsin-Madison 10 DMS : Data Management System Stores Data CRG : Computational Resource Grid Job Management ARM : Algorithm Rule Management Applies Product Rules to Data ING : Ingest System Brings Data into System RUL : Product Production rules Ties software packages to data ALG : Algorithm Host Runs software packages Subsystem Relationships Arrows denote dependency System design minimizes dependencies Eases maintenance

11. Space Science and Engineering Center University of Wisconsin-Madison Space Science and Engineering Center University of Wisconsin-Madison 11 System Activity Diagram

12. Space Science and Engineering Center University of Wisconsin-Madison Space Science and Engineering Center University of Wisconsin-Madison 12 DMS: Data Management System Stores Data CRG: Computational Resource Grid Processes Data ARM: Algorithm Rule Manage Applies Product Rules to Data ING : Ingest System Brings Data into System Subsystem Relationships @ PDR @PDR

13. Space Science and Engineering Center University of Wisconsin-Madison Space Science and Engineering Center University of Wisconsin-Madison 13 System Activity Diagram @ PDR @PDR

14. Space Science and Engineering Center University of Wisconsin-Madison Space Science and Engineering Center University of Wisconsin-Madison 14 ING - Ingest brings data into the Atmosphere PEATE DMS - Data Management System, stores data in the form of files. Provides a web service interface to locate and retrieve files. CRG ALG ARM RUL Subsystems

15. Space Science and Engineering Center University of Wisconsin-Madison Space Science and Engineering Center University of Wisconsin-Madison 15 DMS : Data Management System Stores Data CRG : Computational Resource Grid Job Management ARM : Algorithm Rule Management Applies Product Rules to Data ING : Ingest System Brings Data into System RUL : Product Production rules Ties software packages to data ALG : Algorithm Host Runs software packages System Components, Java Packages

16. Space Science and Engineering Center University of Wisconsin-Madison Space Science and Engineering Center University of Wisconsin-Madison 16 System Activity Diagram

17. Space Science and Engineering Center University of Wisconsin-Madison Space Science and Engineering Center University of Wisconsin-Madison 17 Supports pull ingests Supports push ingests Verification functions localized Notices missing or late data Dedicated Machine Deploy Multiple Copies ING

18. Space Science and Engineering Center University of Wisconsin-Madison Space Science and Engineering Center University of Wisconsin-Madison 18 Schema based persistence XML ingest type configuration Support for FTP, RSYNC ING Configuration

19. Space Science and Engineering Center University of Wisconsin-Madison Space Science and Engineering Center University of Wisconsin-Madison 19 DMS : Data Management System Stores Data CRG : Computational Resource Grid Job Management ARM : Algorithm Rule Management Applies Product Rules to Data ING : Ingest System Brings Data into System RUL : Product Production rules Ties software packages to data ALG : Algorithm Host Runs software packages System Components, Java Packages

20. Space Science and Engineering Center University of Wisconsin-Madison Space Science and Engineering Center University of Wisconsin-Madison 20 System Activity Diagram

21. Space Science and Engineering Center University of Wisconsin-Madison Space Science and Engineering Center University of Wisconsin-Madison 21 Provides well-defined interface deployed as a web service. DMS is autonomous Provides storage Provides catalog Spans multiple file servers on a network DMS = Linux File system + Postgres database DMS

22. Space Science and Engineering Center University of Wisconsin-Madison Space Science and Engineering Center University of Wisconsin-Madison 22 Each file in DMS has associated entry in catalog Important file characteristics are tracked File are distributed through system on entry DMS File Properties

23. Space Science and Engineering Center University of Wisconsin-Madison Space Science and Engineering Center University of Wisconsin-Madison 23 Schema defines DMS directory structure DMS creates directories on installation DMS uses round robin method to fill file systems Configuration not utilizing DB are restricted to one machine Max disk utilization is specified DMS Configuration

24. Space Science and Engineering Center University of Wisconsin-Madison Space Science and Engineering Center University of Wisconsin-Madison 24 File system - hold files Database - holds file information Public Access - implement DMS interface Worker - manages file system Two flavors With DB - spans multiple machines Without DB - single machine DMS Component and Deployment

25. Space Science and Engineering Center University of Wisconsin-Madison Space Science and Engineering Center University of Wisconsin-Madison 25 DMS : Data Management System Stores Data CRG : Computational Resource Grid Job Management ARM : Algorithm Rule Management Applies Product Rules to Data ING : Ingest System Brings Data into System RUL : Product Production rules Ties software packages to data ALG : Algorithm Host Runs software packages System Components, Java Packages

26. Space Science and Engineering Center University of Wisconsin-Madison Space Science and Engineering Center University of Wisconsin-Madison 26 System Activity Diagram

27. Space Science and Engineering Center University of Wisconsin-Madison Space Science and Engineering Center University of Wisconsin-Madison 27 Provides well-defined interface deployed as a web service. Accepts job request Provides Job Status Monitor Job State Scalable Testable CRG

28. Space Science and Engineering Center University of Wisconsin-Madison Space Science and Engineering Center University of Wisconsin-Madison 28 DMS : Data Management System Stores Data CRG : Computational Resource Grid Job Management ARM : Algorithm Rule Management Applies Product Rules to Data ING : Ingest System Brings Data into System RUL : Product Production rules Ties software packages to data ALG : Algorithm Host Runs software packages System Components, Java Packages

29. Space Science and Engineering Center University of Wisconsin-Madison Space Science and Engineering Center University of Wisconsin-Madison 29 System Activity Diagram

30. Space Science and Engineering Center University of Wisconsin-Madison Space Science and Engineering Center University of Wisconsin-Madison 30 One or more per node Retrieves data from DMS Retrieves and runs software packages Saves results to DMS Consumes user jobs by using CRG Web Service ALG

31. Space Science and Engineering Center University of Wisconsin-Madison Space Science and Engineering Center University of Wisconsin-Madison 31 DMS : Data Management System Stores Data CRG : Computational Resource Grid Job Management ARM : Algorithm Rule Management Applies Product Rules to Data ING : Ingest System Brings Data into System RUL : Product Production rules Ties software packages to data ALG : Algorithm Host Runs software packages System Components, Java Packages

32. Space Science and Engineering Center University of Wisconsin-Madison Space Science and Engineering Center University of Wisconsin-Madison 32 System Activity Diagram

33. Space Science and Engineering Center University of Wisconsin-Madison Space Science and Engineering Center University of Wisconsin-Madison 33 Provides well-defined interface deployed as a web service. Assigns jobs to CRG Monitors data in DMS Monitors the status of jobs in CRG Applies rules to data Generates notifications Volatile logic lives here Provides extension point for rules Rules can be added or removed dynamically ARM

34. Space Science and Engineering Center University of Wisconsin-Madison Space Science and Engineering Center University of Wisconsin-Madison 34 DMS : Data Management System Stores Data CRG : Computational Resource Grid Job Management ARM : Algorithm Rule Management Applies Product Rules to Data ING : Ingest System Brings Data into System RUL : Product Production rules Ties software packages to data ALG : Algorithm Host Runs software packages System Components, Java Packages

35. Space Science and Engineering Center University of Wisconsin-Madison Space Science and Engineering Center University of Wisconsin-Madison 35 System Activity Diagram

36. Space Science and Engineering Center University of Wisconsin-Madison Space Science and Engineering Center University of Wisconsin-Madison 36 Mechanism to deploy algorithms into APSPS Rules have names Rules use regular expressions URL of software package specified Destination DMS can be specified Days of lease can be specified RUL

37. Space Science and Engineering Center University of Wisconsin-Madison Space Science and Engineering Center University of Wisconsin-Madison 37 System Activity Diagram

38. Space Science and Engineering Center University of Wisconsin-Madison Space Science and Engineering Center University of Wisconsin-Madison 38 Used by ARM Specified by user Specifies software package to run (SoftwarePackage) Specifies data of interest (FilterRegex) Specifies product output location (DmsUrl) Specifies product lifetime (DaysOfLease) Rules and Software Packages

39. Space Science and Engineering Center University of Wisconsin-Madison Space Science and Engineering Center University of Wisconsin-Madison 39 Development process Testing Strategy Defect Reporting Defect Correction Configuration Management Nightly Build Unit and Regression Testing Testing Scenarios Requirements Mapping Software Development and Verification

40. Space Science and Engineering Center University of Wisconsin-Madison Space Science and Engineering Center University of Wisconsin-Madison 40 Spiral Method Design Implement Test Deploy Development Process

41. Space Science and Engineering Center University of Wisconsin-Madison Space Science and Engineering Center University of Wisconsin-Madison 41 We use standard software industry practices Automate with ANT Test with JUNIT Saves Money and Increases Quality Tests are reproducible Tests can be run more often than they would be if they were manual Tests are improved over time We don’t just build, the process includes testing and verification Testing Strategy

42. Space Science and Engineering Center University of Wisconsin-Madison Space Science and Engineering Center University of Wisconsin-Madison 42 Anyone with an account can report a bug Reporter chooses software system Reporter enters bug information Owner of project is notified automatically by email Owner prioritizes Defect Reporting

43. Space Science and Engineering Center University of Wisconsin-Madison Space Science and Engineering Center University of Wisconsin-Madison 43 Verify software before starting Verify problem exists Fix and Verify Add back to code base Defect Correction

44. Space Science and Engineering Center University of Wisconsin-Madison Space Science and Engineering Center University of Wisconsin-Madison 44 Leverages open source software Provides versioned access to software Provides defect-tracking facilities Used for source code Used for test scripts Used for software packages Backup is made on regular basis Accounts are password protected Configuration Management

45. Space Science and Engineering Center University of Wisconsin-Madison Space Science and Engineering Center University of Wisconsin-Madison 45 Provides versioned access to software Provides defect-tracking facilities Used for source code Used for test scripts Used for binaries Configuration Management @ PDR @PDR

46. Space Science and Engineering Center University of Wisconsin-Madison Space Science and Engineering Center University of Wisconsin-Madison 46 Source code management Software packages distributed to cluster as subversion repositories Software packages may contain large static data sets Issue: Subversion retrieval of large binary datasets slow Subversion stores two copies for diff purposes Solution: Use two repositories Checkout executing software package Export static binary data Verified solution with subversion contributor Subversion

47. Space Science and Engineering Center University of Wisconsin-Madison Space Science and Engineering Center University of Wisconsin-Madison 47 Builds system Tests subsystems Tests scenarios Updates repositories Logs results Scenarios demonstrate requirements Nightly Build

48. Space Science and Engineering Center University of Wisconsin-Madison Space Science and Engineering Center University of Wisconsin-Madison 48 May use internal knowledge interfaces for testing Test and exercise public interfaces Stress test interfaces Evolve to test and verify bugs Tests run in Eclipse Test run from command line Fixed defects have specific tests added Tests run in nightly Tests verify release Layered approach to testing Everything tested, Every Night Unit and Regression Testing

49. Space Science and Engineering Center University of Wisconsin-Madison Space Science and Engineering Center University of Wisconsin-Madison 49 Test ingest function Test forward and reprocess functions Tests rule execution Test user extension points Testing Scenarios - Activity Diagram

50. Space Science and Engineering Center University of Wisconsin-Madison Space Science and Engineering Center University of Wisconsin-Madison 50 Single Rule - Simplest test, connects data with software Double Rule - Chain products together Double Rule Reprocess Half - Chains products, reprocess second half of chain MODIS L1->CM - real world Single MODIS L1-> CM and LEOCAT - real world Double Scenario / Use Case Test Instances

51. Space Science and Engineering Center University of Wisconsin-Madison Space Science and Engineering Center University of Wisconsin-Madison 51 Level 3 requirements are mapped to level 4 requirements in requirements specification, 3600-0003.080402.doc Operations Concepts illustrates requirements, 3600-0004.060911.doc Scenario tests implement Operations Concepts use cases Scenario tests have several instances Generic instances exercise system. Specific instances implement Ops Con use cases Specific instances currently use MODIS and LEOCAT Requirements Mapping

52. Space Science and Engineering Center University of Wisconsin-Madison Space Science and Engineering Center University of Wisconsin-Madison 52 Algorithm Lifecycle Algorithm Ingest Algorithm Qualification LEOCAT Science Investigation Support

53. Space Science and Engineering Center University of Wisconsin-Madison Space Science and Engineering Center University of Wisconsin-Madison 53 Algorithm can come from anywhere. Once qualified, the algorithm can be applied from ARM. Algorithm Lifecycle Use Case Diagram

54. Space Science and Engineering Center University of Wisconsin-Madison Space Science and Engineering Center University of Wisconsin-Madison 54 Algorithm entered into subversion Product created in bugzilla Algorithm is ported and wrapped Tests are created Algorithm Ingest

55. Space Science and Engineering Center University of Wisconsin-Madison Space Science and Engineering Center University of Wisconsin-Madison 55 Write a script to execute algorithm Script manages execution environment Algorithm name, inputs and outputs entered into ARM Algorithm Qualification

56. Space Science and Engineering Center University of Wisconsin-Madison Space Science and Engineering Center University of Wisconsin-Madison 56 APSPS executes bash script Script is passed arguments Binary Directory Working / Output directory Static Ancillary Directory Dynamic Ancillary Directory Inputs as specified by rule Outputs as specified by rule Binary is executed from line in script Results are stored by process that started script. Qualified Algorithm Structure

57. Space Science and Engineering Center University of Wisconsin-Madison Space Science and Engineering Center University of Wisconsin-Madison 57 Create slide content LEOCAT

58. Space Science and Engineering Center University of Wisconsin-Madison Space Science and Engineering Center University of Wisconsin-Madison 58 Create slide content LEOCAT Structure

59. Space Science and Engineering Center University of Wisconsin-Madison Space Science and Engineering Center University of Wisconsin-Madison 59 Org Chart Staffing Hardware Floor Plan Security Software Milestones Logistics

60. Space Science and Engineering Center University of Wisconsin-Madison Space Science and Engineering Center University of Wisconsin-Madison 60 Includes source Includes generated persistence code (JAXB) > 600 lines in rules Does not include axis generated web service code ~ 30000 lines ALG increase reflects subversion issue Increased leveraging of java and third party libraries resulted in code size decrease. Lines of code

61. Space Science and Engineering Center University of Wisconsin-Madison Space Science and Engineering Center University of Wisconsin-Madison 61 Not finished configuration control management plan - need to have backup systems Write goals APSPS accomplishments since PDR

62. Space Science and Engineering Center University of Wisconsin-Madison Space Science and Engineering Center University of Wisconsin-Madison 62 Includes source Includes generated persistence code (JAXB) > 600 lines in rules Does not include axis generated web service code ~ 30000 lines ALG increase reflects subversion issue Increased leveraging of java and third party libraries resulted in code size decrease. Prototype system reuse

63. Space Science and Engineering Center University of Wisconsin-Madison Space Science and Engineering Center University of Wisconsin-Madison 63 Create slide content Organizational Chart

64. Space Science and Engineering Center University of Wisconsin-Madison Space Science and Engineering Center University of Wisconsin-Madison 64 Create slide content Staffing

65. Space Science and Engineering Center University of Wisconsin-Madison Space Science and Engineering Center University of Wisconsin-Madison 65 Create slide content Hardware

66. Space Science and Engineering Center University of Wisconsin-Madison Space Science and Engineering Center University of Wisconsin-Madison 66 PEATE shares space with other projects including GOES archive Operators located in adjoining room Room for expansion PEATE Operations Floor Plan

67. Space Science and Engineering Center University of Wisconsin-Madison Space Science and Engineering Center University of Wisconsin-Madison 67 Create slide content Security

68. Space Science and Engineering Center University of Wisconsin-Madison Space Science and Engineering Center University of Wisconsin-Madison 68 Create slide content Build Plan

69. Space Science and Engineering Center University of Wisconsin-Madison Space Science and Engineering Center University of Wisconsin-Madison 69 Summary