1. 1 of XX February 8-10, 2010 Section 09 – Aquarius Command and Control System Mission Operations Review Aquarius Instrument Operations Aquarius/SAC-D Mission Operations Review February 8 – 10, 2010

2. 2 of XX February 8-10, 2010 Section 09 – Aquarius Command and Control System Mission Operations Review Agenda Peer Review Results Key Requirements Processes/Operations Overview Aquarius Command and Control System Architecture Testing Status Procedures/Status/Plans Risks Risk Mitigation Documentation/Status/Plans for completion

3. 3 of XX February 8-10, 2010 Section 09 – Aquarius Command and Control System Mission Operations Review PEER REVIEW RESULTS Aquarius Instrument Operations

4. 4 of XX February 8-10, 2010 Section 09 – Aquarius Command and Control System Mission Operations Review Recommended Actions  Clearly define the final set of instrument operations procedures  Identify procedures to be written  Identify what procedures must be verified on the Observatory before flight  The project has been addressing these since the Peer Review.  Concern remains that, due to project demands, personnel may not be available to rehearse critical procedures

5. 5 of XX February 8-10, 2010 Section 09 – Aquarius Command and Control System Mission Operations Review KEY REQUIREMENTS Aquarius Instrument Operations

6. 6 of XX February 8-10, 2010 Section 09 – Aquarius Command and Control System Mission Operations Review Key Requirements  Mechanics of the Ground System Design  Use of the Oceans Biology System  ACCS relies on its facilities, networks, security, and software support  Use of CONAE for Mission Operations  The ACCS Scheduler Client and Telemetry Viewer are CONAE deliverables  The AQ Test-bed also interfaces with SInter, the CONAE commanding tool.  ACCS processes and procedures must align with CONAE’s  Instrument Operations - Maximizing Quality and Quantity of Science Data Collection  Science data loss budget of 8%  Science data accuracy to 0.2 PSU  Perform Aquarius command planning within flight rule constraints  Monitor Aquarius science and telemetry to determine state of health  Perform Aquarius anomaly handling

7. 7 of XX February 8-10, 2010 Section 09 – Aquarius Command and Control System Mission Operations Review Key Requirements Flow-down Summary Instrument Operations Responsibilities DO include: Aquarius command planning Aquarius offline state-of-health analysis Aquarius anomaly handling Instrument Operations Responsibilities do NOT include: Real-time commanding Real-time telemetry monitoring Aquarius safing SAC-D anomaly handling All real-time tasks and all SAC-D tasks are the responsibility of the SAC-D Flight Operations Team As captured in the documents:  Aquarius Ground System Level-3 Requirements (AQ-326-0152)  L4 AQ Command and Control System & Test Requirements (AQ-426-0197)

8. 8 of XX February 8-10, 2010 Section 09 – Aquarius Command and Control System Mission Operations Review PROCESSES Aquarius Operations

9. 9 of XX February 8-10, 2010 Section 09 – Aquarius Command and Control System Mission Operations Review Aquarius Instrument Operations Processes The Aquarius Command and Control System (ACCS) supports:  Aquarius command planning  Standard: daily science data downloads; monthly Cold Sky Calibrations  Special: science requests such as Look Up Table changes  Emergency: anomaly response  Aquarius offline state-of-health analysis  Receipt of Aquarius/SAC-D science and housekeeping data from the ADPS.  Receipt of Aquarius/SAC-D pass prediction files from the ADPS.  Processing of Aquarius/SAC-D housekeeping and Aquarius science data for the purpose of monitoring instrument trends and state-of-health.  Publishing of telemetry trends and alarms to Web-based tools to enable off- site health checks.  Aquarius anomaly handling  Aquarius instrument anomaly tracking and resolution.

10. 10 of XX February 8-10, 2010 Section 09 – Aquarius Command and Control System Mission Operations Review Command Planning Overview  Two standard command planning activities:  Perform science data downloads at least once every 12 hours  Plan 2 weeks of downloads every week  CONAE has the flexibility to send a subset of these every few days  CONAE performs C&DH management  Perform a Cold Sky Calibration  No Aquarius commands; purely a request for the S/P state of “Cold Sky Calibration”  Science may request other command planning activities such as:  Thermal set-point change  Adjust the radiometer receive window

11. 11 of XX February 8-10, 2010 Section 09 – Aquarius Command and Control System Mission Operations Review Command Planning Steps  Daily: Check for science requests  Include any science requests in the upcoming command plan  Create a stand-alone command plan if urgent  Weekly: Generate the command plan  Two weeks of downloads  Command plan upload to the S/P is base-lined as being two weeks in the future  Command plan execution by the S/P is base-lined as being 2 weeks + 3 days in the future  Any science requests  Enter command plan into the Scheduler Client  A command plan is an “Action Request” to the Scheduler Client.  An “Action Request” is composed of “Basic Action Components”, “Massive Upload Components” and “Extended Condition Components”.  Example Basic Action Component: start downlink/stop downlink command pair  Example Massive Upload: Aquarius Flight Software Patch  Example Extended Condition Component: X-band transmitter on/off command pair  Scheduler Client is a CONAE deliverable tool  Interfaces with the CONAE Spacecraft Operations Planning tool (SOP)  CONAE SOP provides progress reports on Action Request planning via the Scheduler Client:  Action Request Received by the SOP  Action Request Accepted by the SOP  ACCS verifies reports, then verifies execution via telemetry/downlink file reception

12. 12 of XX February 8-10, 2010 Section 09 – Aquarius Command and Control System Mission Operations Review Standard Command Planning Process at a Glance WTFSSMTWTFSSMTWTFSSMTWT 1234567891011121314151617181920212223 Aquarius Commands Planned at the ACCS Aquarius Action Requests Submitted to the SOP ACCS Receives Action Report: Action Request Received ACCS Receives Action Report: Action Request Accepted

13. 13 of XX February 8-10, 2010 Section 09 – Aquarius Command and Control System Mission Operations Review Command Planning - Emergency The emergency command planning process is defined as follows:  Utilize emergency channels to notify the CONAE MOC of an Aquarius anomaly:  Phone  Email  And also enter a ticket into the:  Aquarius Reporting and Tracking Tool  Generate the (contingency) Aquarius command plan as per standard procedure, but with accelerated implementation:  CONAE will schedule NEN support  CONAE will process received Aquarius command plans as quickly as possible

14. 14 of XX February 8-10, 2010 Section 09 – Aquarius Command and Control System Mission Operations Review Daily Monitoring & Trending Process  Alarm and Error Notification Tool:  Verify expected download files have been received  Download file times match commanded download times  File sizes match expected file sizes  No Cyclic Redundancy Check errors have occurred  Note: Data integrity (ICDS CRC and contiguous blocks) checked automatically  Verify No Alarms present  If Alarms present, perform Anomaly Resolution process  Web-based graphs  Examine science and telemetry trends on various scales:  Daily  Monthly, Seasonal  Yearly, Mission  Telemetry Viewer  Verify Alarms in agreement with Analysis Tool  Verify instrument configuration is as expected  Review data for software messages

15. 15 of XX February 8-10, 2010 Section 09 – Aquarius Command and Control System Mission Operations Review Anomaly Response The SAC-D and Aquarius teams generated a common anomaly response flow. Aquarius Operations has designed an anomaly process aligned with that flow.

16. 16 of XX February 8-10, 2010 Section 09 – Aquarius Command and Control System Mission Operations Review AS Anomaly Process

17. 17 of XX February 8-10, 2010 Section 09 – Aquarius Command and Control System Mission Operations Review I. Discovery a. CONAE FOT or AQ OPS detects telemetry Out of Limits Team has procedures and authority to SAFE Aquarius b. AQ OPS detects abnormal science or telemetry trend c. Science detects abnormal science performance II. Notification a. If detected by CONAE or Science, party notifies AQ OPS via email/phone b. AQ OPS notifies AQ GS Manager, CONAE Manager, AQ PI by submitting a TRAC anomaly ticket III. Gather Investigation Team a. AQ OPS asks AQ GS Manager for additional support b. AQ OPS updates TRAC with criticality, preliminary investigation report; updates TRAC status to “team formation”. (Publishes TRAC ticket and report to AQ/Science/CONAE management) c. AQ GS Manager asks the AQ PI to authorize funds for JPL support d. AQ PI asks JPL science liaison to recruit resolution team e. JPL science liaison recruits team, notifies AQ PI, AQ GS Mgr, AQ OPS of team members f. AQ OPS updates TRAC with team member list; updates TRAC status to “in work”. (Publishes TRAC ticket and report to entire team). Aquarius Specific Anomaly Resolution Process

18. 18 of XX February 8-10, 2010 Section 09 – Aquarius Command and Control System Mission Operations Review IV. Develop Solution a. Technical team uses TRAC to confer, exchange data, develop solution and verification criteria, exercise with Test-bed. b. When ready, technical team buy-in confirmed by setting TRAC status to “solution proposed”. V. Obtain Approvals a. Upon team acceptance, technical team leader updates TRAC status to “solution accepted, technical”. b. CONAE exercises procedure against Observatory EM; forwards accept/reject to CONAE Mgmt c. If “reject”, return to step V. d. Management team (AQ GS/AQ PI/CONAE) reviews solution; upon acceptance, AQ PI updates TRAC status to “solution accepted, management”. VI. Execute and Verify a. AQ OPS plans commands according to procedure b. CONAE FOT executes c. Full team verifies per step V criteria. VII. Report AQ OPS updates TRAC with Final Report and Resolution status: completed/reissued. (If reissued, go to step V). Aquarius operations documents and procedures updated accordingly. Aquarius Specific Anomaly Resolution Process

19. 19 of XX February 8-10, 2010 Section 09 – Aquarius Command and Control System Mission Operations Review ARCHITECTURE Aquarius Command and Control System

20. 20 of XX February 8-10, 2010 Section 09 – Aquarius Command and Control System Mission Operations Review Aquarius Ground System The Aquarius Ground System is contained within the Oceans Biology Data System  The Aquarius Ground System is made up of 2 segments:  The Aquarius Data Processing System (ADPS)  The Aquarius Command and Control System (ACCS)  The Aquarius Data Processing System receives Aquarius science data, SAC-D auxiliary data, and other science ancillary data to generate Aquarius science products.  The Aquarius Command and Control System receives Aquarius science data, SAC-D and Aquarius House Keeping Telemetry, and pass prediction files in order to monitor Aquarius health and status, plan commands, and respond to anomalies.

21. 21 of XX February 8-10, 2010 Section 09 – Aquarius Command and Control System Mission Operations Review Oceans Biology Data System Support to the Aquarius Command and Control System Functions/services provided:  Facilities  Infrastructure  Staff  IT network and security  Inherent redundancy (except CPT; hot backup)  Use of automation  Configuration Management  Software maintenance  Data storage  Use of Oceans Biology Data System interface with CONAE

22. 22 of XX February 8-10, 2010 Section 09 – Aquarius Command and Control System Mission Operations Review Aquarius Command and Control tools and interfaces ACCS Architecture

23. 23 of XX February 8-10, 2010 Section 09 – Aquarius Command and Control System Mission Operations Review TESTING Plans and Status of

24. 24 of XX February 8-10, 2010 Section 09 – Aquarius Command and Control System Mission Operations Review ACCS and Aquarius Instrument Operations Test List  ACCS Ground System Testing  Software Tools  Component  Internal Interfaces  Functional  External Interfaces  Command and Telemetry Testing  Command Database  Telemetry Database  Cross-Tool Verification  Mission Scenarios

25. 25 of XX February 8-10, 2010 Section 09 – Aquarius Command and Control System Mission Operations Review Testing Status  COMPLETED TESTS  Software Tools  Component – baseline completed January 11, 2010  Internal Interfaces – baseline completed January 14, 2010  Functional – baseline completed January 14, 2010  Command and Telemetry Testing  Command Database – completed June 15, 2009  Telemetry Database – completed in instrument I&T at JPL  UPCOMING TESTS  Cross-Tool Verification – test design meeting scheduled February 15, 2010  External Interfaces – will participate in CONAE led tests  Mission Scenarios – under development by the SAC-D and Aquarius Mission System Engineers.

26. 26 of XX February 8-10, 2010 Section 09 – Aquarius Command and Control System Mission Operations Review  Analysis Tool  Alarm and Error Notification Tool  Command Planning Tool  Web-based graphs  Reporting and Tracking Tool  JPL performed the Test-Bed functional test Component Testing included the: ACCS Components Tested

27. 27 of XX February 8-10, 2010 Section 09 – Aquarius Command and Control System Mission Operations Review Interface testing included the:  L0 file delivery  Level 1A reports  Alarm Notifications  HKT file delivery  Text files to Web graphs  Web graph publication  Download time file delivery ACCS Interfaces Tested

28. 28 of XX February 8-10, 2010 Section 09 – Aquarius Command and Control System Mission Operations Review ACCS Functional Test Description  ACCS Functional Test – encompasses component and internal interface tests  100% of Level 4 Requirements verified  Data Monitoring Test:  Correlated science and HKT data files from I&T processed through the full ACCS system  Included a DPU memory download  Command Planning Test  All Aquarius commands and known sequences planned  Each singular command  Downlinks  LUT programming  DPU memory download  Flight software patch  Introduced illegal times, sequences, parameters  Anomaly Handling Test  Simulated use of Recording and Tracking tool in anomaly resolution process  Included file exchanges, milestones, notifications  Simulated science and instrument team external access

29. 29 of XX February 8-10, 2010 Section 09 – Aquarius Command and Control System Mission Operations Review Command and Telemetry Database Tests  Command and Telemetry Database Testing  Command Database  Worked with JPL to verify hex output of CONAE SInter tool matched output of I&T commanding tool  Exercised all legal commands and wide range of legal and illegal command parameters  Telemetry Database – completed in instrument I&T at JPL  The telemetry database is the telemetry dictionary as captured in Telemetry Viewer scripts.  Scripts define how:  Telemetry bytes are extracted from telemetry frames  Hex values are converted to Engineering Units  Alarms are applied  Telemetry Viewer pages display hex, EU values and alarms  Ultimately compared to values from trusted sources:  Power meters  Calibrated sensors  Software tools

30. 30 of XX February 8-10, 2010 Section 09 – Aquarius Command and Control System Mission Operations Review AQ/SAC-D GS External Interface and Verification Test Lists  SDGS-GSD-I-0042-a Science data delivery  SDGS-GSD-I-0044-a ST TM delivery  SDGS-GSD-I-0024-a AR Reception and Confirmation Integration Test  SDGS-GSD-I-0025-a Action Reports Request and Response Integration Test  SDGS-GSD-I-0028-a Voice Communication Integration Test  SDGS-GSD-I-0026-a Orbit Products delivery  SDGS-GSD-V-0002 X band data publication  SDGS-GSD-V-0004 MOC ancillary data production and delivery  SDGS-GSD-V-0006Pass plan generation  SDGS-GSD-V-0008Script validation and compilation  SDGS-GSD-V-0012Execution report generation  SDGS-GSD-V-0011Flight Software replacement These external interface and verification tests are CONAE-led and will verify Aquarius Ground System Level 3 Requirements:

31. 31 of XX February 8-10, 2010 Section 09 – Aquarius Command and Control System Mission Operations Review PROCEDURES Status of

32. 32 of XX February 8-10, 2010 Section 09 – Aquarius Command and Control System Mission Operations Review Key Aquarius Operations Procedures The key procedures for Aquarius instrument operations are: Aquarius Command Planning Aquarius Monitoring and Trending Aquarius Anomaly Response ACCS Configuration Management And the individual standard and contingency instrument procedures.

33. 33 of XX February 8-10, 2010 Section 09 – Aquarius Command and Control System Mission Operations Review ACCS Procedures: Status and Plans for Completion  Key ACCS procedures are in preliminary form.  Extracted from the ACCS Functional Test Procedure.  Exception: Configuration Management Procedure  This procedure provides the steps for performing configuration management  It also identifies what items are to be maintained under CM:  Processes, Procedures  Tool software (including CONAE deliverables)  Flight Software/patches  Command and Telemetry scripts (databases)  Tests and test reports  ACCS Procedures must be updated in lockstep with the Functional Test. This is expected with:  Expansion of Functional Test to encompass Joint Tests  Corrections to Functional Test as Joint and Mission Tests progress

34. 34 of XX February 8-10, 2010 Section 09 – Aquarius Command and Control System Mission Operations Review Aquarius Instrument Procedures, Standard and Contingency Instrument standard and contingency procedures identified to date (except deployment): Standard Procedures  AQ Start downlink  AQ Stop downlink Contingency Procedures  Temperature Stability Failure  ATC to Constant Power  ATC PI parameter change  ATC Reset  ATC Set-point change  ICDS Fails to Accept Commands  ICDS Reset Ground Initiated  No AQ Science Data Down-linked  FIFO errors  Aquarius Power-down  Aquarius Power-up  Recovery from ICDS reset  Patching  Replacement Heaters  Turn Scatterometer Off  Scatterometer Exciter Out of Limits  Scatterometer Beam Mode Select  Scatterometer Receive Only  Scatterometer Receive Window Adjust  Scatterometer reset Chirp Generator  Scatterometer Select Redundant CG  Scatterometer Attenuator Settings Select  Scatterometer Transmit Mode  Scatterometer RFI Flag Threshold Adjust  Scatterometer science anomaly  Radiometer Custom LUT upload  Radiometer Integration Window Adjust  Radiometer Memory Dump  Radiometer Select Operational LUT  Radiometer reset (soft, hard)  Radiometer Science Anomaly

35. 35 of XX February 8-10, 2010 Section 09 – Aquarius Command and Control System Mission Operations Review  Individual Aquarius procedures:  Have been drafted by the instrument team as flowcharts (where needed) with command sequences and telemetry responses identified. AND/OR  Exist as ATLO scripts/procedures  Work to be done:  The Aquarius team should come to consensus on which procedures are operations-critical.  The SAC-D Flight Operations Team and the Aquarius Instrument Ops Team should rehearse/observe these critical procedures. Instrument Operations Procedures Status

36. 36 of XX February 8-10, 2010 Section 09 – Aquarius Command and Control System Mission Operations Review  Aquarius flows and ATLO scripts will be transferred to operations procedures  The transfer steps from ATLO scripts to operations procedures are:  Correlate specific ATLO/I&T scripts to each operations procedure  Parse the scripts by decision point. ACCS procedures direct the use of the CONAE Aquarius procedures. Thus:  ACCS procedures include the decisions within a given ATLO script  CONAE procedures include the steps between decisions  Write the ACCS procedures and the CONAE procedures (scripts) according to the parsed ATLO/I&T scripts  Rehearse/observe the ACCS and CONAE Aquarius procedures against the Observatory. AND/OR  Rehearse the ACCS and CONAE procedures in mission scenarios. Generating Instrument Operations Procedures

37. 37 of XX February 8-10, 2010 Section 09 – Aquarius Command and Control System Mission Operations Review ACCS vs CONAE Procedures 37 Two types of instrument procedures:  ACCS  CONAE Flight Operations  ACCS procedures are documents in step-by-step format  All decision points occur within the ACCS procedures  CONAE procedures are scripts  CONAE Aquarius scripts are written in the Sinter scripting language  The scripts consist of:  Aquarius commands established in known sequences  Associated telemetry responses  Any other real-time activities required  Examples:  Inserted delays between commands  Special data logging  Information to note before proceeding

38. 38 of XX February 8-10, 2010 Section 09 – Aquarius Command and Control System Mission Operations Review RISKS/CONCERNS

39. 39 of XX February 8-10, 2010 Section 09 – Aquarius Command and Control System Mission Operations Review Risk Assessment  After an extensive risk/concern discussion during the Aquarius Instrument Peer Review and follow-up discussions with the Aquarius team, two risks/concerns remain.  The two risks/concerns are listed by priority from the instrument operations perspective.

40. 40 of XX February 8-10, 2010 Section 09 – Aquarius Command and Control System Mission Operations Review Risk RISK: Personnel availability may be too limited to allow for implementing the existing command sequences as procedures and rehearsing them. IMPACT: The Operations Team would then be inadequately trained for Aquarius operations. Mitigation: Project is planning a workshop in Bariloche in the April timeframe to rehearse critical procedures.

41. 41 of XX February 8-10, 2010 Section 09 – Aquarius Command and Control System Mission Operations Review Concern CONCERN: Awaiting Scheduler Client delivery. IMPACT: Unable to develop adequate Aquarius command planning procedure in time for interface tests and scenarios. Interface testing of the command planning link could be delayed. Mitigation: Scheduler Client User’s Guide and ongoing conversations with developers have provided baseline information for ACCS tool and procedure development.

42. 42 of XX February 8-10, 2010 Section 09 – Aquarius Command and Control System Mission Operations Review DOCUMENTATION Description and Status of 42

43. 43 of XX February 8-10, 2010 Section 09 – Aquarius Command and Control System Mission Operations Review Documentation/Status NameStatusVersion Date Aquarius Ground System Level-3 Requirements ( AQ- 326-0152) Signed ACCS Level-4 Requirements (AQ-426-0197) SignedJanuary 19, 2010 Aquarius Command and Control System Software Management Plan ( AQ-316-0239) Ready for SignatureFebruary 25, 2009 Aquarius Ground System Integration Test & Training Plan ( AQ-336-0244) RevisionJanuary 18, 2010 SAC-D Ground System to Aquarius Ground System ICD ( AS-336-0151b) ReviewJuly 14, 2008 Aquarius Ground System to Instrument Team Anomaly ICD ( AQ-336-0522) Ready for SignatureDecember 7, 2009 Aquarius Science Team to Ground System ICD ( AQ-336-0243) Ready for SignatureDecember 7, 2009 Aquarius Flight Operations Plan (AQ-316-0157) Ready for SignatureDecember 3, 2009 ACCS Operations HandbookPreliminaryDecember 28, 2009 Aquarius Standard ProceduresPreliminaryJanuary 4, 2010 Aquarius Anomaly ProceduresDraft ACCS User’s GuidesFinalJanuary 6, 2010

44. 44 of XX February 8-10, 2010 Section 09 – Aquarius Command and Control System Mission Operations Review ACCS – CONAE Rec/Dels - Status Nº:TITLEACCS Rec/Del STATUS 1CAS SCL Reference Manual RecReceived 2CAS SCL User’s Guide RecReceived 3Aquarius Telemetry Descriptions DelPending Due: April 23, 2010 4Aquarius Command Descriptions DelPending Due: April 23, 2010 5SAC-D HKTM Dictionary (TBC) RecPending 6Aquarius Operating Procedures (TBC) DelPending Due: June 10, 2010 7Scheduler Client User Manual (TBC) RecReceived 8TM Viewer User Manual (TBC) RecReceived 9Aquarius Planning Restrictions (TBC) DelPending Due: April 23, 2010 10Aquarius TM Pages and Decoding scripts (TBC) DelSent 11 Sinter Command Tool RecReceived 12 Telemetry Viewer Tool RecReceived 13 Scheduler Client Tool RecReceived

45. 45 of XX February 8-10, 2010 Section 09 – Aquarius Command and Control System Mission Operations Review Instrument Team – ACCS Rec/Dels - Status Nº:TITLEACCS Rec/Del STATUS 14AQ Flight Software User’s Guide RecReceived 15 AQ Instrument Operations Handbook RecReceived 16 Aquarius Telemetry Dictionary RecReceived 17 Aquarius Command Dictionary RecReceived 18 Aquarius Test-Bed User’s Guide Rec Received 19 Aquarius Test-Bed Maintenance Manual Rec Received

46. 46 of XX February 8-10, 2010 Section 09 – Aquarius Command and Control System Mission Operations Review Summary The Aquarius instrument operations team and the ACCS tools are ready to move into the operations testing phase. The testing, process development and procedure development is working in conjunction with SAC-D ground system and Aquarius instrument testing.