GLAST LAT Project ISOC 1 ISOC Status Review June 3, 2004

GLAST LAT Project ISOC 2 Overview –We have developed an organization and staffing plan in concert with the SLAC management. ISOC buildup started, rapid ramp up over next year –We have completed initial work on an operations architecture. –We have made good progress in addressing peer review RFAs –Substantial work remains before CDR but we believe we now understand the scope and will be ready by 7/15.

GLAST LAT Project ISOC 3 LAT/ISOC Organization Post Launch NASA GLAST Project PI: P. Michelson Inst Sci: S.Ritz Instrument Ops Advisory Board H/W subsystem leads, key Technical Advisors from throughout collaboration ISOC Manager W. Craig (Acting) B Science Analysis Center 4.1.B.4 LAT Ops Facility (LOF) 4.1.B.1 Sci. Ops Group (SOG) 4.1.B.2 Sci. Analysis SW (SAS) R. Dubois 4.1.B.3 Software Operations Calib Optimization Flt S/w & Testbed Pipeline Config. Pipeline Anayl Tools Collab Computing Resources Only SSAC Science Analysis Coordination Committee SAC head, Analysis leads, ISOC rep, SAS head

GLAST LAT Project ISOC 4 Staffing Rob Cameron has accepted the ISOC manager position, so there will (finally) be a permanent ISOC manager in place in August. Craig will be responsible for a successful CDR and will keep Cameron updated throughout. –Several month transition period planned Steve Culp has accepted S/W developer position and will start within a week. He will be responsible for fleshing out the architecture and first database implementations.

GLAST LAT Project ISOC 5 Staffing Profiles Excludes SAS and SAC.

GLAST LAT Project ISOC 6 Staffing Profiles (with SAS/SAC) Does not include Stanford, UCSC, NRL, GSFC or collaboration members.

GLAST LAT Project ISOC 7 Architecture Drivers –Minimize V&V burden and total cost –Maintain all science capabilities –Simplify interfaces and allow early testing Recognized that neither of the previously considered options were particularly attractive –ITOS/Commercial packages don’t accommodate complexities of science data –Homegrown system doesn’t have heritage, not ready in time to make project timelines. Most of additional code needed duplicates that in existing packages –  Studied hybrid solutions

GLAST LAT Project ISOC 8 ITOS/Astro RT Trade In favor of either –Both AstroRT and ITOS would provide basic instrument health and safety functions Telemetry display EU conversion Limit checking and monitoring Trending Command and telemetry database access –Both products have learnable interfaces and scripting AstroRT uses LabView for display and Perl scripts for automation ITOS displays are reportedly easy to create, uses STOL for input

GLAST LAT Project ISOC 9 ITOS/Astro RT Trade Against either –Requires use of ITOS or Astro-RT specific interfaces and scripting –Both have ITAR issues –Limitations are not fully understood Believe limitations will not affect monitoring and trending of housekeeping data – only science and instrument diagnostics

GLAST LAT Project ISOC 10 ITOS/Astro RT Trade In favor of AstroRT –LAT is using AstroRT for LAT flight software testing Against AstroRT –Does not handle character strings – not sure if that’s an issue for us (it is with GBM) –Commercial product costing $$$ upfront and for support throughout program life –Probably unable to alter AstroRT code In favor of ITOS –MOC and GBM will be using ITOS –May be able to alter ITOS code or have changes made Against ITOS –None that don’t also exist for AstroRT

GLAST LAT Project ISOC 11 Proposed ISOC S/W Architecture ITOS Cmd DB SOH trending and display All State of Health requirements satisfied within ITOS MOC/GSSC LATTE Ops  LATOPS Data Cmd Science data/performance trending Register load generation Relational database interaction Pipeline/SAS interactions No req’ts on MOC that require LATOPS layer

GLAST LAT Project ISOC 12 RFA responses #RFA SummaryRequestorActioneeComment 1a. Need ISOC Management Plan & Approach b. ISOC Documentation Set R. SchweissW. CraigPlan draft and list of ISOC documents on glast.slac.stanford.edu/ioc/ 2Need overall functional block diagram illustrating the functional capabilities and data flow during various phases R. SchweissL. BatorDraft response – slides attached 3Risk AnalysisR. SchweissW. CraigDraft response – slides attached 4Reschedule ISOC CDRM. Rackley C. Young D. LungDone. CDR scheduled for 8/4/04 5Incomplete Level III requirements for LOF and SOGM. RackleyL. BatorDrafts on glast.slac.stanford.edu/ioc/ 6Staffing plan and profileM. Rackley C. Young W. Craig D. Lung Staffing plan and profile presented, RFA response pending 7Define the ISOC reports for internal use and external use M. RackleyL. BatorResponse complete – slides attached 8The ISOC does not yet know what system it is using to process Observatory HSK data or perform the commanding M. RackleyL. BatorArchitecture presented, RFA response pending 9Describe lesson learned & approachM. RackleyW. CraigResponse complete – slides attached

GLAST LAT Project ISOC 13 RFA responses, cont’d #RFA SummaryRequestorActioneeComment 10ISOC verification does not involve early opportunities to validate/test using LAT instrument M. Rackley N. Johnson L. BatorSee RFA #2, also pending architecture approval 11Verify LAT modesM. DavisL. BatorDraft response – slides attached 12Understand the number of writes to EEPROMC. YoungL. BatorResponse submitted 13ISOC detailed development scheduleK. LehtonenD. LungPending architecture approval 14Enter a more formal agreement with SLAC management on required data storage and processing requirements N. JohnsonW. CraigResponse completed – slides attached 15ISOC organization structure & communicationsN. JohnsonW. Craig D. Lung Organization presented, RFA response pending 16Define mechanism for ISOC requirements being placed on I&T and SAS N. JohnsonW. CraigPending architecture approval 17Define LOF/SOG toolsR. CorbetL. Bator J. Panetta Draft response – slides attached 18Specify plans and requirements for automation of Ops software R. Corbet M. Rackley L. Bator J. Panetta Draft response for 1 st part, awaiting S. Culp for 2 nd 19Specify plans and requirements for Ops SW to be of sufficient robustness R. CorbetL. Bator J. Panetta ECD 6/15/04 – S. Culp 20Specify what other ground system elements will be involved in LAT operations R. CorbetL. Bator D. Lung ECD 7/5/04 - working group on contingency plans

GLAST LAT Project ISOC 14 RFA 2 – ISOC Functional Block Diagram RFA 2 Specific Request: –Need an overall functional block diagram illustrating the functional capabilities and a data flow diagram showing the various data flows, with the differences among the I&T (pre- launch w/GSE) phase, L&EO phase, and nominal on-orbit phase configurations specified –Diagrams for each phase might be needed

GLAST LAT Project ISOC 15 ISOC Dataflow During I&T Single Tower Testing Obtain data during I&T EM2 testing Goal is to read houskeeping data off flat file produced by Online Database development and maintenance is shared between I&T and ISOC

GLAST LAT Project ISOC 16 ISOC Dataflow During I&T Multi-Tower Testing Obtain data during I&T testing Increase in ISOC functionality

GLAST LAT Project ISOC 17 ISOC Dataflow with TestBed - Direct to SIU Direct interface with SIU for CCSDS command and telemetry packets Obtain testbed simulated data via SIU Demonstration of ISOC capability increases as functionality is developed

GLAST LAT Project ISOC 18 ISOC Dataflow with TestBed - With SIIS Interface with SIIS/AstroRT for telemetry packets and commanding Obtain testbed simulated data via SIU and SIIS Demonstration of ISOC capability increases as functionality is developed

GLAST LAT Project ISOC 19 ISOC Dataflow During GRTs, L&EO and On-orbit Shows full ISOC capability for L&EO and On-orbit GRTs will test capabilities as they are available

GLAST LAT Project ISOC 20 RFA 3 - ISOC Risk Analysis Process –Discussion with I&T personnel on risks –Internal discussion performed in concert with RFA’s from peer review –Review and approval by ISOC stakeholders Follow-up –Entry into LAT risk management database by 06/01/04 –Weekly tracking, updating by ISOC management

GLAST LAT Project ISOC 21 RFA 3 – ISOC Risk Analysis NumberDateRankOriginatorDescriptionMitigation ISOC-00015/15/041B. CraigISOC lacks accepted architecture and plan for software implementation. Trade study between possible front ends to be completed by 6/15/04. Hires into s/w architecture position. ISOC-00025/15/043B. CraigNo response to PDR RFAs Schedule and track RFA’s weekly. ISOC-00035/17/042B. CraigInadequate staffing plan for ISOC. Draft staffing plan in progress, to be released by 06/01 First req issued an offer out to highest priority position. ISOC-00045/21/044B. CraigNo facility location identified for ISOC Long-term solution identified, short term space to be requested from SLAC management.

GLAST LAT Project ISOC 22 RFA 3 – ISOC Risk Analysis NumberDateRankOriginatorDescriptionMitigation ISOC-00055/21/042B. CraigNo requirements levied on I&T and Flt S/W subsystems Mechanism in place with I&T, pending with Flt S/W. Implement these only after architecture is defined and accepted. ISOC-00065/21/041B. CraigISOC will be unable to hold schedule due to staffing delays and unscoped work Definition of work plan follows architecture development. If needed additional support will be requested from LAT management.

GLAST LAT Project ISOC 23 RFA 7 – ISOC Reports Specific Request Define and document the types of reports that will be generated by the ISOC for both internal use and for use by external systems (like the MOC and GSSC) Response Reports will be documented in the Operations Product ICD (external reports) and LAT Ops Plan (internal-only reports)

GLAST LAT Project ISOC 24 RFA 7 – ISOC Reports LAT status and planning Reported daily (TBR) Summary of LAT health status Limit violations Alerts received Current LAT configuration Commanding and any other special activities that occurred Mission planning outlook for near term (time period TBD) Generated by LOF with automatic and manual inputs Published to web server

GLAST LAT Project ISOC 25 RFA 7 – ISOC Reports LAT performance Reported daily (TBR) Quick look science data Performance metrics (details TBD) Generated by SOG Published to web server Level 0 data transmission report Data transmission metrics (details TBD) Automatically generated and sent to MOC following receipt of Level 0 data

GLAST LAT Project ISOC 26 RFA 7 – ISOC Reports Data Trending Housekeeping data Environmental data (temp, voltages, currents) Derived science quantities Trigger efficiency Total count rate Bright source monitoring Includes statistical analysis Generated automatically daily/weekly/monthly Published to the web

GLAST LAT Project ISOC 27 RFA 9 - ISOC Lessons Learned Issue –No writeup on lessons learned from visits to other instrument/mission operations center Resolution –Members of the ad hoc planning group for the definition of the LAT IOC (now ISOC) made visits to the operations centers for GP- B (launched April, 2004; Stanford Univ., Tom Langenstein & Brett Stroozas), RHESSI (launched 2002; Berkeley Space Sciences Lab., David Smith & Manfred Bester), and Chandra (launched in 1999; MIT & Harvard-Smithsonian Center for Astrophysics, Dan Schwartz & Paul Plucinsky) –Each of these operations centers integrates mission operations with science (instrument) operations, and so they are not directly comparable to the ISOC in terms of complexity or staffing. (The operations center for RHESSI includes the ground station.) LAT ISOC can learn from others but there are no direct models.

GLAST LAT Project ISOC 28 RFA 9 - Lessons Learned –The science operations center for GP-B is co-located with the science team at Stanford. The GP-B data also will be distributed widely to collaborating institutions, but the co- location at Stanford was deliberate to maximize the interaction with the SOC on data issues. Colocation important to maximize science. –The staffing for RHESSI operations is especially spare. The facility itself is also used to run operations for FAST and CHIPS and the routine operations, like scheduling of contacts and pipeline processing, are automated. Testbeds (simulators for the instrument computers) are maintained, and have been found vital for understanding anomalies as well as for testing flight software updates. Testbeds important for flight software updates.

GLAST LAT Project ISOC 29 RFA 9 - Lessons Learned –The Chandra Operations Control Center has a room with about 4 consoles for the ACIS instrument team to monitor and command the instruments. The ACIS team has developed an impressive, flexible facility for trend analysis. The importance of a flexible system that does not require deciding in advance what needs to be monitored routinely was stressed to us. The ground-based calibration data are still actively used, >4 years into the mission. Colocation of the operations (mission and instrument) and the ACIS instrument team has been important, at least in terms of increased efficiency. Instrument team members (like the PI) at Penn State can feel out of the loop or behind the times. Colocation important to keep all science members in the loop.

GLAST LAT Project ISOC 30 RFA 11 – LAT Modes Specific Request: –LAT Operations Team and Spectrum Astro should work together to verify if any interactions between LAT modes and spacecraft modes need to occur. For example, if a LAT mode change requires the spacecraft to change spacecraft mode and/or configuration Response: –SC modes are understood and accommodate the LAT modes as designed

GLAST LAT Project ISOC 31 RFA 11 – LAT Modes, cont’d Mission ModesSC ModeLAT Mode Launch  S-Band rcvr/xmit  On battery power Off Early Orbit  Inertial capture  S-Band rcvr/xmit  Sun point with solar arrays tracking Survival Engineering  Inertial point, zenith point, or maneuver  Ku-Band xmit, S-Band rcvr/xmit  Solar arrays tracking Engineering Calibration SAA Sky Survey  Zenith point  Ku-Band xmit, S-Band rcvr/xmit  Solar arrays tracking Science Mode Pointed and Repointed  Inertial point, maneuver  Ku-Band xmit, S-Band rcvr/xmit  Solar arrays tracking Science Mode Safemode  Inertial capture, sun point  S-Band rcvr/xmit  Solar arrays fixed Hardware Survival Re-Entry  Cruise, delta-V  S-Band rcvr/xmit  Solar arrays tracking Off

GLAST LAT Project ISOC 32 LAT Modes, cont’d

GLAST LAT Project ISOC 33 RFA-12: Number of EEPROM Writes Specific Request –Understand the number of writes to EEPROM on LAT from all sources Reason –EEPROMs have a limited number of write cycles before they become unreliable Response –Not an issue due to use of TrueFlash File System overlay (full description is on RFA response, available on ISOC web page)

GLAST LAT Project ISOC 34 RFA 14 - ISOC Data Storage Issue –No agreement with SLAC management on how data storage and processing requirements will be funded. Resolution –Estimate of processing and data storage requirements performed for SAS by R. DuBois. Cost determined and built into ISOC outyear funding plan and accepted by SLAC Director of Research –Database costs still being evaluated by database working group but now expected to be minimal or covered completely by SLAC central computing services due to small size (~ 1Tb) of database.

GLAST LAT Project ISOC 35 RFA 14 - Monthly Costs

GLAST LAT Project ISOC 36 RFA 17 – Define LOF/SOG Tools Specific Request: –The tools needed to run the LOF/SOG need to be specified Which HK and science parameters will be monitored and in what way? What actions would be taken based on the results seen with these tools? –How does the ISOC team know from a design perspective that the collection of the described I&T tools will function in the operations environment as an integrated system? Reason/Comment: –The overall requirements on the ISOC have been given –Detailed plans for which software components/libraries such as Python will be used were given –However, lists of which software tools are required to achieve the ISOC’s requirements are needed

GLAST LAT Project ISOC 37 RFA 17 - Response Which HK and science parameters will be monitored and in what way? –HK parameters are defined in LAT-TD –Routinely monitored science parameters are included within the HK data as Low Rate Science Use of high rate science data is being developed by SVAC and will be further developed by SOG –Limits and use of HK data for monitoring are TBD What actions would be taken based on the results seen with these tools? –Calibration activities are in development in the SVAC –Contingency actions are TBD

GLAST LAT Project ISOC 38 RFA 17 - Response, cont’d How does the ISOC team know from a design perspective that the collection of the described I&T tools will function in the operations environment as an integrated system? –Development and testing of ISOC tools is in conjunction with I&T Lists of which software tools are required to achieve the ISOC’s requirements are needed –The following slides detail the ISOC software tools

GLAST LAT Project ISOC 39 RFA 17 - ISOC Software Tools ISOC Tools OPUS ITOS Exists with LATTE Existing Other To be Written 1Data Transport and Management 1.1File retrieval, transmission, and management (internet)XSLAC SCS and Fastcopy 1.2Archive data filesXSLAC SCS 1.3Parse data into databaseX 1.3.1Convert housekeeping data into Engineering UnitsX 1.4Data integrity checksX 1.5Science data reconstructionX 1.6Calibration trackingX

GLAST LAT Project ISOC 40 RFA 17 - ISOC Software Tools, cont’d ISOC Tools OPUS ITOS Exists with LATTE Existing Other To be Written 2Operations Tools 2.1Electronic logbookX 2.1.1ReportingXX 2.1.2Command historyXX 2.2Database management 2.2.1Command and telemetryX 2.2.2Science and calibrationX 2.3Archive managementSLAC SCS

GLAST LAT Project ISOC 41 RFA 17 - ISOC Software Tools, cont’d ISOC Tools OPUS ITOS Exists with LATTE Existing Other To be Written 3Instrument Health (LOF) 3.1Real time housekeeping telemetry displayX 3.2Historical data trending displayXX 3.3Data monitoring and alarming systemsXX 3.3.1Autonomous reportingX 4Instrument Diagnostic Tools 4.1Diagnostic data display and analysisX 4.2Memory dump parsingFSW 4.3Testbed management and operationElec

GLAST LAT Project ISOC 42 RFA 17 - ISOC Software Tools, cont’d ISOC Tools OPUS ITOS Exists with LATTE Existing Other To be Written 5Instrument Performance (SOG) 5.1Visualization toolsXX 5.2Offline calibrationSVAC tools X 5.3Online calibrationX

GLAST LAT Project ISOC 43 RFA 17 - ISOC Software Tools, cont’d ISOC Tools OPUS ITOS Exists with LATTE Existing Other To be Written 6LAT Commanding Tools 6.1Command procedure generation and managementX 6.1.1Instrument file generationFSW File managementX File validation and verificationX File translation to ITOSPerl script 6.1.2Telecommand generationXX 6.2Procedure verification and validation on testbedX 6.3Procedure transmission toolsX 6.3.1Command wrapper generation (for GSSC)X 6.3.2Command load transfer to GSSCFastcopy

GLAST LAT Project ISOC 44 RFA-18: ISOC Operations automation Specific Request –Specify plans and requirements for automation of operations software –Describe the software design for how the automation needs will be met Response –Draft of the plans and requirements has been completed –Software design will commence when ISOC software engineer is hired

GLAST LAT Project ISOC 45 RFA-18: ISOC Operations automation Data retrieval from MOC OPUS: –Archiving raw data –Dispatch science data to SOG –Dispatch housekeeping to LOF LOF automated processing –Housekeeping: limit checks, warnings –Science data: raw data quality –Automated reporting of above (web/paging/ ) Trending: –Weekly/monthly characterization of data Calibration tracking & computation External agency alert retrieval (i.e., SEC, NIST)

GLAST LAT Project ISOC 46 Roadmap to CDR Primary tasks –1) Scenario definition Work with FSW and I&T for all operational modes (BC, LB, SC) July 1 Detailed early orbit plans (BC,LB) July 15 –2) Contingency operations analysis Define possible actions by subsystem (BC,LB) July 7 –3) Draft Instrument Ops Section of Mission Plan (LB, SC, who at GSFC?) July 15 –4) Update requirements documents to reflect architecture (SC, LB) July 15

GLAST LAT Project ISOC 47 CDR Prep Schedule July 8 th Revisit roadmap July 21 st Laydown July 26 th Slides to GSFC July 29 th Dry Run August 4 th ISOC Peer Level CDR August 18 th CDR