LSST Control Software Integration Francisco Delgado OCS Software Manager LSST Commissioning Plan Review January 24-26, 2017

Control Software Scope Integration Timeline Agenda Control Software Scope Integration Timeline Pathfinders during early integration Commissioning readiness Commissioning with ComCam Commissioning with LSSTCam

LSST Control Architecture Goal: ensure successful integration of software products and robust interoperability Observatory Control System Francisco Delgado OCS Communications Middleware Tony Johnson Camera Control System Telescope Control System Data Management Control System Add names & location to boxes Image data Camera Data Acquisition Image data Paul Lotz Gregory Dubois-Felsmann Mike Huffer

Development strategies in preparation for commissioning Focus on interfaces between principal subsystems Define comprehensive set of interfaces represented by Interface Control Documents (ICDs) Provide common communication middleware for messaging Elaborate interfaces in terms of system components architecture Detail network infrastructure Ensure timely progress through Advancing level of detail in ICDs Frequent intersystem communications Milestones to mark and verify progress Early interface testing

Control Software System Level ICDs LSE-71 OCS <-> Camera Control System LSE-72 OCS <-> Data Management LSE-73 OCS <-> Telescope Control System Interface Support Documents LSE-70 OCS Communications Protocol LSE-74 OCS Middleware Communications LSE-209 OCS <-> Software Component

Interface Control Documents Observatory Control System LSE-70 LSE-74 LSE-209 OCS Communications Middleware LSE-71 LSE-72 LSE-73 Camera Control System Telescope Control System Data Management Control System Camera Data Acquisition Image data LSE-66 LSE-67 Image data LSE-68 LSE-69

Common communications middleware OCS Middleware Service Abstraction Layer (SAL) on top of Data Distribution Service (DDS) Object Management Group (OMG) maintains DDS standard LSE-70 documents requirements for SAL and clients, API, behaviors LSE-74 incorporates data dictionary Used by all principal subsystems

OCS Software Components OCS Monitor OCS Operator OCS Remote Control OCS Scheduler OCS EFD OCS Application OCS Sequencer History Telemetry Image Parameters Targets Visits Cmds OCS Communications Middleware Telescope Control System Camera Control System Data Management Control System

OCS+TCS+CCS+DAQ+DMCS Integration Timeline Early Integration Pathfinder Exercises OCS+TCS AT Integration & Test @Summit OCS+CCS+TCS+DAQ ComCam Integration & Test @Tucson 2016 July Middleware v3 EFD v3 Controls v0.1 2017 July Middleware v4 EFD v4 Controls v1 Scheduler v1 2018 May Controls v2 Scheduler v1.3 Commissioning Phase 0 2018 August AT @Summit DAQ @Tucson 2018 October ComCam @Tucson TCS CCS DAQ DMCS Control simulators TCS(AT) DMCS Control CCS(ComCam) Control

OCS+TCS+CCS+DAQ+DMCS Integration Timeline Control System Integration & Test ComCam @Summit Early Science Verification ComCam Scheduler 2019 December Monitor v1 Scheduler v2.0 Control System Integration & Test LSSTCam @SLAC Commissioning Phase 1 (ComCam) ComCam @Summit 2020 January ComCam @Telescope TCS CCS(ComCam) DMCS Control 2020 February LSSTCam @SLAC

OCS+TCS+CCS+DAQ+DMCS Integration Timeline Control System Integration & Test LSSTCam Early Science Verification LSSTCam Scheduler 2019 December Monitor v2 Scheduler v2.2 Commissioning Phase 2 (LSSTCam) TCS CCS DM Control LSSTCam @Summit 2020 November LSSTCam @Telescope

OCS+TCS+CCS+DAQ+DMCS Integration Timeline 2021 March Scheduler Mini-Surveys Scheduler evaluation Survey Monitor evolution Commissioning Phase 3 (Mini-surveys) Reference Zejlko

Frequent intersystem communications Communication forums Series of Principal System Interactions workshops Dedicated face-to-face meetings or teleconferences Breakouts at conferences Working group discussions, systems engineering meetings, change requests, intranet discussions Topics Refinement of ICDs Establishment of common IT requirements Integration planning Pathfinder exercise testing Base for recurring tests in Integration & Test environment

Workshop History The OCS-TCS-CCS-DAQ-DMCS + SE cross project team has been meeting regularly Meetings focus on working out details of interfaces Agendas, meeting minutes, actions, and agreements documented on confluence Dates Confluence Notes May 12-14, 2014 Link Nov 11, 2015 Nov 11-14, 2014 Dec 9, 2015 Feb 19-20, 2015 Mar 30, 2016 Apr 7, 2015 May 4, 2016 May 6-8, 2015 June 15, 2016 Jul 8-10, 2015 August 19, 2016 Oct 8, 2015 January 17, 2017

OCS-TCS-CCS-DAQ-DMCS early integration Activity Objectives Status OCS-CCS-DM Communication Test objective: ensure middleware communication works properly between OCS, DM and Camera Complete CCS basic simulator Test objective: ensure Camera commandable device works properly with OCS Populating EFD with Camera Telemetry Transfer simulated camera telemetry sent to the OCS and put into the EFD. In progress OCS-TCS-CCS-DAQ-DMCS Start/End of night test Exercise the interfaces the OCS uses to interface with devices. Test component state machine functionality. DMCS Demonstrates ability to compile and link against the v1 DAQ software To show that DM and Camera DAQ are using compatible toolsets. Test of EFD replication under DMCS device control Connect the DM translation engine to the EFD. Properly query the EFD and retrieve all the information from the EFD. Full OCS-TCS-CCS-DAQ-DMCS Test: Complete Visit Basic flow: Configure, enable, take one visit, disable, shut down. This requires all components to do their appropriate functions. Execute the single visit sequence diagram with simulations of data as needed. Basic flow: Configure, enable, take one visit, disable, shut down. Includes DAQ. Full OCS-TCS-CCS--DAQ-DMCS Test: Mini Night Include multiple visits, filter changes, cold starts. Use the OCS to coordinate getting all the systems to the right state. Take visits, give filter change command and make sure that DM knows which filter is in place. Multi visit - make sure DM doesn't confuse images. Full OCS-TCS-CCS-DAQ-DMCS Test: Daily Cycle, Including Calibration Simulation of a full 24 hour cycle.

Exercises specified in OCS & SE pages

Simulations for early validation OpSim4 (Operations Simulator) Observations Database Scheduler SOCS (Simulated OCS) MAF (Metrics Analysis Framework) Detailed observatory kinematic model Sky brightness model including twilight Real weather data from the LSST site

Analysis with MAF

Running simulated survey in OpSim Simulator provides: Weather data from the LSST site Downtime simulation Observatory telemetry simulation Scheduler provides: Observatory kinematic model Sky brightness model Science proposals implementing the LSST survey Scheduling algorithm Output Visit by visit simulation including telemetry record and observing conditions Detailed expected behavior of the observatory

OpSim preparation for Commissioning OpSim4 utilizes OCS Scheduler code Including the same Middleware DDS interface Build references for Mini-surveys Estimated system performance Estimated science metrics Monitor & Analysis tools Prepare toolset to support the analysis of mini-surveys Scheduler tested by simulating survey and mini-surveys

Control Software Verification Several control capabilities are fully tested before commissioning phase 1 (ComCam @summit), including: Middleware messaging system (Commands, Events and Telemetry) Engineering & Facility Database data capture OCS hierarchical control over TCS with hardware Telemetry Monitoring TCS ATCS (Auxiliary Telescope Control System) DMCS CCS (ComCam release) DAQ (ComCam release)

Control Software Verification Some key requirements can only be tested at full system integration Hierarchical Control with all Control Systems with hardware TCS, CCS, DAQ, DMCS Automatic Survey Real Telemetry for internal and external conditions Actual observatory kinematic behavior against internal Scheduler models Actual sky brightness distribution against internal Scheduler models

Control System Simulators Simulators for each Control system are used for early integration Interface Simulators available as OCS Middleware Software Development Kit Basic behavioral simulators provided for next path finder exercise (start/end night) Incremental complexity for following exercises (visit, night)

TCS ready for Commissioning with ComCam TCS verification has been implemented for the TCS supervisor application functioning with all TCS components. All functionalities available (except where LSSTCam is required) TCS supervisor Each TCS component tested with hardware TCS integrated with CCS-DAQ and OCS EFD in incremental steps DAQ test stand in Tucson ComCam integration in Tucson LSSTCam integration in SLAC And later ComCam in the summit LSSTCam in the summit

OCS ready for Commissioning with ComCam OCS verification has been implemented with all the OCS components functioning, commanding and supervising the observatory subsystems. The OCS Middleware v4 is transporting all the messages (commands, events and telemetry) between the OCS, TCS, CCS and DM. It is also transporting all the messages between TCS and its control devices. Support for C++, Python, Java and LabVIEW languages Transports all Commands, Events, Telemetry Control interfaces implemented in XML for all the control entities Callback mechanism White lister (control authorization mechanism) Heartbeat Quality of Service tuning (QoS)

OCS ready for Commissioning with ComCam The OCS Engineering and Facility Database v4 is storing all the Middleware messages. Integrated to Middleware v4 Time series tables for all topics Telemetry data stream records Command/Response records Event records Configuration records Large File Objects

OCS ready for Commissioning with ComCam The OCS Monitor v1 implements telemetry visualization and user interfaces deployed in the control room. Status summary and alarms of all main subsystems Environmental conditions Survey progress Data trends analysis display The OCS Controls v2 (Application & Sequencer) implement all the defined operational modes. Science, Engineering and Calibration observations Manual, Scripted and Fully autonomous operations

OCS ready for Commissioning with ComCam The OCS Scheduler v2.0 implements the sciences for the baseline survey and configurable proposals for commissioning activities. Scripted survey Rolling cadence Area Distribution with timed grouped visits option Time Distribution with deep drilling option Deterministic look-ahead Automatic dithering Filter swap logic Configurable value functions to balance intra and inter proposal progress Configurable cost functions to balance scientific goals with slew time and filter changes

OCS ready for Commissioning with LSSTCam The OCS Monitor v2 implements telemetry visualization and user interfaces deployed in the control room, evolved from the lessons learned from ComCam. The OCS Scheduler v2.2 implements the sciences for the baseline survey and configurable proposals for commissioning activities, with additional optimizations. Weather forecast handling Optimizations

Integration & Test environment Integration environment at the Base Integrate & Test control software Control Software in hardware simulation mode Used for integration, commissioning and subsequent releases during the software evolution Used for testing the software fixes and extensions implemented after issues have been reported and addressed Integrated test procedures based on the pathfinder exercises Telemetry exchange Start/End night sequence Visit sequence Full night Scheduler driven Failures and alarms handling

Quality Control Unit Tests Automated regressions tests Release Tests Integration Tests involving OCS+TCS+CCS+DAQ+DMCS

Scheduler Quality Control Unit Tests Automated regressions tests Release Tests Integration Tests involving OCS+TCS+CCS+DAQ+DMCS Integration Tests in OpSim environment Release Tests (includes installation and reference runs) Automated analysis tools Validation runs by Systems Engineering Simulations Scientific Validation by Science Collaborations

End of presentation