1. LSST Control Software Integration Francisco Delgado OCS Software Manager LSST Commissioning Plan Review January 24-26, 2017

2. Control Software Scope Integration Timeline
Agenda
Control Software Scope
Integration Timeline
Pathfinders during early integration
Commissioning readiness
Commissioning with ComCam
Commissioning with LSSTCam

3. LSST Control Architecture
Goal: ensure successful integration of software products and robust interoperability
Observatory Control System
Francisco Delgado
OCS Communications Middleware
Tony Johnson
Camera
Control System
Telescope
Control System
Data Management
Control System
Add names & location to boxes
Image data
Camera Data Acquisition
Image data
Paul Lotz
Gregory Dubois-Felsmann
Mike Huffer

4. Development strategies in preparation for commissioning
Focus on interfaces between principal subsystems
Define comprehensive set of interfaces represented by Interface Control Documents (ICDs)
Provide common communication middleware for messaging
Elaborate interfaces in terms of system components architecture
Detail network infrastructure
Ensure timely progress through
Advancing level of detail in ICDs
Frequent intersystem communications
Milestones to mark and verify progress
Early interface testing

5. Control Software System Level ICDs
LSE-71 OCS <-> Camera Control System
LSE-72 OCS <-> Data Management
LSE-73 OCS <-> Telescope Control System
Interface Support Documents
LSE-70 OCS Communications Protocol
LSE-74 OCS Middleware Communications
LSE-209 OCS <-> Software Component

6. Interface Control Documents
Observatory Control System
LSE-70
LSE-74
LSE-209
OCS Communications Middleware
LSE-71
LSE-72
LSE-73
Camera
Control System
Telescope
Control System
Data Management
Control System
Camera Data Acquisition
Image data
LSE-66
LSE-67
Image data
LSE-68
LSE-69

7. Common communications middleware
OCS Middleware
Service Abstraction Layer (SAL) on top of Data Distribution Service (DDS)
Object Management Group (OMG) maintains DDS standard
LSE-70 documents requirements for SAL and clients, API, behaviors
LSE-74 incorporates data dictionary
Used by all principal subsystems

8. OCS Software Components
OCS Monitor
OCS Operator
OCS Remote
Control
OCS Scheduler
OCS
EFD
OCS Application
OCS Sequencer
History
Telemetry
Image Parameters
Targets
Visits
Cmds
OCS Communications Middleware
Telescope
Control System
Camera
Control System
Data Management
Control System

9. OCS+TCS+CCS+DAQ+DMCS Integration Timeline
Early Integration
Pathfinder Exercises
OCS+TCS
AT Integration & Test
@Summit
OCS+CCS+TCS+DAQ
ComCam
Integration & Test
@Tucson
2016 July
Middleware v3
EFD v3
Controls v0.1
2017 July
Middleware v4
EFD v4
Controls v1
Scheduler v1
2018 May
Controls v2
Scheduler v1.3
Commissioning Phase 0
2018 August AT
@Summit
DAQ
@Tucson
2018 October
TCS CCS DAQ DMCS
Control simulators
TCS(AT) DMCS
Control
CCS(ComCam)
Control

10. OCS+TCS+CCS+DAQ+DMCS Integration Timeline
Control System
Integration & Test
ComCam
@Summit
Early Science Verification
ComCam
Scheduler
2019 December
Monitor v1
Scheduler v2.0
Control System
Integration & Test
LSSTCam
@SLAC
Commissioning Phase 1 (ComCam)
2020 January
TCS CCS(ComCam) DMCS
Control
2020 February
LSSTCam
@SLAC

11. OCS+TCS+CCS+DAQ+DMCS Integration Timeline
Control System
Integration & Test
LSSTCam
Early Science Verification
LSSTCam
Scheduler
2019 December
Monitor v2
Scheduler v2.2
Commissioning Phase 2 (LSSTCam)
TCS CCS DM
Control
LSSTCam
@Summit
2020 November
LSSTCam
@Telescope

12. OCS+TCS+CCS+DAQ+DMCS Integration Timeline
2021 March
Scheduler
Mini-Surveys
Scheduler evaluation
Survey Monitor evolution
Commissioning Phase 3 (Mini-surveys)
Reference Zejlko

13. Frequent intersystem communications
Communication forums
Series of Principal System Interactions workshops
Dedicated face-to-face meetings or teleconferences
Breakouts at conferences
Working group discussions, systems engineering meetings, change requests, intranet discussions
Topics
Refinement of ICDs
Establishment of common IT requirements
Integration planning
Pathfinder exercise testing
Base for recurring tests in Integration & Test environment

14. Workshop History
The OCS-TCS-CCS-DAQ-DMCS + SE cross project team has been meeting regularly
Meetings focus on working out details of interfaces
Agendas, meeting minutes, actions, and agreements documented on confluence
Dates
Confluence Notes
May 12-14, 2014
Link
Nov 11, 2015
Nov 11-14, 2014
Dec 9, 2015
Feb 19-20, 2015
Mar 30, 2016
Apr 7, 2015
May 4, 2016
May 6-8, 2015
June 15, 2016
Jul 8-10, 2015
August 19, 2016
Oct 8, 2015
January 17, 2017

15. OCS-TCS-CCS-DAQ-DMCS early integration
Activity
Objectives
Status
OCS-CCS-DM Communication
Test objective: ensure middleware communication works properly between OCS, DM and Camera
Complete
CCS basic simulator
Test objective: ensure Camera commandable device works properly with OCS
Populating EFD with Camera Telemetry
Transfer simulated camera telemetry sent to the OCS and put into the EFD.
In progress
OCS-TCS-CCS-DAQ-DMCS Start/End of night test
Exercise the interfaces the OCS uses to interface with devices. Test component state machine functionality.
DMCS Demonstrates ability to compile and link against the v1 DAQ software
To show that DM and Camera DAQ are using compatible toolsets.
Test of EFD replication under DMCS device control
Connect the DM translation engine to the EFD. Properly query the EFD and retrieve all the information from the EFD.
Full OCS-TCS-CCS-DAQ-DMCS Test: Complete Visit
Basic flow: Configure, enable, take one visit, disable, shut down. This requires all components to do their appropriate functions. Execute the single visit sequence diagram with simulations of data as needed.
Basic flow: Configure, enable, take one visit, disable, shut down. Includes DAQ.
Full OCS-TCS-CCS--DAQ-DMCS Test: Mini Night
Include multiple visits, filter changes, cold starts. Use the OCS to coordinate getting all the systems to the right state. Take visits, give filter change command and make sure that DM knows which filter is in place. Multi visit - make sure DM doesn't confuse images.
Full OCS-TCS-CCS-DAQ-DMCS Test: Daily Cycle, Including Calibration
Simulation of a full 24 hour cycle.

16. Exercises specified in OCS & SE pages

17. Simulations for early validation
OpSim4
(Operations Simulator)
Observations Database
Scheduler
SOCS
(Simulated OCS)
MAF
(Metrics Analysis Framework)
Detailed observatory kinematic model
Sky brightness model including twilight
Real weather data from the LSST site

18. Analysis with MAF

19. Running simulated survey in OpSim
Simulator provides:
Weather data from the LSST site
Downtime simulation
Observatory telemetry simulation
Scheduler provides:
Observatory kinematic model
Sky brightness model
Science proposals implementing the LSST survey
Scheduling algorithm
Output
Visit by visit simulation including telemetry record and observing conditions
Detailed expected behavior of the observatory

20. OpSim preparation for Commissioning
OpSim4 utilizes OCS Scheduler code
Including the same Middleware DDS interface
Build references for Mini-surveys
Estimated system performance
Estimated science metrics
Monitor & Analysis tools
Prepare toolset to support the analysis of mini-surveys
Scheduler tested by simulating survey and mini-surveys

21. Control Software Verification
Several control capabilities are fully tested before commissioning phase 1 including:
Middleware messaging system (Commands, Events and Telemetry)
Engineering & Facility Database data capture
OCS hierarchical control over TCS with hardware
Telemetry Monitoring
TCS
ATCS (Auxiliary Telescope Control System)
DMCS
CCS (ComCam release)
DAQ (ComCam release)

22. Control Software Verification
Some key requirements can only be tested at full system integration
Hierarchical Control with all Control Systems with hardware
TCS, CCS, DAQ, DMCS
Automatic Survey
Real Telemetry for internal and external conditions
Actual observatory kinematic behavior against internal Scheduler models
Actual sky brightness distribution against internal Scheduler models

23. Control System Simulators
Simulators for each Control system are used for early integration
Interface Simulators available as OCS Middleware Software Development Kit
Basic behavioral simulators provided for next path finder exercise (start/end night)
Incremental complexity for following exercises (visit, night)

24. TCS ready for Commissioning with ComCam
TCS verification has been implemented for the TCS supervisor application functioning with all TCS components.
All functionalities available (except where LSSTCam is required)
TCS supervisor
Each TCS component tested with hardware
TCS integrated with CCS-DAQ and OCS EFD in incremental steps
DAQ test stand in Tucson
ComCam integration in Tucson
LSSTCam integration in SLAC
And later
ComCam in the summit
LSSTCam in the summit

25. OCS ready for Commissioning with ComCam
OCS verification has been implemented with all the OCS components functioning, commanding and supervising the observatory subsystems.
The OCS Middleware v4 is transporting all the messages (commands, events and telemetry) between the OCS, TCS, CCS and DM. It is also transporting all the messages between TCS and its control devices.
Support for C++, Python, Java and LabVIEW languages
Transports all Commands, Events, Telemetry
Control interfaces implemented in XML for all the control entities
Callback mechanism
White lister (control authorization mechanism)
Heartbeat
Quality of Service tuning (QoS)

26. OCS ready for Commissioning with ComCam
The OCS Engineering and Facility Database v4 is storing all the Middleware messages.
Integrated to Middleware v4
Time series tables for all topics
Telemetry data stream records
Command/Response records
Event records
Configuration records
Large File Objects

27. OCS ready for Commissioning with ComCam
The OCS Monitor v1 implements telemetry visualization and user interfaces deployed in the control room.
Status summary and alarms of all main subsystems
Environmental conditions
Survey progress
Data trends analysis display
The OCS Controls v2 (Application & Sequencer) implement all the defined operational modes.
Science, Engineering and Calibration observations
Manual, Scripted and Fully autonomous operations

28. OCS ready for Commissioning with ComCam
The OCS Scheduler v2.0 implements the sciences for the baseline survey and configurable proposals for commissioning activities.
Scripted survey
Rolling cadence
Area Distribution with timed grouped visits option
Time Distribution with deep drilling option
Deterministic look-ahead
Automatic dithering
Filter swap logic
Configurable value functions to balance intra and inter proposal progress
Configurable cost functions to balance scientific goals with slew time and filter changes

29. OCS ready for Commissioning with LSSTCam
The OCS Monitor v2 implements telemetry visualization and user interfaces deployed in the control room, evolved from the lessons learned from ComCam.
The OCS Scheduler v2.2 implements the sciences for the baseline survey and configurable proposals for commissioning activities, with additional optimizations.
Weather forecast handling
Optimizations

30. Integration & Test environment
Integration environment at the Base
Integrate & Test control software
Control Software in hardware simulation mode
Used for integration, commissioning and subsequent releases during the software evolution
Used for testing the software fixes and extensions implemented after issues have been reported and addressed
Integrated test procedures based on the pathfinder exercises
Telemetry exchange
Start/End night sequence
Visit sequence
Full night Scheduler driven
Failures and alarms handling

31. Quality Control
Unit Tests
Automated regressions tests
Release Tests
Integration Tests involving OCS+TCS+CCS+DAQ+DMCS

32. Scheduler Quality Control
Unit Tests
Automated regressions tests
Release Tests
Integration Tests involving OCS+TCS+CCS+DAQ+DMCS
Integration Tests in OpSim environment
Release Tests (includes installation and reference runs)
Automated analysis tools
Validation runs by Systems Engineering Simulations
Scientific Validation by Science Collaborations

33. End of presentation