High-Level Software Overview for SOCS Ecosystem Michael Reuter OpSim Working Meeting July 7, 2017

Outline Locations SOCS Configuration UI OpSim4 Tools Docker

Repository Locations SOCS (https://github.com/lsst-sims/sims_ocs) Configuration UI (https://github.com/lsst-sims/opsim4_config_ui) Survey Fields (https://github.com/lsst/sims_survey_fields) OpSim4 Tools (https://github.com/lsst-sims/opsim4_tools) Scheduler (https://stash.lsstcorp.org/projects/TS/repos/ts_scheduler/ browse) Observatory Model (https://stash.lsstcorp.org/projects/TS/repos/ts_observatory_model/browse) Astronomical Sky Model (https://stash.lsstcorp.org/projects/TS/repos/ts_astrosky_model/browse) Date/Location (https://stash.lsstcorp.org/projects/TS/repos/ts_dateloc/browse) All Stash repositories are mirrored on GitHub (https://github.com/lsst-ts) for read-only access

Documentation Locations SOCS (https://lsst-sims.github.io/sims_ocs/) Configuration UI (https://lsst-sims.github.io/opsim4_config_ui/) Survey Fields (https://sims-survey-fields.lsst.io/) OpSim4 Tools (README.rst in every directory)

First Up

Operations Simulator Version 3 Scheduler

Operations Simulator Version 4 Simulated OCS (SOCS) Central Logger DDS Scheduler

What is SOCS? SOCS is NOT the Scheduler!!!!! Very lightweight derivation of OCS Just enough information to get Scheduler working Uses same SAL/DDS communication backplane as OCS Interoperability Is NOT the OCS simulator!!!!!

SOCS as a Project Nominally completed Remaining major development effort Weather maps Warm start Minor development necessary for parameters

Version 4 Block Diagram SOCS Simulation Targets Kernel Control Time SB Targets Sched Config Targets Control Downtime Downtime Model Time Sequencer Visits Current observation Observatory Observatory conditions Observatory Survey Database Model Sched Telem Sched Telem Environment Model Environmental Environment conditions Configuration SAL Config Configuration as built

Code Layout doc docker python/lsst/sims/ocs scripts tests ups

Configuration Encapsulates all configuration for SOCS and Scheduler Sub-packages handle observatory and proposal configuration Code stores baseline configuration (self-representation) Uses PexConfig system from DM Python object configuration Simple override system Full dump can be saved via driver flag Uses ConfigurationCommunicator to pass information across DDS to Scheduler

Database Encapsulates interactions with Survey Database SQLite only Two databases One session database Many simulation databases Table/Column documentation contained in implementation Except SummaryAllProps which is a view Documentation system culls this information SummaryAllProps needs columns handled

Downtime Scheduled downtime Same as version 3 SQLite DB Unscheduled downtime Uses same algorithm to derive v3 information Not the same schedule as v3 170/152 down days (v3: 185/137) 1 catastrophic downtime per 10 years https://github.com/mareuter/notebooks/blob/master/LSST/SOCS/UnscheduledDowntime.ipynb https://github.com/mareuter/notebooks/blob/master/LSST/SOCS/UnschedDowntimeSeedFinder.ipynb Can be run randomly (BEWARE!)

Environment Cloud and Seeing values from v3 Both models have concept of month start Can be overridden with other files

Kernel Simulator Overall orchestrator DDS message exchanges called from here TimeHandler Handles timestamp interactions DowntimeHandler Coordinates both downtime schedules Handles overlaps (if any) Sequencer Retrieves observatory state Performs observation of targets Passes along filter swaps Helpers for collecting proposal information

Observatory Uses TS Observatory Model package “Slews” observatory “Performs” visit Has simple variation model Linearly degrade telescope and dome velocities and accelerations Helpers for gathering exposure and slew information

SAL Abstracts some of the details of the SAL/DDS layer Create topic and perform publish or subscribe process in one step SOCS Publishers Configurations (12) Time Information Observatory State Cloud and Seeing Observation SOCS Subscribers Target Interested Proposal Filter Swap Topic definitions: https://docushare.lsstcorp.org/docushare/dsweb/Get/Document-18105/Scheduler_ICD.docx

Setup Argument parser setup Logging system configuration Logs to both console and file Levels effect detail (console only does two levels) Program configuration Information kept in OS level configuration file Run log tracking

Utilities Constants File path helper Session information helpers Exceptions

Scripts add_tracking Can insert records into Run Log from DB and log file central_logger Responsible for handling log info from SOCS and Scheduler manage_db Create or recreate session DB and configuration opsim4 Driver for Operations Simulator

Tests 300+ unit tests Approximately 95% coverage (grain of salt here) Not hooked into scons (cleanup task?) Run via pytest (install separately) drun py.test

Doc System for making GitHub Pages Pages are following versioned releases generate_tables.py Responsible for auto-generating survey database table documentation SummaryAllProps additions/subtractions need to be taken care of here to ensure consistency Top-level command to build GitHub Pages drun make gh-pages

Docker Contains necessary things for creating Docker container Scripts invoke the Docker builds build_base.sh Does OS and OS dependencies build_stack.sh Does LSST stack and stack installable packages build_opsim4.sh Does source installs and setup

Next Up

UI in a Nutshell View of currently “connected” SOCS configuration Visual feedback (color and indicator) for changed parameters Difference report tool to see them all together Apply overrides from another directory Starting from previously saved state Changes shown in normal way Proposal creation wizard Handles both types: General and Sequence

Code Features Written using PyQt5 Code uses Model-View-Controller (MVC) architecture Controllers do the coordination One per main tab, main program and base feature controller Models parse the SOCS configuration One per tab/sub-tab Widgets handle tab views Wizard widgets do not rely on controllers/models Unit tests only deal with models

Code Layout doc images (icons and backgrounds) python/lsst/opsim4/config/ui (build only, generated version) python/lsst/sims/opsim4 controller model widgets wizard scripts tests ups

Operations Simulator Organization https://github.com/lsst-opsim Only has repo related to MAF serving from ops2 Intended home for public-facing data configurations One configuration per repository Recommend not using repos for iterative work Unless really controlled sharing is desired Dropbox/Google Drive better

Next Up OpSim4 Tools

simple_qa check_runs.py Takes 2 OpSim4 SQLite DBs Does brute force comparison between all tables Can compare unequal length files Checks repeatability when same configuration is used

simple_plots plot_footprint.py Uses SOCS cut system to display proposal footprints Uses MAF for display

live_view opsim_live_polar.py Attaches to a observation message channel Either running simulation or served file db_server.py Creates a observation message stream from a SQLite DB Works on both v4 and v3 files Both require SAL/DDS library

Next Up

Particulars Scripts for building covered earlier Installation nearly identical, but not same, to source SAL library pre-compiled sims_skybrightness_pre handled as repo not package Optional sims_skybrightness package installed causing bloat in image size  Wrapper script not necessary One simulation per container GUI items need extra help (socat) Not deployed to DockerHub since not certified Mainly for OSX, Linux has permission problems

SSH Keys for Stash Generate and add key into profile on Stash Install key via ssh-add Clone repositories by the following: git clone ssh://git@stash.lsstcorp.org:7999/ts/<repo>.git