1. High-Level Software Overview for SOCS Ecosystem Michael Reuter OpSim Working Meeting July 7, 2017

2. Outline
Locations
SOCS
Configuration UI
OpSim4 Tools
Docker

3. Repository Locations SOCS (https://github.com/lsst-sims/sims_ocs)
Configuration UI (
Survey Fields (
OpSim4 Tools (
Scheduler ( browse)
Observatory Model (
Astronomical Sky Model (
Date/Location (
All Stash repositories are mirrored on GitHub ( for read-only access

4. Documentation Locations
SOCS (
Configuration UI (
Survey Fields (
OpSim4 Tools (README.rst in every directory)

5. First Up

6. Operations Simulator Version 3
Scheduler

7. Operations Simulator Version 4
Simulated OCS (SOCS)
Central Logger
DDS
Scheduler

8. What is SOCS?
SOCS is NOT the Scheduler!!!!!
Very lightweight derivation of OCS
Just enough information to get Scheduler working
Uses same SAL/DDS communication backplane as OCS
Interoperability
Is NOT the OCS simulator!!!!!

9. SOCS as a Project
Nominally completed
Remaining major development effort
Weather maps
Warm start
Minor development necessary for parameters

10. Version 4 Block Diagram SOCS Simulation Targets Kernel Control
Time SB
Targets
Sched Config
Targets
Control
Downtime
Downtime Model
Time
Sequencer
Visits
Current observation
Observatory
Observatory conditions
Observatory
Survey Database
Model
Sched Telem
Sched Telem
Environment Model
Environmental
Environment conditions
Configuration
SAL
Config
Configuration
as built

11. Code Layout
doc
docker
python/lsst/sims/ocs
scripts
tests
ups

12. Configuration
Encapsulates all configuration for SOCS and Scheduler
Sub-packages handle observatory and proposal configuration
Code stores baseline configuration (self-representation)
Uses PexConfig system from DM
Python object configuration
Simple override system
Full dump can be saved via driver flag
Uses ConfigurationCommunicator to pass information across DDS to Scheduler

13. Database
Encapsulates interactions with Survey Database
SQLite only
Two databases
One session database
Many simulation databases
Table/Column documentation contained in implementation
Except SummaryAllProps which is a view
Documentation system culls this information
SummaryAllProps needs columns handled

14. Downtime Scheduled downtime Same as version 3 SQLite DB
Unscheduled downtime
Uses same algorithm to derive v3 information
Not the same schedule as v3
170/152 down days (v3: 185/137)
1 catastrophic downtime per 10 years
Can be run randomly (BEWARE!)

15. Environment
Cloud and Seeing values from v3
Both models have concept of month start
Can be overridden with other files

16. Kernel Simulator Overall orchestrator
DDS message exchanges called from here
TimeHandler
Handles timestamp interactions
DowntimeHandler
Coordinates both downtime schedules
Handles overlaps (if any)
Sequencer
Retrieves observatory state
Performs observation of targets
Passes along filter swaps
Helpers for collecting proposal information

17. Observatory
Uses TS Observatory Model package
“Slews” observatory
“Performs” visit
Has simple variation model
Linearly degrade telescope and dome velocities and accelerations
Helpers for gathering exposure and slew information

18. SAL Abstracts some of the details of the SAL/DDS layer
Create topic and perform publish or subscribe process in one step
SOCS Publishers
Configurations (12)
Time Information
Observatory State
Cloud and Seeing
Observation
SOCS Subscribers
Target
Interested Proposal
Filter Swap
Topic definitions:

19. Setup
Argument parser setup
Logging system configuration
Logs to both console and file
Levels effect detail (console only does two levels)
Program configuration
Information kept in OS level configuration file
Run log tracking

20. Utilities
Constants
File path helper
Session information helpers
Exceptions

21. Scripts
add_tracking
Can insert records into Run Log from DB and log file
central_logger
Responsible for handling log info from SOCS and Scheduler
manage_db
Create or recreate session DB and configuration
opsim4
Driver for Operations Simulator

22. Tests
300+ unit tests
Approximately 95% coverage (grain of salt here)
Not hooked into scons (cleanup task?)
Run via pytest (install separately)
drun py.test

23. Doc
System for making GitHub Pages
Pages are following versioned releases
generate_tables.py
Responsible for auto-generating survey database table documentation
SummaryAllProps additions/subtractions need to be taken care of here to ensure consistency
Top-level command to build GitHub Pages
drun make gh-pages

24. Docker
Contains necessary things for creating Docker container
Scripts invoke the Docker builds
build_base.sh
Does OS and OS dependencies
build_stack.sh
Does LSST stack and stack installable packages
build_opsim4.sh
Does source installs and setup

25. Next Up

26. UI in a Nutshell
View of currently “connected” SOCS configuration
Visual feedback (color and indicator) for changed parameters
Difference report tool to see them all together
Apply overrides from another directory
Starting from previously saved state
Changes shown in normal way
Proposal creation wizard
Handles both types: General and Sequence

27. Code Features
Written using PyQt5
Code uses Model-View-Controller (MVC) architecture
Controllers do the coordination
One per main tab, main program and base feature controller
Models parse the SOCS configuration
One per tab/sub-tab
Widgets handle tab views
Wizard widgets do not rely on controllers/models
Unit tests only deal with models

28. Code Layout doc images (icons and backgrounds)
python/lsst/opsim4/config/ui (build only, generated version)
python/lsst/sims/opsim4
controller
model
widgets
wizard
scripts
tests
ups

29. Operations Simulator Organization
Only has repo related to MAF serving from ops2
Intended home for public-facing data configurations
One configuration per repository
Recommend not using repos for iterative work
Unless really controlled sharing is desired
Dropbox/Google Drive better

30. Next Up
OpSim4 Tools

31. simple_qa
check_runs.py
Takes 2 OpSim4 SQLite DBs
Does brute force comparison between all tables
Can compare unequal length files
Checks repeatability when same configuration is used

32. simple_plots
plot_footprint.py
Uses SOCS cut system to display proposal footprints
Uses MAF for display

33. live_view
opsim_live_polar.py
Attaches to a observation message channel
Either running simulation or served file
db_server.py
Creates a observation message stream from a SQLite DB
Works on both v4 and v3 files
Both require SAL/DDS library

34. Next Up

35. Particulars Scripts for building covered earlier
Installation nearly identical, but not same, to source
SAL library pre-compiled
sims_skybrightness_pre handled as repo not package
Optional sims_skybrightness package installed causing bloat in image size 
Wrapper script not necessary
One simulation per container
GUI items need extra help (socat)
Not deployed to DockerHub since not certified
Mainly for OSX, Linux has permission problems

36. SSH Keys for Stash
Generate and add key into profile on Stash
Install key via ssh-add
Clone repositories by the following:
git clone