1. LSST Operations Planning Beth Willman Joint Technical Meeting March 8, 2017

2. Timeline and November Review Outcomes

3. LSST Development Timeline

4. LSST Development Timeline
Commissioning
Pre- Operations
Start of construction
Operations Proposal Due

5. Commissioning and Pre-Operations
Full survey operations is scheduled to begin on October 1, 2022
The Operations Proposal will be submitted to the NSF and DOE in Summer 2017
Expected start of NSF, DOE and LSSTC operations funding is FY 2019

6. Key operations planning activities since July
A resourced loaded FY19-34 Plan, including escalation and institutional mapping
The costing continues to be refined in response to: updates to the DOE contributions to LSST Operations, updates to the AURA labor costing model, reviewing non-labor expenses for missing scope, and feedback from the November Operations Review.
Version 1 of LSST’s Operations Plan
Version 2 of this document will be generated (expected by the end of March), based on feedback from the November review.
A comprehensive review of LSST’s Operations Plan Nov 7-9.
A variety of reviews have shined a light on the need to improve fidelity of transition plans.

7. November Operations Review
15 reviewers in 5 topical sub-groups: Michael Tuts (Chair), Mark Adams, Eric Christensen, Stuartt Corder, Kyle Dawson, Richard Hislop, Michiel van der Hoeven, Don Hough, Eric Lancon, Patty McBride, Mark Phillips, Dan Reed, Connie Rockosi, Arfon Smith, Frank Wuerthwein
Sub-groups: 1. Organization and Management, Director’s Office, Business Operations, 2. Observatory Operations, 3. Data Products Production, 4. Science Operations, 5. EPO/Communications

8. Key Recommendations
Develop a more detailed transition plan, once the Commissioning Plan has been developed and reviewed.
Look for additional operational efficiencies, as the NCOA model develops.
Clarify and increase the level of integration of Observatory Operations, Science Operations, Data Facility and Operations, and EPO activities.
After implementing cross-department optimizations and NCOA efficiencies, related cost-scrubbing should be done.

9. Additional Related Follow-up Activities
Develop additional cross-departmental use-cases, and add to Operations Plan
Work on a less silo’ed approach to Operations Planning; Develop plans for cross-departmental transition activities.
Review the travel budget
Review possible optimization of the EPO data needs w/ planned LSST Data Facility (NCSA) functionality
Improve fidelity of the plan for relative roles of Director and Deputies, with an eye on facilitating cross-departmental interactions
Think more explicitly about the management tools to use during operations

10. High Level planning activities Jan – Apr 2017
Finalize the governance model
Incorporate LSST FTE model into NCOA plan
“Finish” the Operations Plan document
Develop Commissioning – Operations transition plans
============
Not a planning activity: First draft of Operations Proposal in April.
There is a (non-MREFC) budget identified for supporting proposal-specific work.

11. Basic LSST Operations Model

12. Total budget comparison with FDR estimates
DRAFT
These are all in then year dollars
FDR – Escalated costs as presented in FDR Ops Plan
Nov Review – The costing as presented at the November Review
Dec Update – Costing updated with refreshed AURA Labor rates, and minor corrections to non-labor costs.

13. FTEs per year – Nov 2016 model
post operations
pre-operations
10 year survey
DRAFT

14. Distribution of Job Types
DRAFT

15. Institutional and geographic distribution are also necessary inputs to cost model
French satellite center (CC-IN2P3, Lyon, France)
Data Release Production (50%)
French DAC
SLAC Center
Data Products Production Support
Science Operations and Community Support
Archive Site
Archive Center
Alert Production
Data Release Production (50%)
Long-term Storage (copy 2)
Data Access Center
Data Access and User Services
Summit and Base Sites
Telescope and Camera Operations
Data Acquisition Long-term storage (copy 1)
Chilean Data Access Center
HQ Site
Science Operations headquarters Operations office Education and Public Outreach

16. Operations will be distributed
LSST Operations will be conducted at different geographic locations, and by staff employed by different institutions: NCOA/AURA, SLAC (and other DOE Labs), NCSA, and subcontracted affiliates (e.g. UW, Princeton, IPAC).
NCOA, NCSA and subcontracted affiliate staff will be funded by the NSF and LSSTC. All funding will pass through AURA.
SLAC, Fermilab and other DOE lab staff will be funded by the DOE. All funding will likely pass through SLAC.
Distributing operations activities is beneficial because:
Specialized expertise is geographically and institutionally distributed
Efficiency can be introduced when activities are focused at centers with similar, ongoing activities
Funding boundary conditions

17. Functional Mapping of Operations Activities
NCOA + subcontracted affiliates
Leads all aspects of operations, except for (i) the Data Facility and operations, (ii) camera operation and maintenance.
SLAC (lead), Fermilab, and other DOE labs
Provide leadership to the operations project (a SLAC deputy director)
All aspects of camera operations on the summit, support of camera maintenance [including retention of expertise], including systems operations and monitoring as they relate to the LSST camera. Observing support.
Support the Data Facility operations
Support science operations activities, community support for LSST software systems, and science products and pipelines activities most closely related to DE-specific science requirements
NCSA
Leads the LSST Data Facility and Operations

18. Institutional mapping of operations FTEs Draft Jan 2017 model of FY27 Plan ~20% of FTEs
AURA, and subcontracted affiliates + NCSA
SLAC (on behalf of DOE labs)
156.7 FTEs
DOE Labs
Chile
DOE Labs
DOE Labs
Sub-contractors HQ
DOE Labs
DOE Labs
Chile HQ
NCSA HQ
Chile
NCSA
Chile HQ
43.75
6.25 10
0.5
4.0 3
18.5 25
17.7
1.25
0.5 1
9.5
8.5 3 4
Observatory
Ops
Science
Ops
LSST Data Facility
Director’s Office
Business
EPO and comm.
DRAFT

19. Transition Planning: Proposed pre-operations activities

20. Commissioning and Pre-Operations
Full survey operations is scheduled to begin on October 1, 2022
The Operations Proposal will be submitted to the NSF and DOE in Summer 2017
Expected start of NSF, DOE and LSSTC operations funding is FY 2019

21. The Pre-Operations Phase
A pre-operations phase will take place from FY19 - FY22, in parallel with commissioning. This is a ramp up period, in advance of full operations, to prepare to operate effectively and efficiently at the start of full operations on October 1, 2022.
Goals of pre-operations
Knowledge transfer, staff ramp-up and training
Training in key procedures for operations on the fully integrated system, e.g. mirror re-coat + transport, camera maintenance that doesn’t require opening dewar
Ramp up full Data Access and Archive center Operations, including scaling up community use of the Science Platform, and full user authentication services
Begin Science operations activities, including setting up procedures for community support, ingesting and reacting to community feedback, a helpdesk, early assessment of the Scheduler’s efficacy for science goals

22. The Pre-Operations Phase
The planned pre-operations team will be able to support the dissemination of commissioning data to the community. A commissioning data release policy has been drafted to balance the mutual benefits of early data dissemination with realistic resource limitations, as well as balance the benefits of maximizing early science w/ the need to provide equal access to all data. The Project needs to put structures in place to ingest and react to feedback from the community.

23. Proposed Pre-Operations Activities

24. Summary
Operations planning never ends, but this phase of operations planning will soon wrap up.
The LSST Operations proposal will be submitted before the August All-Hands meeting.

25. Extra Slides

26. Defining the Boundary of LSST Operations
Science Operations functions will include:
Provide limited support for community use of and access to data products and computing resources, including provision of docs;
Maintain and evolve the Science User Interface and associated services (e.g. Jupyter notebook capabilities);
Identify and implement necessary changes to the scientific algorithms and pipelines for data processing
Example use cases discussed with the agencies to clarify the boundary of Operations activities to fund through the LSST Operations project.
There are other related ongoing discussions, some related to understanding descope options.
A dynamic field. How we access/analyze/interact with data isn’t yet clear. A lot of development is going on. What DataLab is doing to support interactions w/ DES is of significant benefit for us to learn from.

27. Defining the Boundary of LSST Operations
User Support:
High: Triage and delegate data anomalies, e.g. Why does a color-magnitude diagram look odd, if triage finds that it’s not user error
High: Documentation
Medium: Synthesizing common user issues, and updating documentation and/or procedures to address
No: Helping users w/ basics e.g. Why does a color-magnitude diagram look odd, if triage finds that the user is insufficiently knowledgeable
No: 24/7 support. Real-time support.
Some of these “out of scope” examples, as well as additional community support functionality, will become the responsibility of an NOAO/NCOIRA Community Science Center

28. Defining the Boundary of LSST Operations
Science Platform group:
High: Fixing bugs
High: Fixing issues that affect the general usability of the data products
Medium: Implementing upgrades to remain consistent with technical and community standards
Low: Requests for new features of general usefulness
No: Requests for troubleshooting user’s python notebooks
Some of these “out of scope” examples, as well as additional community support functionality, will become the responsibility of an NOAO/NCOIRA Community Science Center

29. Defining the Boundary of LSST Operations
Science algorithms, pipelines, and calibration development
High: Reacting to errors in measured quantities, including calibration, exceed the limits set by the requirements
High: Reacting to large numbers of spurious sources (including alerts and problems with deblending algorithm)
Medium: Modifications of existing algorithms to enhance the scientific utility of data products (including improved calibration procedures and ingesting contributed code by others)
No: Development of new algorithms to enable additional science programs (this would be Level 3)
Some of these “out of scope” examples, as well as additional community support functionality, will become the responsibility of an NOAO/NCOIRA Community Science Center

30. Beyond the Boundary of LSST Operations
Study timing: Fall 2016-Fall 2017
Workshop: May 2-4, 2016
Report posted: Oct 5, 2016

31. Beyond the Boundary of LSST Operations
LSST’s data products themselves will revolutionize many areas of astronomy.
The need for support outside of the LSST operations boundary to maximize LSST science has been identified and studied by the 2015 Elmegreen report, and the Najita & Willman community-driven “Maximizing Science” report. Anticipated additional support needs include:
Follow-up time on telescopes/instruments
An alert broker
Community training in and support for large-scale astronomical analysis techniques, and computing technologies
Computing and software resources to support intensive science analysis

32. Beyond the Boundary of LSST Operations
NOAO/NCOA Community Science Support
NOAO has been asked to develop a concept for a Community Science Center that will support LSST science users. This center will focus on helping the community go from data products to science, including:
(i) time-domain science and follow-up,
(ii) assisting large-catalog science analysis + related follow-up,
(iii) assisting generation of user-developed data products and scale-up of analyses to HPC resources,
(iv) community-based planning exercises,
(v) schools and training.
Beth Willman, Adam Bolton (Associate Director for NOAO’s Community Science and Data Center), and Dara Norman (Deputy AD for NOAO’s CSDC) are in active communication about this planning.
Given this scope and qualitative level of service expected from Science Operations department – What are the leadership roles of science operations, and what are the interfaces between these leads and other departments and the community?