LSST Operations Planning Beth Willman Joint Technical Meeting March 8, 2017

Timeline and November Review Outcomes

LSST Development Timeline

LSST Development Timeline Commissioning Pre- Operations Start of construction Operations Proposal Due

Commissioning and Pre-Operations Full survey operations is scheduled to begin on October 1, 2022 The Operations Proposal will be submitted to the NSF and DOE in Summer 2017 Expected start of NSF, DOE and LSSTC operations funding is FY 2019

Key operations planning activities since July A resourced loaded FY19-34 Plan, including escalation and institutional mapping The costing continues to be refined in response to: updates to the DOE contributions to LSST Operations, updates to the AURA labor costing model, reviewing non-labor expenses for missing scope, and feedback from the November Operations Review. Version 1 of LSST’s Operations Plan Version 2 of this document will be generated (expected by the end of March), based on feedback from the November review. A comprehensive review of LSST’s Operations Plan Nov 7-9. A variety of reviews have shined a light on the need to improve fidelity of transition plans.

November Operations Review 15 reviewers in 5 topical sub-groups: Michael Tuts (Chair), Mark Adams, Eric Christensen, Stuartt Corder, Kyle Dawson, Richard Hislop, Michiel van der Hoeven, Don Hough, Eric Lancon, Patty McBride, Mark Phillips, Dan Reed, Connie Rockosi, Arfon Smith, Frank Wuerthwein Sub-groups: 1. Organization and Management, Director’s Office, Business Operations, 2. Observatory Operations, 3. Data Products Production, 4. Science Operations, 5. EPO/Communications

Key Recommendations Develop a more detailed transition plan, once the Commissioning Plan has been developed and reviewed. Look for additional operational efficiencies, as the NCOA model develops. Clarify and increase the level of integration of Observatory Operations, Science Operations, Data Facility and Operations, and EPO activities. After implementing cross-department optimizations and NCOA efficiencies, related cost-scrubbing should be done.

Additional Related Follow-up Activities Develop additional cross-departmental use-cases, and add to Operations Plan Work on a less silo’ed approach to Operations Planning; Develop plans for cross-departmental transition activities. Review the travel budget Review possible optimization of the EPO data needs w/ planned LSST Data Facility (NCSA) functionality Improve fidelity of the plan for relative roles of Director and Deputies, with an eye on facilitating cross-departmental interactions Think more explicitly about the management tools to use during operations

High Level planning activities Jan – Apr 2017 Finalize the governance model Incorporate LSST FTE model into NCOA plan “Finish” the Operations Plan document Develop Commissioning – Operations transition plans ============ Not a planning activity: First draft of Operations Proposal in April. There is a (non-MREFC) budget identified for supporting proposal-specific work.

Basic LSST Operations Model

Total budget comparison with FDR estimates DRAFT These are all in then year dollars FDR – Escalated costs as presented in FDR Ops Plan Nov Review – The costing as presented at the November Review Dec Update – Costing updated with refreshed AURA Labor rates, and minor corrections to non-labor costs.

FTEs per year – Nov 2016 model post operations pre-operations 10 year survey DRAFT

Distribution of Job Types DRAFT

Institutional and geographic distribution are also necessary inputs to cost model French satellite center (CC-IN2P3, Lyon, France) Data Release Production (50%) French DAC SLAC Center Data Products Production Support Science Operations and Community Support Archive Site Archive Center Alert Production Data Release Production (50%) Long-term Storage (copy 2) Data Access Center Data Access and User Services Summit and Base Sites Telescope and Camera Operations Data Acquisition Long-term storage (copy 1) Chilean Data Access Center HQ Site Science Operations headquarters Operations office Education and Public Outreach

Operations will be distributed LSST Operations will be conducted at different geographic locations, and by staff employed by different institutions: NCOA/AURA, SLAC (and other DOE Labs), NCSA, and subcontracted affiliates (e.g. UW, Princeton, IPAC). NCOA, NCSA and subcontracted affiliate staff will be funded by the NSF and LSSTC. All funding will pass through AURA. SLAC, Fermilab and other DOE lab staff will be funded by the DOE. All funding will likely pass through SLAC. Distributing operations activities is beneficial because: Specialized expertise is geographically and institutionally distributed Efficiency can be introduced when activities are focused at centers with similar, ongoing activities Funding boundary conditions

Functional Mapping of Operations Activities NCOA + subcontracted affiliates Leads all aspects of operations, except for (i) the Data Facility and operations, (ii) camera operation and maintenance. SLAC (lead), Fermilab, and other DOE labs Provide leadership to the operations project (a SLAC deputy director) All aspects of camera operations on the summit, support of camera maintenance [including retention of expertise], including systems operations and monitoring as they relate to the LSST camera. Observing support. Support the Data Facility operations Support science operations activities, community support for LSST software systems, and science products and pipelines activities most closely related to DE-specific science requirements NCSA Leads the LSST Data Facility and Operations

Institutional mapping of operations FTEs Draft Jan 2017 model of FY27 Plan ~20% of FTEs AURA, and subcontracted affiliates + NCSA SLAC (on behalf of DOE labs) 156.7 FTEs DOE Labs Chile DOE Labs DOE Labs Sub-contractors HQ DOE Labs DOE Labs Chile HQ NCSA HQ Chile NCSA Chile HQ 43.75 6.25 10 0.5 4.0 3 18.5 25 17.7 1.25 0.5 1 9.5 8.5 3 4 Observatory Ops Science Ops LSST Data Facility Director’s Office Business EPO and comm. DRAFT

Transition Planning: Proposed pre-operations activities

Commissioning and Pre-Operations Full survey operations is scheduled to begin on October 1, 2022 The Operations Proposal will be submitted to the NSF and DOE in Summer 2017 Expected start of NSF, DOE and LSSTC operations funding is FY 2019

The Pre-Operations Phase A pre-operations phase will take place from FY19 - FY22, in parallel with commissioning. This is a ramp up period, in advance of full operations, to prepare to operate effectively and efficiently at the start of full operations on October 1, 2022. Goals of pre-operations Knowledge transfer, staff ramp-up and training Training in key procedures for operations on the fully integrated system, e.g. mirror re-coat + transport, camera maintenance that doesn’t require opening dewar Ramp up full Data Access and Archive center Operations, including scaling up community use of the Science Platform, and full user authentication services Begin Science operations activities, including setting up procedures for community support, ingesting and reacting to community feedback, a helpdesk, early assessment of the Scheduler’s efficacy for science goals

The Pre-Operations Phase The planned pre-operations team will be able to support the dissemination of commissioning data to the community. A commissioning data release policy has been drafted to balance the mutual benefits of early data dissemination with realistic resource limitations, as well as balance the benefits of maximizing early science w/ the need to provide equal access to all data. The Project needs to put structures in place to ingest and react to feedback from the community.

Proposed Pre-Operations Activities

Summary Operations planning never ends, but this phase of operations planning will soon wrap up. The LSST Operations proposal will be submitted before the August All-Hands meeting.

Extra Slides

Defining the Boundary of LSST Operations Science Operations functions will include: Provide limited support for community use of and access to data products and computing resources, including provision of docs; Maintain and evolve the Science User Interface and associated services (e.g. Jupyter notebook capabilities); Identify and implement necessary changes to the scientific algorithms and pipelines for data processing Example use cases discussed with the agencies to clarify the boundary of Operations activities to fund through the LSST Operations project. There are other related ongoing discussions, some related to understanding descope options. A dynamic field. How we access/analyze/interact with data isn’t yet clear. A lot of development is going on. What DataLab is doing to support interactions w/ DES is of significant benefit for us to learn from.

Defining the Boundary of LSST Operations User Support: High: Triage and delegate data anomalies, e.g. Why does a color-magnitude diagram look odd, if triage finds that it’s not user error High: Documentation Medium: Synthesizing common user issues, and updating documentation and/or procedures to address No: Helping users w/ basics e.g. Why does a color-magnitude diagram look odd, if triage finds that the user is insufficiently knowledgeable No: 24/7 support. Real-time support. Some of these “out of scope” examples, as well as additional community support functionality, will become the responsibility of an NOAO/NCOIRA Community Science Center

Defining the Boundary of LSST Operations Science Platform group: High: Fixing bugs High: Fixing issues that affect the general usability of the data products Medium: Implementing upgrades to remain consistent with technical and community standards Low: Requests for new features of general usefulness No: Requests for troubleshooting user’s python notebooks Some of these “out of scope” examples, as well as additional community support functionality, will become the responsibility of an NOAO/NCOIRA Community Science Center

Defining the Boundary of LSST Operations Science algorithms, pipelines, and calibration development High: Reacting to errors in measured quantities, including calibration, exceed the limits set by the requirements High: Reacting to large numbers of spurious sources (including alerts and problems with deblending algorithm) Medium: Modifications of existing algorithms to enhance the scientific utility of data products (including improved calibration procedures and ingesting contributed code by others) No: Development of new algorithms to enable additional science programs (this would be Level 3) Some of these “out of scope” examples, as well as additional community support functionality, will become the responsibility of an NOAO/NCOIRA Community Science Center

Beyond the Boundary of LSST Operations Study timing: Fall 2016-Fall 2017 Workshop: May 2-4, 2016 Report posted: Oct 5, 2016 http://www.noao.edu/meetings/lsst-oir-study/

Beyond the Boundary of LSST Operations LSST’s data products themselves will revolutionize many areas of astronomy. The need for support outside of the LSST operations boundary to maximize LSST science has been identified and studied by the 2015 Elmegreen report, and the Najita & Willman community-driven “Maximizing Science” report. Anticipated additional support needs include: Follow-up time on telescopes/instruments An alert broker Community training in and support for large-scale astronomical analysis techniques, and computing technologies Computing and software resources to support intensive science analysis

Beyond the Boundary of LSST Operations NOAO/NCOA Community Science Support NOAO has been asked to develop a concept for a Community Science Center that will support LSST science users. This center will focus on helping the community go from data products to science, including: (i) time-domain science and follow-up, (ii) assisting large-catalog science analysis + related follow-up, (iii) assisting generation of user-developed data products and scale-up of analyses to HPC resources, (iv) community-based planning exercises, (v) schools and training. Beth Willman, Adam Bolton (Associate Director for NOAO’s Community Science and Data Center), and Dara Norman (Deputy AD for NOAO’s CSDC) are in active communication about this planning. Given this scope and qualitative level of service expected from Science Operations department – What are the leadership roles of science operations, and what are the interfaces between these leads and other departments and the community?