1. Summit-Base Tiger Team and IT Services Committee Results
August 14, 2017

2. LSST2017 - Tucson, AZ - August 14 - 18, 2017
Agenda
Background and Context
Summit – Base Information Technology and Communications Tiger Team (SBTT)
Charter and members
Accomplishments
Future
Ways to interact with the SBTT
Information Technology Services Committee (ITSC)
Accomplishments of the ITSC
Future of the ITSC
Ways to interact with the ITSC
LSST Tucson, AZ - August , 2017

3. PMO Task – Victor Krabendam email 20160524
Need vehicle(s) to support LSST MREFC Project on “all things ITC”
Act as “consultant” to PMO and project (hardware and software deliverables remain responsibility of subsystems)
Task 1: Ensure Summit – Base Complex is optimally designed/engineered to avoid ITC gaps, redundancies, inefficiencies (SBTT)
Task 2: Ensure consistency of life cycle management and support of all IT Services during MREFC project and as foundation for operations (ITSC)
Be sure to address IT Security in both
As you all know, DM Interim Leadership is in position and we are now formally seeking a new DM Manager to lead that subsystem. With this change Jeff Kantor's role has shifted to being the Technical/Control Account Manager for the DM long haul network work package and supporting me in the Project Office. Jeff will be working with the Telescope team to do some independent assessment and will be looking into overall IT work across the Project.
For the latter point, several of you have or will be contacted by Jeff to provide some data on current and planned use of software and hardware for both development and operational aspects of LSST.  I appreciate that you give this some attention as it is the first part of an overall assessment that I want to do regarding the LSST IT system. There are potential opportunities for commonality and efficiencies of labor and equipment in many areas of LSST IT, so we need to assemble the basic inventory of IT services and infrastructure to evaluate our plan for technical performance and completeness, efficiency, and commonality. 
A particular example area of current focus is the summit and base complex. There are many aspects that could be addressed consistently, from racks and power, to common hardware, to the networks, to the size of the buffers.  In addition to needing more definition in many subsystem areas, we have cross-cutting architectural aspects to consider, such as cyber-security.   Our WBS structure associated with the base and summit complex is spread across all three subsystems, so there are seams and boundaries to navigate, and we need to verify that there are no “holes” or misaligned interfaces.
The current motivation for the research Jeff is doing is for summit - base immediate needs, however we have spoken and met many times over the last 5 years about the need to have better coordination of IT designs across the subsystems. Meetings have been convened and topical coordination has occurred, but actions have often terminated without comprehensive steps forward. We have had positive feedback across the technical teams about defining common IT infrastructure that needs to be better defined and turned into action. With Jeff's availability we have an opportunity to get a single comprehensive view of what is being done, where, by whom, on what time scale, and to what ends. I believe that the scope resides within the three major subsystems, but without the single full view we cannot do a proper evaluation nor plan for the commonality and efficiencies everyone has expressed as desirable.
Concurrently we are also having discussions on how to best handle general IT services in the Project Office. The "business" IT team is doing an excellent job but the load is getting bigger and the effort is becoming more complicated. We are looking to better integrate expertise from SQuARE and PMO IT so that these co-located teams, in particular, are not stove-piped and might find a good staff and workload sharing model to serve the construction project that can then better inform how operations can be executed efficiently.
The first step to our optimization and overall assessment is to aggregate the pieces. I hope that the information gathering can happen alongside other work already in progress, like the current Operations Planning and Base development for example, so please continue to work with Jeff on the inventory so we can prepare the information for collective discussion on ways forward.

4. Task 1: Summit – Base Complex Integrity
We need to engineer the Summit – Base Complex as one ITC entity, with a consistent, coherent design
This cuts across subsystems that contribute facilities, equipment, software, operational policies and procedures
Many point-to-point communications and some workshops had tried to address various aspects of this as the subsystems have tried to do the work required in their “silos”, but…
We needed a more integrated approach to achieve the goal in time for early integration and through end of commissioning.
This was chartered in the form of a cross-subsystem “tiger team”, with a defined set of deliverables, a plan/schedule to provide them, and a T/CAM to track this and report on it
PW - I brought up the question on PTP requirement at the observatory and the conclusion is there isn’t a requirement driving the need for PTP. “It’s just what was written in to the requirements.” was the general response. However, there is a change request:  doesn’t explicitly state which time mechanism is needed, it does express that NTP would meet the defined needs. Should we update the IT document: Summit - Base Scientific ITC
o  to reflect that PTP isn’t needed and point to the change request?

5. Summit – Base ITC Design
We agreed we need Summit – Base ITC Design document(s) that are subsystem agnostic in terms of who delivers what and where it resides, and focus on:
End-to-end data flows and processes
Partitioning/allocation of responsibilities for above into logical nodes (e.g. “commissioning cluster, diagnostic cluster, image buffer, etc.)
Full definition of interfaces between nodes (expand ICDs as necessary)
Mapping of logical nodes to physical ITC infrastructure(DAQ, DWDM switch, etc.) and location (Summit, Base, both)
Designs for cross-cutting services (e.g. System Time, Security, Network(s), etc.)
AND ONLY AFTER THAT
Mapping of the above to WBS elements and subsystems providing the implementation of the design

6. SBTT Charter and members
The SBTT is a cross-subsystem team empowered to make technical decisions to optimize Summit Base ITC as an integrated system, i.e. not silo'ed by WBS.
Members:
Gregory Dubois-Felsmann (DM/SE)
Mike Huffer (Camera DAQ)
Tony Johnson (Camera CCS)
Jeff Kantor (PMO TCAM)
Ron Lambert (DM/TS)
German Schumacher (TS OCS/TCS/Integration)
Kian-Tat Lim (DM/SE)
Donald Petravick (DM)
Alexander Withers (ISO)
Paul Domagala (DM)
Iain Goodenow (PMO/ITS)
Chuck Claver (SE/Commissioning)
Dave Mills (TS OCS/TCS)
Note: Members are the only people who are given assignments; they are expected to reach out to others on the project as necessary to achieve their assignments.

7. SBTT Accomplishments
A wide range of design elements of the Summit – Base ITC design have been addressed
All of the TT proceedings are captured in weekly meeting minutes
The decisions/recommendations are all captured in LSE-309 Summit – Base ITC Design Document
LCR-963 was approved to establish initial baseline version of LSE-309
LCR-??? Has been submitted to address some open issues relative to the CCS fiber cable run and Clean Room, White Room, Staging Areas, misc. other
The current version is stable with respect to the overall complex, the Summit, and the Summit – Base Network, which are already in construction/equipment procurement and deployment
A new version will be prepared by the end of CY17 to expand to full coverage of the Base, to support FY18 construction/equipment procurement and deployment
LSE-299 Summit Computer Room Requirements was also developed, imported into Magic Draw, and updated
LCR-964 is under second round of CCB review to baseline LSE-299

8. SBTT Future
The Tiger Team started in the last quarter of FY16, and has continued through FY17 to accomplish Task 1
LSE-309 is the result of those meetings.  
The Tiger Team has fulfilled its primary mission when:
LSE-309 is baselined with the entire complex covered
The methods and timeline for implementation and verification are defined and accepted by the PMO, SE, and the subsystems
The current intent is to disband the Tiger Team at that time.
Any future architecture and design work and modifications to this document are expected to be a function of the LSST System Engineering team.

9. Ways to interact with the SBTT
We have a slack channel
We publish weekly meeting agendas and minutes in confluence
We invite topical experts to meetings when something of broad interest is covered
You can contact us via any project-standard vehicle, including JIRA issues, weekly management meetings, etc.
We all represent our respective project organizations, feel free to contact your SBTT member by any means you find convenient

10. Task 2: IT Services Life Cycle Management
We have many entities (subsystems, SE, SQuARE, PMO) selecting, purchasing, deploying, supporting, upgrading, and retiring IT Services for the MREFC project
We don’t have good vehicles for keeping everyone informed about all the services: who provides them, how to get support, are other groups planning something similar or have done something that can be leveraged, etc.
There are no internal “service levels” defining what level of service and support can be expected
Many point-to-point communications and “local” channels have been set up to to address various aspects, but…
We needed a more integrated approach to achieve the goal of optimal management and support
This was chartered in the form of a broad representative committee and a process
PW - I brought up the question on PTP requirement at the observatory and the conclusion is there isn’t a requirement driving the need for PTP. “It’s just what was written in to the requirements.” was the general response. However, there is a change request:  doesn’t explicitly state which time mechanism is needed, it does express that NTP would meet the defined needs. Should we update the IT document: Summit - Base Scientific ITC
o  to reflect that PTP isn’t needed and point to the change request?

11. ITSC Ecosystem and Roles
Service Users
LSST PO LSST Subsystems LSST Corporation
LSST Science Collaborations LSST Community
Strategy Governance Coordination
Service Providers
LSST ITS LSST SQuaRE NCSA
NOAO (N & S) University of Arizona
Commercial Service Providers

12. ITSC Charter To advise what ITS the project provides/uses
To make sure interoperability exists among ITS where possible
To combine, reuse, recycle ITS when possible
To prevent ITS from becoming stagnant or security hazards
Act as an agent of the project to keep project advised of what is going on at all spectrums of ITS
To make recommendation whether a particular IT Service continues, when to retire/replace
To make recommendations how the Project Office ITS (mail, web, issues, etc.) will transition into commissioning and operations

13. Governance & Transparency
The Project Office has final authority over any action or decision made by the IT Services Committee, especially if it results in a budgetary increase or has other consequences to project or observatory operations
Actions or Decisions cannot violate existing Information Technology Security Program and Compliance Policies
LPM 121 – LSST Master Information Security Plan
LPM 123 – Acceptable Use Policy
Subsystem IT Security Plans: LPM , LDM, LTS, LCAM
LPM 141 – Compliance Policy

14. Membership The IT Services Council (ITSC) consists of: Core Members
Representative Members

15. Core Members
The Core Members consist of representatives from entities that are key providers of IT Services to the project
Members:
PMO : Daniel Calabrese (chair), Jeff Kantor (ex officio)
Tucson IT Services Team: Iain Goodenow, Shahram Sobhani
LSST Information Security Officer: Alex Withers
NCSA: Paul Domogala
SQuARE: Frossie Economou
Non-voting
LSST Compliance Officer: Veronica Kinnison

16. Representative Members
The Representative Members represent a cross-section of the various project groups that have a responsibility for delivery consisting of and/or a vested interest in:
IT Assets (including Services)
Accessing IT Infrastructure
Impacting IT Security
Other IT related topics
Members
Systems Engineering: Brian Selvy, Michael Reuter
Data Management: Frossie Economou (also Core Member)
Telescope and Site: Harini Sundararaman
Camera: Tony Johnson
EPO: Ben Emmons

17. ITSC Accomplishments
Reference White Paper entitled: “General Architecture Principles” Version 1 –
LPM-51 update to to include ITSC role
Services Database with comprehensive inventory of current services and initial inventory of planned services
Clarification of Use of Third Party Tools: Only Whitelist is Supported
IT Requests for Consensus (RFCs) Process
ITRFCs to date:
ITRFC-1 Ben Emmons IMPLEMENTED LSST Digital Asset Manager
ITRFC-2 Iain Goodenow ADOPTED Upgrade Docushare
ITRFC-3 Paul Domagala ADOPTED Consolidate LSST Service Status
ITRFC-4 Paul Domagala ADOPTED Provision/integrate new IDM
ITRFC-5 Paul Domagala ADOPTED Unified incidents & requests
ITRFC-6 Iain Goodenow PROPOSED Puppet to be CMS
ITRFC-7 Alex Withers PROPOSED Dynamic Emergency Contact List

18. ITSC Future
The ITSC’s role of coordination and recommendation on ITS is currently chartered only in construction and commissioning
It is even more important that this role be performed in Operations
It is expected the ITSC or something like it will be chartered by the Observatory Director and will coordinate across Data Facility, Business, and Observatory Operations groups

19. Ways to interact with the ITSC
Your Representative Member is your primary interface to the ITSC
Responsible for making you aware of others IT Services plans, proposals, etc.
Responsible for making ITSC aware of your IT Services plans, proposals, etc.
Meetings are open to anyone, agendas and minutes are public
ITRFCs in JIRA are public (use Project = ITRFC to search issues)
Services database is currently accessible by ITSC, will be public in future