SuperTask / Butler working group Gregory Dubois-Felsmann LSST JTM, 6 March 2017, Glendale, CA

LSST JTM – 6 March 2017 – Glendale, CA What is the issue? Key Questions: What is the interface through which Science Pipelines defines the actual units of work to be done in the production system? How are other useful prepackaged units of work delivered? For automated testing and science verification For extensibility of the user interface For ad-hoc use by scientists How are these composed from the “Task”-level units used for the delivery of algorithmic code? Translation How should we evolve from CmdLineTask? LSST JTM – 6 March 2017 – Glendale, CA

LSST JTM – 6 March 2017 – Glendale, CA Other questions How are these units of work configured? How do they get their data? (How do they get their Butler, input DataIDs, output DataIDs?) Tasks are supposed to not do I/O; need to more clearly define the layer that does do I/O… Many constituencies & interested parties Science Pipelines Production workflow system SQuaRE SUIT Data Access LSST JTM – 6 March 2017 – Glendale, CA

Formation of a working group Ideas have been circulating for quite some time under the “SuperTask” name… Work in various areas has been inhibited by a lack of a clear decision on how to proceed. January 2017 DMLT meeting decided: Set up a working group to resolve this issue and a cluster of related ones Use this as an opportunity to prove out a way to charter finite-duration working groups to answer technical questions LSST JTM – 6 March 2017 – Glendale, CA

LSST JTM – 6 March 2017 – Glendale, CA Charge Taking into account the needs of all stakeholders: Determine whether there is a need for a new architectural element of this nature in our design If so… Define the requirements Produce a functional and API-level design Cover those elements of the Butler that are tightly bound to the design Enable a minimum viable implementation in S17, leading to widespread use, and migration from CmdLineTask, in F17 Coordinate planning among T/CAMs to facilitate this Deliver the principal work products by April 12, 2017 LSST JTM – 6 March 2017 – Glendale, CA

LSST JTM – 6 March 2017 – Glendale, CA Membership Members of the Working Group are not there in a personal capacity – they are expected by the DM leadership to actively represent the interests of their constituencies. Gregory Dubois-Felsmann, chair: Architecture, SUIT Andy Salnikov: Data Access (and SuperTask developer) Nate Pease: Data Access (and Butler developer) Jonathan Sick: SQuaRE Michelle Gower: Workflow Middleware / NCSA Jim Bosch: Science Pipelines The members are your representatives – please communicate through them! LSST JTM – 6 March 2017 – Glendale, CA

LSST JTM – 6 March 2017 – Glendale, CA Method of work The working group began its work 11 days ago… We will have weekly meetings – we have met twice so far We will document our work in public, primarily on Confluence for in-process work, but moving to formal DM documents https://confluence.lsstcorp.org/display/DM/SuperTask Working group members are formally expected (agreed with T/CAMs) to put one day per week into supporting the effort; more work may be needed and will be scheduled We will publish our recommendations as an RFC. Following review this is expected to settle the issue and enable work. LSST JTM – 6 March 2017 – Glendale, CA

LSST JTM – 6 March 2017 – Glendale, CA Initial actions Review the range of work required to implement the LSST data productions, from the perspective of enumerating the transformations to be performed and understanding their inputs and outputs (especially their multiplicity and naming) We have begun an analysis of the DRP workflows Understand the requirements of the other stakeholders, initially: Production workflow Automated testing and documentation We will use tomorrow’s timeslot (Tuesday 15:30) for a working group meeting LSST JTM – 6 March 2017 – Glendale, CA