1. SuperTask / Butler working group
Gregory Dubois-Felsmann LSST JTM, 6 March 2017, Glendale, CA

2. LSST JTM – 6 March 2017 – Glendale, CA
What is the issue?
Key Questions:
What is the interface through which Science Pipelines defines the actual units of work to be done in the production system?
How are other useful prepackaged units of work delivered?
For automated testing and science verification
For extensibility of the user interface
For ad-hoc use by scientists
How are these composed from the “Task”-level units used for the delivery of algorithmic code?
Translation
How should we evolve from CmdLineTask?
LSST JTM – 6 March 2017 – Glendale, CA

3. LSST JTM – 6 March 2017 – Glendale, CA
Other questions
How are these units of work configured?
How do they get their data? (How do they get their Butler, input DataIDs, output DataIDs?)
Tasks are supposed to not do I/O; need to more clearly define the layer that does do I/O…
Many constituencies & interested parties
Science Pipelines
Production workflow system
SQuaRE
SUIT
Data Access
LSST JTM – 6 March 2017 – Glendale, CA

4. Formation of a working group
Ideas have been circulating for quite some time under the “SuperTask” name…
Work in various areas has been inhibited by a lack of a clear decision on how to proceed.
January 2017 DMLT meeting decided:
Set up a working group to resolve this issue and a cluster of related ones
Use this as an opportunity to prove out a way to charter finite-duration working groups to answer technical questions
LSST JTM – 6 March 2017 – Glendale, CA

5. LSST JTM – 6 March 2017 – Glendale, CA
Charge
Taking into account the needs of all stakeholders:
Determine whether there is a need for a new architectural element of this nature in our design
If so…
Define the requirements
Produce a functional and API-level design
Cover those elements of the Butler that are tightly bound to the design
Enable a minimum viable implementation in S17, leading to widespread use, and migration from CmdLineTask, in F17
Coordinate planning among T/CAMs to facilitate this
Deliver the principal work products by April 12, 2017
LSST JTM – 6 March 2017 – Glendale, CA

6. LSST JTM – 6 March 2017 – Glendale, CA
Membership
Members of the Working Group are not there in a personal capacity – they are expected by the DM leadership to actively represent the interests of their constituencies.
Gregory Dubois-Felsmann, chair: Architecture, SUIT
Andy Salnikov: Data Access (and SuperTask developer)
Nate Pease: Data Access (and Butler developer)
Jonathan Sick: SQuaRE
Michelle Gower: Workflow Middleware / NCSA
Jim Bosch: Science Pipelines
The members are your representatives – please communicate through them!
LSST JTM – 6 March 2017 – Glendale, CA

7. LSST JTM – 6 March 2017 – Glendale, CA
Method of work
The working group began its work 11 days ago…
We will have weekly meetings – we have met twice so far
We will document our work in public, primarily on Confluence for in-process work, but moving to formal DM documents
Working group members are formally expected (agreed with T/CAMs) to put one day per week into supporting the effort; more work may be needed and will be scheduled
We will publish our recommendations as an RFC. Following review this is expected to settle the issue and enable work.
LSST JTM – 6 March 2017 – Glendale, CA

8. LSST JTM – 6 March 2017 – Glendale, CA
Initial actions
Review the range of work required to implement the LSST data productions, from the perspective of enumerating the transformations to be performed and understanding their inputs and outputs (especially their multiplicity and naming)
We have begun an analysis of the DRP workflows
Understand the requirements of the other stakeholders, initially:
Production workflow
Automated testing and documentation
We will use tomorrow’s timeslot (Tuesday 15:30) for a working group meeting
LSST JTM – 6 March 2017 – Glendale, CA