Input from DMG (and SsS) requirements that would help to streamline the data reduction workflow E. Villard

Inputs from SSG Open & unresolved SCIREQ tickets (367) Inputs from SSG to Feb 2016 AMT/IXT meeting (190) Recent inputs from SSG: AQUA Archive P2G Project tracker Scheduler SLT Source catalog

SCIREQ/ICT workflow could be improved Some ICT tickets created without a prior SCIREQ ticket Makes it difficult to get an overall picture Most SCIREQ tickets have more than one component Makes it difficult to filter by subsystems No match of components between SCIREQ and ICT tickets SCIREQ tickets not used just for requirements, but also for discussions and keeping track of work within WG

It’s a global issue Alternative is to relax the QA2 requirements Data processing is difficult because it is at the end of the flow, receiving all issues/inefficiencies from previous parts of the flow Data processing will benefit from: More detailed commissioning Strict technical assessment More complete E2E tests Strict scheduling/observing Cleaner data Strict/more detailed QA0 Alternative is to relax the QA2 requirements

Streamlining? Streamlining means making more efficient, not prettier Although prettier can mean more efficient, of course How to improve the efficiency of data processing? Make sure, before starting data processing, that the data have a very high chance of passing QA2. This means a very thorough QA0. Automate as much as possible the various steps of data processing Transfer temporary tools to long-term subsystems, to relieve the persons maintaining the temporary tools

1. A more thorough QA0 I think a very thorough QA0 means: A strict assessment of whether an EB is useful An accurate assessment of how useful an EB is Execution count for sensitivity/angular resolution Pre-QA0 flags Binary flags, online flags, QL flags (+ manual/AoD?) QL flags not fully implemented. Could be very helpful, but difficult, so would need careful attention. Would be ideal if those flags could be taken into account at QA0, but not sure how this could be done? Additional checks: SNR on calibrators, metadata

2. Automation Imaging Weblog reviewing State transitions First imaging PL is in operations. First impressions are quite good. I think effort needs to continue on development. Weblog reviewing Effort currently on streamlining layout. This is good, but I think we should start working on having an overall QA score (machine learning). State transitions Further triggering from AQUA QA0 or QA2 status

3. Transfer of temporary tools (see next talk) Developed initially for prototyping They are now working very well. The only issue is that they take time to maintain. Knowledge from mature tools should be transferred to official tools It will help free resources for data processing. But the outcome will not be just positive: the data processing workflow is still evolving, so scientists and developers will need to continue collaborating closely together for some time. Transition from EPT to PT will have to be planned carefully. I should not say that we want to have scientists and developers work together in the same place.

Two additional suggestions

Trending Some say: ”If you can’t measure it, you can’t manage it” Not entirely true, but a bit We need to be able to determine statistics, plot values, correlate various quantities related to the data processing workflow. Helps evaluate the performance of the tools (e.g. PL) Helps see an issue before it becomes serious Helps evaluate the effectiveness of a solution when trying to resolve an issue

Public release date This is working fine in most cases. Except when an issue is affecting a large number of datasets. It would be great if it could be set automatically. For instance: If an OUS is transitioned to QA3inProgress, the public release date could be set to default date far in the future. After an OUS in QA3 was re-observed and processed, the public release date could be re-set automatically, following the policy on extension of proprietary period.