1. Input from DMG (and SsS) requirements that would help to streamline the data reduction workflow
E. Villard

2. Inputs from SSG Open & unresolved SCIREQ tickets (367)
Inputs from SSG to Feb 2016 AMT/IXT meeting (190)
Recent inputs from SSG:
AQUA
Archive
P2G
Project tracker
Scheduler
SLT
Source catalog

3. SCIREQ/ICT workflow could be improved
Some ICT tickets created without a prior SCIREQ ticket
Makes it difficult to get an overall picture
Most SCIREQ tickets have more than one component
Makes it difficult to filter by subsystems
No match of components between SCIREQ and ICT tickets
SCIREQ tickets not used just for requirements, but also for discussions and keeping track of work within WG

4. It’s a global issue Alternative is to relax the QA2 requirements
Data processing is difficult because it is at the end of the flow, receiving all issues/inefficiencies from previous parts of the flow
Data processing will benefit from:
More detailed commissioning
Strict technical assessment
More complete E2E tests
Strict scheduling/observing
Cleaner data
Strict/more detailed QA0
Alternative is to relax the QA2 requirements

5. Streamlining? Streamlining means making more efficient, not prettier
Although prettier can mean more efficient, of course
How to improve the efficiency of data processing?
Make sure, before starting data processing, that the data have a very high chance of passing QA2. This means a very thorough QA0.
Automate as much as possible the various steps of data processing
Transfer temporary tools to long-term subsystems, to relieve the persons maintaining the temporary tools

6. 1. A more thorough QA0 I think a very thorough QA0 means:
A strict assessment of whether an EB is useful
An accurate assessment of how useful an EB is
Execution count for sensitivity/angular resolution
Pre-QA0 flags
Binary flags, online flags, QL flags (+ manual/AoD?)
QL flags not fully implemented. Could be very helpful, but difficult, so would need careful attention.
Would be ideal if those flags could be taken into account at QA0, but not sure how this could be done?
Additional checks: SNR on calibrators, metadata

7. 2. Automation Imaging Weblog reviewing State transitions
First imaging PL is in operations. First impressions are quite good. I think effort needs to continue on development.
Weblog reviewing
Effort currently on streamlining layout. This is good, but I think we should start working on having an overall QA score (machine learning).
State transitions
Further triggering from AQUA QA0 or QA2 status

8. 3. Transfer of temporary tools (see next talk)
Developed initially for prototyping
They are now working very well. The only issue is that they take time to maintain.
Knowledge from mature tools should be transferred to official tools
It will help free resources for data processing.
But the outcome will not be just positive: the data processing workflow is still evolving, so scientists and developers will need to continue collaborating closely together for some time.
Transition from EPT to PT will have to be planned carefully.
I should not say that we want to have scientists and developers work together in the same place.

9. Two additional suggestions

10. Trending Some say: ”If you can’t measure it, you can’t manage it”
Not entirely true, but a bit
We need to be able to determine statistics, plot values, correlate various quantities related to the data processing workflow.
Helps evaluate the performance of the tools (e.g. PL)
Helps see an issue before it becomes serious
Helps evaluate the effectiveness of a solution when trying to resolve an issue

11. Public release date
This is working fine in most cases. Except when an issue is affecting a large number of datasets.
It would be great if it could be set automatically. For instance:
If an OUS is transitioned to QA3inProgress, the public release date could be set to default date far in the future.
After an OUS in QA3 was re-observed and processed, the public release date could be re-set automatically, following the policy on extension of proprietary period.