CWG9 and Event Display B. von Haller CERN

CWG 9 in a nutshell ▶ Started in May 2013 along with O 2 ▶ Group working on ▶ the Data Quality Monitoring ▶ the Quality Assurance ▶ the Visualization ▶ For Run 2 and Run 3 B. von Haller | CWG9 DQM-QA-Visualization |

CWG9 Members ▶ Renu Bala ▶ Francesca Bellini ▶ Mihaela Gheata ▶ Lukasz Graczykowski ▶ Malgorzata Janik ▶ Andreas Morsch ▶ Mihai Niculescu ▶ Jeremi Niedziela ▶ Ankita Sharma ▶ Maciej Szymanski ▶ Barthélémy Von Haller B. von Haller | CWG9 DQM-QA-Visualization |

Definitions of DQM and QA Data Quality Monitoring & Quality Assurance B. von Haller | CWG9 DQM-QA-Visualization | ▶ Feedback on the quality of data ▶ Online (DQM) ▶ Make sure to record high quality data ▶ Identify and solve problem(s) early ▶ Offline (QA) ▶ Make sure to analyze high quality data ▶ Identify high quality runs ▶ Involves ▶ [Online gathering of data] ▶ Analysis by user-defined algorithm ▶ Production of monitoring objects such as histograms ▶ Assessment of the quality of the data based on the objects ▶ Storage of monitoring data ▶ Visualization (+ human assessment of quality)

B. von Haller | CWG9 DQM-QA-Visualization | CWG9 Tasks ▶ Group focusing on the data Quality Assurance, online and offline, and the visualization of its results and the data itself ▶ Run 3 ▶ Study how to monitor data efficiently and in plenty without interfering with the data taking ▶ Discuss QA output and results, incremental QA and procedures to formalize if the results are acceptable or not ▶ Determine the needs, and design the software, to access, visualize and interpret the results ▶ Define and develop the software to visualize data, raw and reconstructed ▶ Participate to the writing of the Technical Design Report and to the possible prototyping ▶ Run 2 ▶ Production data taking period -> coordinate maintenance and improvements of software ▶ Opportunity to test concepts and software for Run 3 4 Dataflow Monitoring Object Generation Automatic Quality Assessment Storage Visualization

Event Display Maintenance and development taken over by Warsaw group ▶ Jeremi Niedziela – PhD student at CERN ▶ Maciej Szymański – Service Task on ED ▶ Jakub Sala, Jakub Abelski, Adam Felis – Summer students ▶ Warsaw WUT for support ▶ Mihai – Offline, former ED developer People currently working directly on ED B. von Haller | CWG9 DQM-QA-Visualization |

HLT HLT ED Offline reco Event Display Event Display ▶ Run 2 requirements ▶ Split reconstruction and online ED ▶ Allow multiple sources and unify existing EDs ▶ Bookmarks ▶ Under development by J. Niedziela and M. Szymanski ▶ Work will continue over the summer ▶ Ready for the commissioning and cosmics Run 2 tasks B. von Haller | CWG9 DQM-QA-Visualization | HLT HLT ED Off. Reco ED HLT Off. Reco ED

Future work ▶ Event Display ▶ Alternative platforms (mostly web and/or mobile) ▶ Summer students from Faculty of Mathematics and Computer Science started working on it ▶ Longer term collaboration on this topic is possible in order to implement a full fledge event display ▶ Possible use of early alfa components ▶ Prototyping needed here ▶ DQM/QA ▶ Prototype of merging QA objects using map-reduce with ZeroMQ ▶ Web based ROOT objects browser B. von Haller | CWG9 DQM-QA-Visualization |

Future work – CWG10 ▶ Control and Configuration of hierarchical distributed systems ▶ Development of a zeroMQ-based prototype for configuration distribution and process control ▶ Evaluation of performance based on number of nodes, hierarchical levels, processes and configuration size ▶ Process Management ▶ Evaluation of technologies for process execution in sandbox/container (Docker, etc.) ▶ Evaluation of performance based on number of nodes, hierarchical levels, processes ▶ Evaluation of ZooKeeper for possible usage in O2 ▶ Configuration management ▶ Synchronization ▶ Process control Control & Configuration B. von Haller | CWG9 DQM-QA-Visualization |

Conclusion ▶ CWG9 is an important and dynamic working group in the O 2 project ▶ WUT is a key player in CWG9 ▶ In terms of people and responsibility ▶ Crucial for the future of Visualization in ALICE ▶ The ED is already profiting greatly from its involvement ▶ There are many more tasks and opportunities for collaboration in CWGs B. von Haller | CWG9 DQM-QA-Visualization |

10

Backups B. von Haller | CWG9 DQM-QA-Visualization |

▶ April 2014: First draft ▶ August 2014: Submission ▶ CWG9 participates to ▶ Chapter 4 System architecture : Quality control and assessment ▶ Table 1: detectors needs ▶ Explain DQM/QA architecture and the choices made ▶ Figure 1: DQM/QA architecture ▶ Explain Event Display architecture and the choices made ▶ Figure 2: Event Display architecture ▶ Chapter 5 Technology survey, evaluations and prototypes 1.Mergers architecture and feasibility tests with 0MQ 2.Results of storage tests (e.g. DB technologies) 3.[Web gui architecture (ROOT JS lib + DABC)] 4.Event display design as tested (cf Run 2) ▶ Chapter 6 System Design B. von Haller | CWG9 DQM-QA-Visualization | TDR and feasibility tests 12

Past and Current tasks ▶ Bring everyone aboard [done] ▶ Make people aware of others’ work in the field ▶ Give a picture of the current situation to everyone ▶ Run 2 ▶ Event Display review and meetings, Warsaw involvement ▶ Core refactoring ▶ New features ▶ Knowledge transfer  Gain stability and support for Run 2 ▶ DQM/QA review and preparation ▶ Proposal for the online reconstruction and calibration ▶ Run 3 ▶ System requirements and system functionalities document [done] ▶ Detectors needs survey ▶ Definition of the future architecture and design ▶ Prototypes and feasibility tests ▶ Technical Design Report redaction B. von Haller | CWG9 DQM-QA-Visualization |

O 2 Technical Design Report ▶ Schedule ▶ October ‘13: ▶ Define table of content ▶ Establish editorial board ▶ December ‘13: ▶ System Requirement Document ▶ High-level dataflow model ▶ Computing platforms benchmarks ▶ Networking benchmark ▶ June ‘14 ▶ Software framework architecture ▶ Sep ‘14 ▶ TDR 14 B. von Haller | CWG9 DQM-QA-Visualization |

CWG9 TDR Timeline ▶ January 14 ▶ Define list of tables and figures ▶ Draft of the architecture of the system ▶ Launch subsystems exhaustive survey  Submit list of tables and figures to TDR EC ▶ February 14 ▶ Draft tables and figures ▶ Skeleton of and 5.6  Submit skeleton to TDR EC ▶ March 14 ▶ Finalize tables and figures, including subsystems input ▶ Iterate on text using input of TDR EC  Submit text and final tables and figures to TDR EC ▶ April 14 ▶ Finalize text  Submit final text to TDR EC ▶ May 14 ▶ Iterate over our sections using CWGs input ▶ Review work of other CWGs (especially what concerns us!) Proposal B. von Haller | CWG9 DQM-QA-Visualization |

TDR – skeleton, tables & figures ▶ 4.2 Data processing and computing model ▶ DQM and QA ▶ « Quality control and assessment » ▶ Table 1: detectors needs ▶ Explain architecture ▶ Figure 1: architecture ▶ Explain the choices ▶ Figure 2: Event display arch. Chapter 4 System architecture B. von Haller | CWG9 DQM-QA-Visualization |

TDR – skeleton, tables & figures ▶ 5.6 DQM and QA ▶ « Quality control and assessment » ▶ Technologies and design choices available concerning key points of our system ▶ Storage ▶ Access to results worldwide ▶ Event display ▶ Feasability tests & prototypes ▶ Table 1: results of storage tests (e.g. DB technologies) ▶ Figure 1: Web gui architecture (ROOT JS lib + DABC) ▶ Figure 2: Event display design as tested (cf Run 2) Chapter 5 Technology survey, evaluations and prototypes B. von Haller | CWG9 DQM-QA-Visualization |

Subsystems survey ▶ What “tasks” (name it agents or algorithms if you prefer) will your subsystem need in Run 3 ? ▶ For each of these task or group of tasks, tell us ▶ Whether it already exist today and if so what is its performance. ▶ What is the expected performance of such a task in Run 3. ▶ How many plots are expected to be produced (for the shifter and for the experts). ▶ Percentage of events needed to carry out the task online (minimum, optimal). ▶ What is the input ? i.e. at which stage will it run ? ▶ How fast the response has to be taken into account in the data flow ? ▶ Whether the DQM/QA results have to become persistent and for how long ? ▶ What does “Calibration QA” mean to you ? B. von Haller | CWG9 DQM-QA-Visualization |

Survey status B. von Haller | CWG9 DQM-QA-Visualization | SubsysAckContact pers.Under disc.AnsweredCleared ACO CPV EMC FMDAbsent from Run 3 HMPDuring meeting MCH MTRSee MID PHOS PMD SDDSee ITS2 SPDSee ITS2 SSDSee ITS2 T0See FIT TOF TPC TRD V0See FIT ZDC Trigger DAQ MFT MID FIT ITS 2

Current and short term work ▶ Run 2 : Prepare a proposal for the QA in relation with the online reco, calibration and monitoring (QA tasks, validation, bridge to DQM, …) ▶ Online Calibration ▶ Mainly for TPC ▶ Many open questions on ▶ Requirements ▶ General architecture ▶ Implementation ▶ CDB ▶ Working on a proposal to meet requirements while minimizing work ▶ Use analysis QA train within HLT for reco monitoring ▶ Use analysis QA train within HLT for calib monitoring ▶ Use AMORE for raw data monitoring ▶ Use AMORE infrastructure for storage and visualization ▶ Run 2 – Review and preparation ▶ Detectors « interviews » ▶ (DATE Monitoring update) ▶ Run 3 : Prepare requirements of the future system following CWG1 input ▶ For the TDR (2014) ▶ Define requirements and general architecture and features of the QA-DQM-Viz for Run 3 ▶ Write it ▶ Event Display ▶ Decentralized model under implementation ▶ Better stability ▶ Split GUI and reconstruction ▶ Possibility to switch between offline and HLT reco ▶ Bookmarks (for users and for PR) ▶ Involvement of the Warsaw group ▶ Implemented by the end of 2013 B. von Haller | CWG9 DQM-QA-Visualization |

B. von Haller | CWG9 DQM-QA-Visualization | HCDB DCS, GRP Reco Selection/ filters Calib OCDB Data on Castor Reading Writing Producing DBs data procedures ESDs Raw Calib param PHYSICS Run QC (sort of)