1. Kay Graf KM3NeT Computing and Software WG Meeting 2014/09/11 Overview

2. Topics for today ●activity reports ● DB status (CB) ● QA/QC forms (GA) ● Jpp (MdJ) ● aanet (AH) ● Restless Oyster & Introducing Rainbow Alga (TG) ●topics w/out reports: ● software strategy and its implementation ● discussion, feedback ● open action points ● software overview ● progress towards GRID ● offline data format ● Jpp status ● data processing for PPM-DU – asked for a meeting Comp-Soft-WG Meeting - 2014/09/11 - K. Graf2

3. Software Strategy ●workshop on 31 st of July at Erlangen – indico page: http://indico.cern.ch/event/332843/ http://indico.cern.ch/event/332843/ ●attendees: working group coordinators of the physics and software group and the main developers of the main KM3NeT software packages ●minutes: http://indico.cern.ch/event/332843/material/minutes/0.rtfhttp://indico.cern.ch/event/332843/material/minutes/0.rtf ●aim: define a common ground, from where a dependable, fast data processing chain can be set up and the necessary, missing software can be developed. Comp-Soft-WG Meeting - 2014/09/11 - K. Graf3

4. Software Strategy – Proposal (to be ratified by the PSC) decisions of the meeting are the following — and should act as strong proposal for the further line of software development: ●Accept and implement the dataformat-driven strategy (see KM3NeT_SOFT_WD_2014_003) for the KM3NeT offline software (both simulation and experiment data processing): KM3NeT_SOFT_WD_2014_003) KM3NeT will develop a data processing chain with well-defined interfaces (mainly i/o and configuration data formats) at each processing step. One official software package per step will be maintained (with alternatives for crosschecks), which can read/write the official data formats directly. Comp-Soft-WG Meeting - 2014/09/11 - K. Graf4

5. Software Strategy – Proposal (to be ratified by the PSC) ●Accept the following data formats as working solution — documented also in the wiki:wiki: ● global detector description format as proposed by AH ● evt data format for simulation (based on ANTARES-SOFT-1998-007 and ANTARES-SOFT-1999-003)ANTARES-SOFT-1998-007 ANTARES-SOFT-1999-003) ●raw- and online data format as defined by the DAQ group (see wiki page)wiki page) ●offline data format as proposed by AH (see this document)this document) Comp-Soft-WG Meeting - 2014/09/11 - K. Graf5 global_det_id ndoms \n dom_id line_id floor_id npmts \n pmt_id_global x y z dx dy dz t0 \n... pmt_id_global x y z dx dy dz t0 \n repeat for each dom

6. Software Strategy – Proposal (to be ratified by the PSC) ●Definitions: ● no software may change standard definitions (e.g. the standard coordinates that are to be defined (see action point)) ● the time of KM3NeT is UTC for all software (including DB) ● simulation data only contains pure photon/photoelectron information with possibly some merging (<1ns) - no smearing ● particle ID from PDG is used in evt file format ● no MC truth data is removed within the processing chain ● Documentation of all software packages to include doxygen (annotated sources) and a wiki page (small tutorial, programme parameters) Comp-Soft-WG Meeting - 2014/09/11 - K. Graf6

7. Software Strategy – Discussion ●asked for feedback on my mail from 4 th of August – none received (other than some modifications of the detector description format). ●Comments are highly welcome, we should discuss and prepare implementing these points in the next months - the final decision will be taken at the next steering committee meeting and on the next collaboration meeting. Comp-Soft-WG Meeting - 2014/09/11 - K. Graf7

8. Software Strategy – Open Action Points 1 ●Present offline data format to collaboration via mail beginning of September, for final approval during Paris collaboration meeting -> AH, KG ● ⇒ first a “running” version of aanet has to be finishedaanet ●Revise usage of antcc classes in Jpp (use offline classes directly) -> MdJ, RB ● ⇒ at a later stage ●Reiterate compatibility between online and offline data format -> MdJ, RB, AH ● ⇒ to be done (especially data types) ●Define proposal for the KM3NeT standard coordinate system -> TE ● ⇒ will be based on ANTARES, TE puts together ANTARES conventions currently ●Assemble information for all software used in KM3NeT, create matrix (used in which processing step, responsible person, status) -> KG, AT ● ⇒ see later ● thereafter: decide on standard packages and formalise processing chain Comp-Soft-WG Meeting - 2014/09/11 - K. Graf8

9. Software Strategy – Open Action Points 2 ●Create a set of standard events (cascades and muon bundles) at CC-Lyon (in iRODs /in2p3/km3net/mc/benchmark) (for benchmarking) -> AT ● ⇒ no feedback ●Change particle ID and detector coordinate change in km3 -> CJ ● ⇒ will be implemented in the next two weeks ●Implement detector description format in Jpp -> RB ● ⇒ done Comp-Soft-WG Meeting - 2014/09/11 - K. Graf9

10. Software Overview ●KM3NeT_SOFT_WD_2014_005KM3NeT_SOFT_WD_2014_005 ●one sheet: software description, one software matrix – see next slide ●missing: ● ORCA chain ● anything in the HE chain? Comp-Soft-WG Meeting - 2014/09/11 - K. Graf10

11. Comp-Soft-WG Meeting - 2014/09/11 - K. Graf11 Color Legend (all with respect to be used in the final processing chain with the proposed data formats): can be used (almost) as is to be modified not maintained/used Data FormatProcessing StepSoftware StatusSoftwar e StatusSoftwareStatusSoftwareStatusSoftwareStatus Chain/Software PackagePrevious Standard ChainAlternativesHOURSJppSeaTray Simulation Data FormatEvent GenerationGENHEN Corsika HOURSGen_G enie, HOURSGen_v7 simple generators included Particle GenerationMUPAGE Event Simulation and Cherenkov Light Generation km3 (in fact three programs: gen, hit, and km3mc, together the ‘km3 package’) KM3Sim JSirene clsim Raw Data FormatRaw Data Processing JTriggerProces sor,DataQueu e, JDataFilter, JDataWriter Online Data FormatBackground Light Generation MODK40(evt format) HOURSOmSim JModK40 single rates included Detector Response JTriggerEfficie ncy ToT included Data Filtering simple algorithms included Offline Data FormatData Calibration position calibrationANTARES Event Classification classification: RDFs, various hit selection i3 format Event Reconstructionreco(evt format) aashowe rfit track: aafit, bbfit, gridfit, krakefit, osffit, selectfit, scanfit (old), shower: q, dusj, energy: R, ANN, dEdx,, all up-to-date i3 format Analyses Outside data processing chain Quasi-online Processing Calibration Data Quality dataquality algorithms i3 format

12. GRID ●thanks to Christos and Vania, there has been progress towards data processing in the GRID ●unfortunately our proposal to EGI has been rejected – however with a new (EGI-encouraged) possibility next year ●EGI has started support nevertheless: ● access to the RAL cvmfs repo (/cvmfs/km3net.gridpp.ac.uk) ⇒ every cluster, data center or Grid Site that replicates that repo can use it ● Christos has install Jpp on that repository and distributed it as “VO_KM3NET_ORG_SW_DIR" at the GridSites supporting the KM3NeT.org VO (tested successfully JSirene on HellasGrid) ● we are invited to the EGI / Géant event Wednesday the 24th Sept. afternoon - a dedicated session on storage will happen there: ● http://indico.egi.eu/indico/conferenceDisplay.py?confId=2160 http://indico.egi.eu/indico/conferenceDisplay.py?confId=2160 ● https://indico.egi.eu/indico/sessionDisplay.py?sessionId=23&confId=2160# 20140924 https://indico.egi.eu/indico/sessionDisplay.py?sessionId=23&confId=2160# 20140924 ● are you interested to go there? Comp-Soft-WG Meeting - 2014/09/11 - K. Graf12

13. Offline data format ●Aart is developing aanet as KM3NeT offline data format (with manipulation classes), see this document):document ● following "aa" classes from Antares; break compatibility with ANTARES ● principles: KISS, YAGNI, less is more, zen of python, etc. ● very simple class hierarchy: you can look at a header file and immediately know what's in and Evt. ­ goal is that anyone (new students, seniors) can do real work as easily as possible. ● Use as much a possible general tools (c++, root, python). ­ learning new stuff is always hard, but at least these are good things to learn for all students. ● code is written with 3 uses in mind ● TTree::Draw ● pyroot/Cint scripting ● compiled c++ ●doxygen at http://antares.in2p3.fr/users/heijboer/aanetdoxhttp://antares.in2p3.fr/users/heijboer/aanetdox Comp-Soft-WG Meeting - 2014/09/11 - K. Graf13

14. Jpp status (MdJ, RB) ●implemented.detx detector format and conversion tools between formats (.txt is now incompatible – use r879 if you need that format!) ●implemented interfaces for adding alternative algorithms, e.g. for ToT- response simulation ●added example and data directory, to be run “as is” ● a selection will be used as test scripts during make ●new release tagged: ● see http://elog.km3net.de/Computing+and+Software/13http://elog.km3net.de/Computing+and+Software/13 Comp-Soft-WG Meeting - 2014/09/11 - K. Graf14

15. PPM-DU Data Processing ●all runs < run 448 (GPS-tagged) are available at iRODS: data/raw/PPM-DU/filter/2014 (note: no monthly subdir!) ●processed in Lyon using JPP in /afs/in2p3.fr/throng/km3net/src/Jpp/Jpp_install_tag_r879 ● converterv1Tov2 -d 1 -t GPS1 –u GPS2 –v GPS3 ● JTriggerProcessor -d 1 -T 1 -@ "writeL1=1" -@ "trigger3DShower.numberOfHits=1" -a /afs/in2p3.fr/home/g/graf/km3net_ppm_du_timeoffsets_corrected.txt -c 3 ● (note “1” instead of “true” is necessary for writeL1 even if it is a bool) ●now move to nightly processing at CP ● scripts are prepared but not tested yet Comp-Soft-WG Meeting - 2014/09/11 - K. Graf15

16. PPM-DU Data Processing ●missing: ● check.txt →.detx detector conversion is correct (results are identical): http://elog.km3net.de/Computing+and+Software/12 http://elog.km3net.de/Computing+and+Software/12 ● determination of effect of inter-DOM offsets ● define the final quasi-online processing parameters ● change the retrieval of GPS times from runlog.txt to runlog.xml (database xml upload format) Comp-Soft-WG Meeting - 2014/09/11 - K. Graf16