HADES Event Builder Status HADES Coll. Meeting XXX, Lisboa Jörn Adamczewski-Musch GSI / RBEE 7.10.2015

Overview Event Building (EB) EB hardware status EB software migration to DABC EB monitoring: ICINGA, EPICS, HTTP,... Outlook 07.10.2015 J.Adamczewski-Musch - HADES Coll. Meeting XXX, Lissabon

DAQ/EB server overview lxhadesdaq: (spare: lxhadesdaqold) central nfs server (/home/hadaq, /var/diskless/etrax_fs), DAQ and EB configuration and startup Oracle gateway for DAQ/EB run infos (nfs client of EB servers) lxhadeb05, lxhadeb02, lxhadeb03, lxhadeb04: event builder servers: 4 EB processes/server -> 16 EB 12/16 core, 64GB, 14/22 x 2TB local disks, 2x 10Gb ethernet lxhadeb06: (spare: lxhadeb07) central monitoring server: ( EPICS, QA, ICINGA , several vnc servers) 32 (64) core, 128 (256)GB, 44TB RAID6 hades33: (no spare!?) Hmon web server -, QA display hades31: (no spare!?) trbnet gateway to frontend controls (PEXOR board), trbcmd DAQOPSERVER hades30, hadesdaq01, hadesdaq02, hades17,...: interactive operator machines, 07.10.2015 J.Adamczewski-Musch - HADES Coll. Meeting XXX, Lissabon

DAQ/EB server report Recent maintenance: lxhadesdaq: 3/2014 - changed complete hardware (previous lxhadeb01) 8 core, 32GB lxhadeb0i: 5/2014 – Linux upgrade Debian 7 wheezy lxhadeb07: 3/2015 – exchanged complete RAM (apparently corrupt) Recent problems: lxhadeb06: 3/2015 - RAID6 failure after power down -> replace disk, restore data ( >1 day without monitoring/slow control!) hadesdaq01: 4/2015 - replaced broken disk, rearranged RAID1 lxhadeb05: 8/2015 – temporarily hangs for 3 days (reason unclear!) -> power cycle lxhadesdaq: 8/2015 – /home/hadaq file system error after power failure -> XFS repaired, recovered ( ~1 day without DAQ !) No proper automatic startup of all machines and services at power on! (nfs mount interdependencies, power fuses, lost dhcp request,...) 07.10.2015 J.Adamczewski-Musch - HADES Coll. Meeting XXX, Lissabon

DAQ/EB server plans To do before next beam time (2017 ?): Linux system upgrades (next: Debian 8) Improve automatic startup and service restarts Avoid single points of failure! duplicate central machines, distribute crucial services to different nodes optimize power distribution (fuses, UPS?) New hardware investments ? replacement and spare PCs, spare harddisks, spare network components Mind ageing of components! 07.10.2015 J.Adamczewski-Musch - HADES Coll. Meeting XXX, Lissabon

HADES EBs before 2014 TRB RFIO TRB TRB TRB TRB HYDRA2 TRB TRB TRB TRB hld RFIO /data0i /lustre Event builder TRB /shm IOC TRB TRB Event builder hld /shm IOC HYDRA2 TRB small hld TRB Event builder hld /shm server TRB IOC TRB client Event builder hld TRB /shm IOC TRB EPICS GUI 07.10.2015 J.Adamczewski-Musch - HADES Coll. Meeting XXX, Lissabon

HADES EBs since 2014 DABC TRB RFIO TRB TRB DABC TRB TRB HYDRA2 DABC hld RFIO /data0i /hera output Combiner stream TRB http /shm TRB DABC hld Combiner output stream http TRB /shm HYDRA2 TRB DABC hld Combiner output stream http TRB server TRB /shm DABC hld Combiner output stream http TRB client TRB /shm TRB EPICS GUI 07.10.2015 J.Adamczewski-Musch - HADES Coll. Meeting XXX, Lissabon

EB software migration to DABC Data Acquisition Backbone Core (http://dabc.gsi.de) Linux, C++, multithreading, modular, extensable plug-ins, multi-node DAQ DABC hadaq plugin : UDP subevent receiver, event builder functionality compatible hld file output to RFIO and disk EPICS interface same as before, intermediate shared memory writes same run info for ORACLE logbook configurable by existing DAQ/EB scripts and setup files QA monitoring events provided by „streamserver“ socket HTTP server for EB monitoring and control May optionally run pre-analysis processing (TDC calibration -> S. Linev‘s talk) 07.10.2015 J.Adamczewski-Musch - HADES Coll. Meeting XXX, Lissabon

EB performance comparison DABC performance (August 2014) 29 subsystems, 25 kHz pulser, 3 EB, RFIO and local disk file => 38 Mbyte/s per EB process, max CPU load 78%, no event loss => would achieve 608 Mbyte/s with all 16 EB processes (?) typical pion beam data rate 2014: 9 Mbyte/s at <2kHz trigger rate) netmem/evtbuild performance (August 2011) 28 subsystems, Au-beam trigger, 8 EB, RFIO and local disk file => typical 17 Mbyte/s per EB process without event loss => overall 135 Mbyte/s 07.10.2015 J.Adamczewski-Musch - HADES Coll. Meeting XXX, Lissabon

EB monitoring and control ICINGA DABC web EPICS Hmon Ipmi 07.10.2015 J.Adamczewski-Musch - HADES Coll. Meeting XXX, Lissabon

EB monitoring and control HMON (J. Michel et al) Hades DAQ monitoring (perl, trbcmd, etc.) http server (http://hades33/mon) Cluster monitoring: ICINGA http server (http://lxhadeb06/icinga) nodes status, services (oracle, disk, EPICS,...) service restart handlers EPICS (S. Yurevich, P. Zumbruch) EB data monitoring (rates, triggers, flags,...) EB run id synchronization! (start new hld files) DABC HTTP server in EB processes similar to new ROOT THttpServer (S.Linev) generic GUI in any web browser common access from monitoring scripts Server hardware: IPMI chassis, power, remote „local console“ 07.10.2015 J.Adamczewski-Musch - HADES Coll. Meeting XXX, Lissabon

DABC web interface 07.10.2015 J.Adamczewski-Musch - HADES Coll. Meeting XXX, Lissabon

footnote: ROOT web interface development by B.Bellenot and S.Linev related to DABC web see ROOT2015 workshop: JSROOT, THttpServer class available in standard ROOT v5.34/32 and >v6.0x, no DABC required! users: OpenData, Geant4, VISPA, ATLAS Tile Cal, ALICE DQM, TRIUMF, CernBox perhaps also interesting for HADES QA applications? 07.10.2015 J.Adamczewski-Musch - HADES Coll. Meeting XXX, Lissabon

Outlook DAQ/EB/monitoring server hardware is still sufficient (but may need upgrades and more spare hardware for the future!) EB software was moved to DABC framework successful data taking in 2014 pion beamtime DABC EB offers HTTP control server replace EPICS interface via shm? replace EPICS for EB completely? similar HTTP monitoring is available for standard ROOT analysis (->QM?) DABC EB can do trb3-TDC online calibration before hld file output probably change HADES event building network topology in the future! see presentation of Sergei Linev 07.10.2015 J.Adamczewski-Musch - HADES Coll. Meeting XXX, Lissabon