1. HADES Event Builder Status HADES Coll. Meeting XXX, Lisboa
Jörn Adamczewski-Musch
GSI / RBEE

2. Overview Event Building (EB)
EB hardware status
EB software migration to DABC
EB monitoring: ICINGA, EPICS, HTTP,...
Outlook
J.Adamczewski-Musch - HADES Coll. Meeting XXX, Lissabon

3. DAQ/EB server overview
lxhadesdaq: (spare: lxhadesdaqold)
central nfs server (/home/hadaq, /var/diskless/etrax_fs),
DAQ and EB configuration and startup
Oracle gateway for DAQ/EB run infos (nfs client of EB servers)
lxhadeb05, lxhadeb02, lxhadeb03, lxhadeb04:
event builder servers: 4 EB processes/server -> 16 EB
12/16 core, 64GB, 14/22 x 2TB local disks, 2x 10Gb ethernet
lxhadeb06: (spare: lxhadeb07)
central monitoring server: ( EPICS, QA, ICINGA , several vnc servers)
32 (64) core, 128 (256)GB, 44TB RAID6
hades33: (no spare!?)
Hmon web server -, QA display
hades31: (no spare!?)
trbnet gateway to frontend controls (PEXOR board), trbcmd DAQOPSERVER
hades30, hadesdaq01, hadesdaq02, hades17,...:
interactive operator machines,
J.Adamczewski-Musch - HADES Coll. Meeting XXX, Lissabon

4. DAQ/EB server report
Recent maintenance:
lxhadesdaq: 3/ changed complete hardware (previous lxhadeb01) 8 core, 32GB
lxhadeb0i: 5/2014 – Linux upgrade Debian 7 wheezy
lxhadeb07: 3/2015 – exchanged complete RAM (apparently corrupt)
Recent problems:
lxhadeb06: 3/ RAID6 failure after power down -> replace disk, restore data ( >1 day without monitoring/slow control!)
hadesdaq01: 4/ replaced broken disk, rearranged RAID1
lxhadeb05: 8/2015 – temporarily hangs for 3 days (reason unclear!) -> power cycle
lxhadesdaq: 8/2015 – /home/hadaq file system error after power failure -> XFS repaired, recovered ( ~1 day without DAQ !)
No proper automatic startup of all machines and services at power on! (nfs mount interdependencies, power fuses, lost dhcp request,...)
J.Adamczewski-Musch - HADES Coll. Meeting XXX, Lissabon

5. DAQ/EB server plans To do before next beam time (2017 ?):
Linux system upgrades (next: Debian 8)
Improve automatic startup and service restarts
Avoid single points of failure!
duplicate central machines, distribute crucial services to different nodes
optimize power distribution (fuses, UPS?)
New hardware investments ?
replacement and spare PCs, spare harddisks, spare network components
Mind ageing of components!
J.Adamczewski-Musch - HADES Coll. Meeting XXX, Lissabon

6. HADES EBs before 2014 TRB RFIO TRB TRB TRB TRB HYDRA2 TRB TRB TRB TRB
hld
RFIO
/data0i
/lustre
Event builder
TRB
/shm
IOC
TRB
TRB
Event builder
hld
/shm
IOC
HYDRA2
TRB
small hld
TRB
Event builder
hld
/shm
server
TRB
IOC
TRB
client
Event builder
hld
TRB
/shm
IOC
TRB
EPICS GUI
J.Adamczewski-Musch - HADES Coll. Meeting XXX, Lissabon

7. HADES EBs since 2014 DABC TRB RFIO TRB TRB DABC TRB TRB HYDRA2 DABC
hld
RFIO
/data0i
/hera
output
Combiner
stream
TRB
http
/shm
TRB
DABC
hld
Combiner
output
stream
http
TRB
/shm
HYDRA2
TRB
DABC
hld
Combiner
output
stream
http
TRB
server
TRB
/shm
DABC
hld
Combiner
output
stream
http
TRB
client
TRB
/shm
TRB
EPICS GUI
J.Adamczewski-Musch - HADES Coll. Meeting XXX, Lissabon

8. EB software migration to DABC
Data Acquisition Backbone Core (
Linux, C++, multithreading, modular, extensable plug-ins, multi-node DAQ
DABC hadaq plugin :
UDP subevent receiver, event builder functionality
compatible hld file output to RFIO and disk
EPICS interface same as before, intermediate shared memory
writes same run info for ORACLE logbook
configurable by existing DAQ/EB scripts and setup files
QA monitoring events provided by „streamserver“ socket
HTTP server for EB monitoring and control
May optionally run pre-analysis processing (TDC calibration -> S. Linev‘s talk)
J.Adamczewski-Musch - HADES Coll. Meeting XXX, Lissabon

9. EB performance comparison
DABC performance (August 2014)
29 subsystems, 25 kHz pulser, 3 EB, RFIO and local disk file
=> 38 Mbyte/s per EB process, max CPU load 78%, no event loss
=> would achieve 608 Mbyte/s with all 16 EB processes (?)
typical pion beam data rate 2014: 9 Mbyte/s at <2kHz trigger rate)
netmem/evtbuild performance (August 2011)
28 subsystems, Au-beam trigger, 8 EB, RFIO and local disk file
=> typical 17 Mbyte/s per EB process without event loss
=> overall 135 Mbyte/s
J.Adamczewski-Musch - HADES Coll. Meeting XXX, Lissabon

10. EB monitoring and control
ICINGA
DABC web
EPICS
Hmon
Ipmi
J.Adamczewski-Musch - HADES Coll. Meeting XXX, Lissabon

11. EB monitoring and control
HMON (J. Michel et al)
Hades DAQ monitoring (perl, trbcmd, etc.)
http server (
Cluster monitoring: ICINGA
http server (
nodes status, services (oracle, disk, EPICS,...)
service restart handlers
EPICS (S. Yurevich, P. Zumbruch)
EB data monitoring (rates, triggers, flags,...)
EB run id synchronization! (start new hld files)
DABC HTTP server in EB processes
similar to new ROOT THttpServer (S.Linev)
generic GUI in any web browser
common access from monitoring scripts
Server hardware: IPMI
chassis, power,
remote „local console“
J.Adamczewski-Musch - HADES Coll. Meeting XXX, Lissabon

12. DABC web interface
J.Adamczewski-Musch - HADES Coll. Meeting XXX, Lissabon

13. footnote: ROOT web interface
development by B.Bellenot and S.Linev related to DABC web
see ROOT2015 workshop: JSROOT, THttpServer class
available in standard ROOT v5.34/32 and >v6.0x, no DABC required!
users: OpenData, Geant4, VISPA, ATLAS Tile Cal, ALICE DQM, TRIUMF, CernBox
perhaps also interesting for HADES QA applications?
J.Adamczewski-Musch - HADES Coll. Meeting XXX, Lissabon

14. Outlook
DAQ/EB/monitoring server hardware is still sufficient (but may need upgrades and more spare hardware for the future!)
EB software was moved to DABC framework
successful data taking in 2014 pion beamtime
DABC EB offers HTTP control server
replace EPICS interface via shm? replace EPICS for EB completely?
similar HTTP monitoring is available for standard ROOT analysis (->QM?)
DABC EB can do trb3-TDC online calibration before hld file output
probably change HADES event building network topology in the future!
see presentation of Sergei Linev
J.Adamczewski-Musch - HADES Coll. Meeting XXX, Lissabon