1. May. 11, 2015 David Lawrence JLab Counting House Operations

2. Spring Commissioning Run Written to tape – 72 TB – 4297 EVIO files – 635 Runs (maybe 10-20 TB of useful beam data for offline analysis)

3. Select runs runI solenoid I beam modestartkB/eventradiatorNevents 2931800A100nA74-27 1:4128 kBJ1A5074.6M 30791200A45nA84-30 1:2519 kB1 x 10 -4 25.9M 31380A60-100nA85-1 4:49189 kB1 x 10 -4 7.4M 31801300A70nA75-2 3:1733 kB1 x 10 -4 41.2M 31851300A12nA75-2 12:0728 kBJ1A5019.9M 31861300A10-20nA75-2 13:5833 kBJ1A50*6.5M 32000A100-200nA85-3 5:55224 kB1 x 10 -4 12.5M * Run 3186 has ~13M events, but the magnet quenched about halfway through the run. The first 8 files (= 6.5M events) should be good.

5. 12.4 kB/event I beam = 0 mode 7 BCAL/FCAL trigger 227 kB/event I beam = 180nA mode 8 FCAL/ST trigger run 3201 run 3205

6. Mar. 11, 2015 May. 4, 2015 Integrated disk usage for EVIO files only calculated from Nevents*avg. event size in datmon DB

7. 32-bit word types in EVIO event

8. Run 2931 beam cosmics 74.6M events Diamond radiator Solenoid @800A Mode 7 (i.e. integrals only) Apparent beam burst while writing file 93 of 107 Data footprint changed after this… From EPICS archive via Myaviewer From EPICS events in EVIO files 120nA

9. Before incident After incident ALL fADC125 modules show same behavior CDC FDC

10. Run 3185: -Diamond radiator -1300A solenoid -10nA beam current -Mode 7 -19.9M events Probably closest to GlueX production running conditions MONITORING FARM USAGE 100 CPU NETWORK

11. Well documented on wiki Some info automatically entered every run Most data from “Sean” DB inserted into RCDB DB in counting house is being replicated to DB outside of counting house Public web interface not quite live yet (Dmitry is still touching up some things) Work needed from others (e.g. me) to populate RCDB with things being written to other Run info. DB Run Conditions DB (RCDB)

12. “Sean” DB https://halldweb.jlab.org/cgi-bin/data_monitoring/run_conditions.pl

13. “Dave” DB https://www.jlab.org/Hall-D/test/RunInfo/

14. “Dave” DB https://www.jlab.org/Hall-D/test/RunInfo/

15. Automatic E-log entries

18. CODA versions EPICS values for Beamline EPICS values for BCAL EPICS values for CDC EPICS values for FDC Trigger/DAQ config.

19. Improved Hall Operations Rewired console computers to UPS (thanks power outage!) Printer now works from CH computers RHEL7 tested with CSS JTabs issue identified and is being corrected (by accelerator) RAID Disk mapping and semiautomatic-deletion script tested Added BCAL_Eff, BCAL_inv_mass, HLDetectorTiming plugins to monitoring

20. Backup Slides Online Status -- David Lawrence20

21. Counting house computer systems Computer(s) processor General Purpose Network DAQ Network I.B. Network comments gluonfs1 N/A X ~1.6TB with snapshot backup gluonraid1-2 Intel E5-2630 v2 @2.6GHz XXX RAID disk host ER process gluon01-05 i5-3570 @3.4GHz X Shift taker consoles gluon20-23 AMD 2347 X Controls 8core gluon24-30 E5-2420 @1.9GHz X Controls (gluon24 is web/DB/cMsg server) 12core + 12ht gluon40-43 AMD 6380 XXX 16core + 16”ht” gluon46-49 E5-2650 v2 @2.6GHz XX (gluon47 &49) X 16core + 16ht gluon100-111 E5-2650 v2 @2.6GHz XX 16core + 16ht rocdev1 Pentium 4 @2.8GHz X RHEL5 system for compiling ROLs for DAQ hdguest0-3 i5-3470 @3.2GHz X (outside network) Guest consoles in cubicles (outside network) Online Status -- David Lawrence21

22. Run 2931 data words beam cosmics f125 block header f125 filler Places where red is higher are digitized values (others are headers, trailers, timestamps,etc…) 32-bit word types in EVIO event

23. Data Rates ROC Event Builder Event Recorder Tape Library ROC Spec: 100MB/sec Tested: ~30MB/sec Spec: 3000MB/sec Tested: 600MB/sec Spec: 300MB/sec Tested: 600MB/sec Spec: 300MB/sec Tested: 450MB/sec “Tested” means with actual data while it was being acquired. In some cases, offline testing has achieved significantly higher rates. 72TB x2 RAID disk Online Status -- David Lawrence23 (L3 farm) 125.9TB in 147,355 files written to tape in 2014 commissioning run

24. Mode 7 (fADC integrals) Mode 7 (full samples) fADC250 fADC125 fADC250 fADC250/F1TDCfADC125 Mode 7 (fADC integrals) Online Status -- David Lawrence24 FCAL BCAL FCAL BCAL FDC CDC fADC250/F1TDC

25. Mode 7 (fADC integrals) Mode 8 (fADC full samples) Online Status -- David Lawrence25 232 kB/event 69 kB/event

26. Online Monitoring Online Status -- David Lawrence26

27. Rough Specs. Review 10 8  /s on LH 2 target -> ~400kHz hadronic rate L1 trigger goal is to cut away ~50% leaving 200kHz L3 trigger goal is to reduce by ~90% leaving 20kHz Early simulation suggested ~15kB/event Design specs*: – 15kB/event @ 200 kHz = 3000 MB/s (front end) – L3 reduction by factor of 10 = 300MB/s to RAID disk – 3 days storage on RAID = 300MB/s*3days = 78TB – Maintain 300MB/s transfer from RAID to tape Online Status -- David Lawrence27 *L3 not officially part of 12GeV upgrade project

28. Mode 7 (fADC Integrals) Mode 8 (fADC full samples) Each 32bit word in the EVIO file tallied to identify what file space is being used for Comparison between mode 7 and mode 8 data made Example: some of the fADC250 word types Online Status -- David Lawrence28

29. Event Size Simulation was consistent with initial estimate of event size Actual data was more than x4 larger Much of the data was taken in “raw” mode where fADC samples were saved Online Status -- David Lawrence29

30. DAQ to Detector Translation Table The Translation Table is used to convert from DAQ system coordinates (rocid, slot, channel) into detector-specific coordinates (e.g. BCAL module, layer, sector, end) ~23k channels defined in SQLite DB file Stored in CCDB as XML string for offline analysis with complete history: – /Translation/DAQ2detector Online Status -- David Lawrence30

31. hdmon Monitoring Plugins Online Status -- David Lawrence31 BCAL_online CDC_online DAQ_online FCAL_onlineFDC_onlinePS_onlineST_onlineTAGH_onlineTAGM_onlineTOF_online Each detector system provides 1 or more plugins that create histograms for monitoring All plugins are attached to a Common DANA process (hdmon) A “rootspy” plugin publishes all histograms to the network rootspy

32. Raw Data Formatted Files (from simulated data) Online Status -- David Lawrence32 CCDB hdgeant_smeared.hddm run0002.evio (Data file in same format as will be produced by CODA DAQ system) roc002.evio roc003.evio roc004.evio......

33. L3 and monitoring architecture EB ER L3 and monitoring processes are decoupled. They could run on same nodes though if desired. gluon53gluonraid1 gluon46 (Data flows from left to right) farm manager Online Status -- David Lawrence33

34. hdmongui Online Status -- David Lawrence34 multiple “levels” supported processes run multi-threaded

35. Online Status -- David Lawrence35