May. 11, 2015 David Lawrence JLab Counting House Operations

Spring Commissioning Run Written to tape – 72 TB – 4297 EVIO files – 635 Runs (maybe TB of useful beam data for offline analysis)

Select runs runI solenoid I beam modestartkB/eventradiatorNevents A100nA :4128 kBJ1A5074.6M A45nA :2519 kB1 x M 31380A60-100nA85-1 4:49189 kB1 x M A70nA75-2 3:1733 kB1 x M A12nA :0728 kBJ1A5019.9M A10-20nA :5833 kBJ1A50*6.5M 32000A nA85-3 5:55224 kB1 x M * Run 3186 has ~13M events, but the magnet quenched about halfway through the run. The first 8 files (= 6.5M events) should be good.

12.4 kB/event I beam = 0 mode 7 BCAL/FCAL trigger 227 kB/event I beam = 180nA mode 8 FCAL/ST trigger run 3201 run 3205

Mar. 11, 2015 May. 4, 2015 Integrated disk usage for EVIO files only calculated from Nevents*avg. event size in datmon DB

32-bit word types in EVIO event

Run 2931 beam cosmics 74.6M events Diamond radiator Mode 7 (i.e. integrals only) Apparent beam burst while writing file 93 of 107 Data footprint changed after this… From EPICS archive via Myaviewer From EPICS events in EVIO files 120nA

Before incident After incident ALL fADC125 modules show same behavior CDC FDC

Run 3185: -Diamond radiator -1300A solenoid -10nA beam current -Mode M events Probably closest to GlueX production running conditions MONITORING FARM USAGE 100 CPU NETWORK

Well documented on wiki Some info automatically entered every run Most data from “Sean” DB inserted into RCDB DB in counting house is being replicated to DB outside of counting house Public web interface not quite live yet (Dmitry is still touching up some things) Work needed from others (e.g. me) to populate RCDB with things being written to other Run info. DB Run Conditions DB (RCDB)

“Sean” DB

“Dave” DB

“Dave” DB

Automatic E-log entries

CODA versions EPICS values for Beamline EPICS values for BCAL EPICS values for CDC EPICS values for FDC Trigger/DAQ config.

Improved Hall Operations Rewired console computers to UPS (thanks power outage!) Printer now works from CH computers RHEL7 tested with CSS JTabs issue identified and is being corrected (by accelerator) RAID Disk mapping and semiautomatic-deletion script tested Added BCAL_Eff, BCAL_inv_mass, HLDetectorTiming plugins to monitoring

Backup Slides Online Status -- David Lawrence20

Counting house computer systems Computer(s) processor General Purpose Network DAQ Network I.B. Network comments gluonfs1 N/A X ~1.6TB with snapshot backup gluonraid1-2 Intel E XXX RAID disk host ER process gluon01-05 X Shift taker consoles gluon20-23 AMD 2347 X Controls 8core gluon24-30 X Controls (gluon24 is web/DB/cMsg server) 12core + 12ht gluon40-43 AMD 6380 XXX 16core + 16”ht” gluon46-49 E XX (gluon47 &49) X 16core + 16ht gluon E XX 16core + 16ht rocdev1 Pentium X RHEL5 system for compiling ROLs for DAQ hdguest0-3 X (outside network) Guest consoles in cubicles (outside network) Online Status -- David Lawrence21

Run 2931 data words beam cosmics f125 block header f125 filler Places where red is higher are digitized values (others are headers, trailers, timestamps,etc…) 32-bit word types in EVIO event

Data Rates ROC Event Builder Event Recorder Tape Library ROC Spec: 100MB/sec Tested: ~30MB/sec Spec: 3000MB/sec Tested: 600MB/sec Spec: 300MB/sec Tested: 600MB/sec Spec: 300MB/sec Tested: 450MB/sec “Tested” means with actual data while it was being acquired. In some cases, offline testing has achieved significantly higher rates. 72TB x2 RAID disk Online Status -- David Lawrence23 (L3 farm) 125.9TB in 147,355 files written to tape in 2014 commissioning run

Mode 7 (fADC integrals) Mode 7 (full samples) fADC250 fADC125 fADC250 fADC250/F1TDCfADC125 Mode 7 (fADC integrals) Online Status -- David Lawrence24 FCAL BCAL FCAL BCAL FDC CDC fADC250/F1TDC

Mode 7 (fADC integrals) Mode 8 (fADC full samples) Online Status -- David Lawrence kB/event 69 kB/event

Online Monitoring Online Status -- David Lawrence26

Rough Specs. Review 10 8  /s on LH 2 target -> ~400kHz hadronic rate L1 trigger goal is to cut away ~50% leaving 200kHz L3 trigger goal is to reduce by ~90% leaving 20kHz Early simulation suggested ~15kB/event Design specs*: – 200 kHz = 3000 MB/s (front end) – L3 reduction by factor of 10 = 300MB/s to RAID disk – 3 days storage on RAID = 300MB/s*3days = 78TB – Maintain 300MB/s transfer from RAID to tape Online Status -- David Lawrence27 *L3 not officially part of 12GeV upgrade project

Mode 7 (fADC Integrals) Mode 8 (fADC full samples) Each 32bit word in the EVIO file tallied to identify what file space is being used for Comparison between mode 7 and mode 8 data made Example: some of the fADC250 word types Online Status -- David Lawrence28

Event Size Simulation was consistent with initial estimate of event size Actual data was more than x4 larger Much of the data was taken in “raw” mode where fADC samples were saved Online Status -- David Lawrence29

DAQ to Detector Translation Table The Translation Table is used to convert from DAQ system coordinates (rocid, slot, channel) into detector-specific coordinates (e.g. BCAL module, layer, sector, end) ~23k channels defined in SQLite DB file Stored in CCDB as XML string for offline analysis with complete history: – /Translation/DAQ2detector Online Status -- David Lawrence30

hdmon Monitoring Plugins Online Status -- David Lawrence31 BCAL_online CDC_online DAQ_online FCAL_onlineFDC_onlinePS_onlineST_onlineTAGH_onlineTAGM_onlineTOF_online Each detector system provides 1 or more plugins that create histograms for monitoring All plugins are attached to a Common DANA process (hdmon) A “rootspy” plugin publishes all histograms to the network rootspy

Raw Data Formatted Files (from simulated data) Online Status -- David Lawrence32 CCDB hdgeant_smeared.hddm run0002.evio (Data file in same format as will be produced by CODA DAQ system) roc002.evio roc003.evio roc004.evio......

L3 and monitoring architecture EB ER L3 and monitoring processes are decoupled. They could run on same nodes though if desired. gluon53gluonraid1 gluon46 (Data flows from left to right) farm manager Online Status -- David Lawrence33

hdmongui Online Status -- David Lawrence34 multiple “levels” supported processes run multi-threaded

Online Status -- David Lawrence35