Offline Status Review M. Moulson, P. Valente, for the Offline Group 16 March 2001 Outline: Status update FILFO: new developments (G. Finocchiaro) Questions to discuss MC production on Linux (P. Valente)

Processing of fall 2000 data Reference data set: Runs 15197—17330 (23 Sep—11 Dec) 5.61G triggers, files, 13.6 TB raw data Offline  L dt = 23.1 pb -1 DBV-6: 5.5 pb -1 DBV-7: 18.0 pb -1 Total events collected: 130M Bhabhas 67M f decays est. s f = 2.9 m b 10.9M K L tags 7.2M K L crash 19.5M K + K - w/ vertex 6.2M rp tags

Event streaming 2050 Hz 1650 Hz raw EmC recon. DC recon. Evt. Class bha kpm ksl rpi rad clb flt afl 400 Hz 12 Hz Evt. Class cos prescaled cosmic 40 Hz MB cosmic ÷10 ÷100 Bhahba 40 Hz 50 Hz 450 Hz Typical raw data composition, Nov-Dec ’00  L  = ~ 9.5  cm -2 s -1  Tot. rate ~2050 Hz Unvetoed cosmics~ 700 Hz Prescaled cosmics~ 400 Hz Mach. Bkg + Bhabha < 20°~ 900 Hz Physics ( f + Bhabha > 20°) < 100 Hz

Processing statistics for fall 2000 data Raw data:5605M Events, 13.6 TB Physics:2.0% of total events 25% of raw volume 160 GB/pb -1 Bhabhas:2.3% of total events 20% of raw volume 114 GB/pb -1 StreamEvents (M)Volume (GB)Summary kpm ksl rpi rad clb M Events 3.6 TB bha flt afl cos M Events 4.2 TB Avg. fraction of events tracked: 26% Avg. bandwidth on 40 CPU’s: 1060 Hz Avg. time/trigger 38 ms Avg. time/tracked evt. 151 ms

datarec performance in fall 2000 Daily averages Average uptime efficiency: 63% Staging and procedural losses count Evts. acquiredEvts. processed Once in stable situation, able to follow with ~75% of data processed No DC trig End of data Follow-up After Xmas Missed runs Reprocessing Acquistion rate Reconstruction rate Farm bandwidth 23 Sep 11 Dec

Brief history of datarec executable DBV-4 First version of datarec with all tracking upgrades complete Used for 1999 MC production DBV-5 Event classification sub-flags modified Problem discovered: All events with 2 p  and >1 g thrown away DBV-6 DBV-5 bug fixed Sub-flags for rpi, clb streams modified Very large angle Bhabha counter added Problem discovered: FILFO Bhabha filter causing losses in neutral radiative channels DBV-7 DBV-6 condition for running FILFO Bhabha filter fixed LAB selection cuts modified (E max )

New version of datarec executable Already in DBV-8: CLUFIXENE Now runs on MC events Calibration factor adjusted EMCALIB Slewing corrections added VTXFIN Path length corrected Other minor bugs fixed (Almost) ready for inclusion in datarec: FILFO Fast tracking cuts N hits definition excludes pileup A_Czlib event compression FILFO gg counter added Fills ECLS for rejected events Can write ECLS w/o ECLMOD TRCLMOD Track-cluster projection fixed ECLMODK S semileptonic tag added rp tag modified New ECLS bank version VLABNew luminosity module

New version of GEANFI GEANFI version 1.06/07 released on 6 Feb: EVA generator inserted p + p - g generator with different models for ISR, FSR, interference with f 0 g w  hg decay inserted h  3g decay (C-violating) inserted Improved simulation of K   3 p decays ·f  rp+3p generator added parameter to specify fraction of decays that don’t proceed via rp Modifications to GEANT tracking to allow generation of specific K L topologies K L decaysin beam pipe before entering DC before entering calorimeter in calorimeter Change name of output banks: DTCE  MDTC, CELE  MCEL Makes it easier to re-reconstruct MC files that have already been reconstructed New default value for magnetic field: I = 2300 A

Status of MC production Event typeRequestTodayAdded f  all 2.4M f  K S K L K S  pp ; K L  various K S  p + p - ; K L  all (1.1M) 2.9M4.8M Mostly K S  p + p -, K L  3p 0 Some K S  p + p -, K L  p + p anywhere K S  semileptonic, K L  all   K + K - K ±  pmn, p e n 0.24M radiative channels f  p 0 g, hg, hg, f 0 g, a 0 g e + e -  m + m - g, p + p - g, wp 0 2.0M2.6M Mostly ppg events calibration channels e + e -  e + e - e + e -  m + m - cosmic rays 0.8M0 Total~8.3M10.0M MC production for 2000 data started 9 Feb

What needs reprocessing? Reprocess DBV-6 (5.5 pb -1 ) for neutral radiatives? FILFO Bhabha filter deleterious to neutral radiative events Rejects 40% of f  hg  ggg events Pre DBV-7: Only runs on “charged” events Charged event: N hits > 30 Pileup contribution to N hits in 2000 data DBV-7: Bhabha filter vetoed by neutral radiative tag instead of N hits cut Luminosity to 1% only possible with > DBV-7 reconstruction from raw Reprocess from raw w/o tracking Retain only rad, bha streams Reprocess DBV-6, 7 (23.1 pb-1) for tracking updates? Track length, TCA projection corrections can be handled at analysis stage

Wait on DC calibrations for datarec? s-t constants sensitive to pressure Calibration in 2000 when residuals > 20 m m Mean frequency of calibration: 1/week Observation: Large changes (~100 m m) in residual distributions observed only after hardware interventions Proposal: 50 m m < residuals < 100 m m Estimated loss in e VTX = 1% Warning to calibration operators Reconstruction not stopped Residuals > 100 m m Stop reconstruction Wait for calibration Essentially protects against human error Atmospheric Pressure

Wait on EmC calibrations for datarec? Two regimes: 100 nb -1 in ~hours: Easy to wait, don’t have to wait long! When calibrations not available, plenty of data on tape to reconstruct Requires stable & reliable calibration processes t fluctuations > t calibration : does it matter? 100 nb -1 in ~days: May have to wait a long time! When calibrations arrive, they have limited validity: t fluctuations < t calibration These runs typically not used for analysis Intermediate regime: Could be where waiting matters most Available flags: CT0GT RF, DT cable gg events, every run CABTfine D t 0 adjust by column gg events, every 100 nb -1 CABEAbs. E scale/column equalization Bhabha + gg events, every 100 nb -1 Other considerations: Observation from last 4 weeks of 2000 data: Use of calibrations from previous 100 nb -1 usually resulted in D t 0 within tolerance EmC reconstruction gets redone for Ntuples and DST’s in any case Currently evaluating impact on streaming

Retain flt and afl streams? flt stream (FILFO-selected events) Adds ~25% to raw data volume (3.8 TB on top of 13.6 TB) Has not been seriously used to date Potentially useful for recovery from tracking problems Possible solution: delete flt files after the fact afl stream (1% of FILFO-rejected events) Has not been seriously used to date 75% of raw events are FILFO-rejected, FILFO is fast Negligible benefit to keeping afl stream in datarec chain

How to handle prescaled cosmics? Trigger cosmic veto known to cause ~50% inefficiency for ppg events with high M 2 ( pp ) 1/5 of vetoed cosmics retained by trigger to study bias (“prescaled cosmics”) Current situation 10% of prescaled cosmics retained as raw events in cos stream Same 10% passed through reconstruction chain, ppg events end up in rad stream New proposal New code (P. Branchini) selects cosmic vetoes that could be ppg events Efficiency: No losses on ~1000 cos events that pass ppg filter Rejection: 93% rejection of uncut cos event sample Incorporate in selcos module for an intelligent software downscale Entire sample of cos ppg events ends up in rad stream Useful to retain unbiased cos stream for other analyses? Effect of cosmic veto on K + K - < 1% (M. Martemianov, Kaon WG) Expected effect for K S K L events minimal p  spectrum of p ’s that trigger comsic veto dies out below 300 MeV

What information do we keep in DB2? Counters:VLAB, Gold Bhabha, gg, K S  p + p -, K S  p 0 p 0, K L crash 20 fields K + K -, rp, Neutral Radiative, Rejected Bhabha, cosmic, MB Job table+8 unused fields Fitted quantities:  s from Bhabhas, E tot from EmC, T - L/c from Bhabhas 20 fields M(K S  p + p - ), s M (K S  p + p - ) Job table+15 unused fields Status flags:Biases: Indicates any reconstruction errors that cause biases in stream 2 flags per stream Obsolete: Indicates if file has been superseded by a new reconstruction datarec file table Encoding of status flags: Word of bits independently set to indicate reasons for bias/motive for reprocessing Calibrations (DC, EmC t, EmC E) Bugs (FILFO, clustering, tracking, each ECL filter, etc.) No bits set means stream good to use for analysis Example KID query: input url "dbdatarec:stream_code=’rad’ and run_nr between and and Biases=0 and Obsolete=0"

What is needed for MC production off-site? Obstacles to off-site production: Inability to read-write connect to DB from outside LNF Read GEANFI cards from DB2 Update processing/processed flags, random seeds Keep track of files produced Relatively easy to enable outside access for individual nodes on request Scripts that coordinate generation/reconstruction designed for LNF setup Need modifications and testing to run at remote sites Need mechanism for archival output files

Proposal for mini-DST production Mini-DST’s to be produced: in dedicated production pass in zlib-compressed YBOS format with a bank subset essential for physics analysis, including summary banks for different event classification substreams Status: zlib compressionDone ScriptsSome work needed Retained banksNeeds definition Summary banksNeeds definition, coding SubstreamingNeeds definition Waiting on physics input—Technical aspects relatively simple

DST substreams: initial proposal Initial proposal based on ksl Ntuple, radiant substreams, datarec streams, and EVCL tags ks2piK S  p + p -, K S  p 0 p 0 ksemilK S semileptonic decays klcrK L crash kl2p0K L  p 0 p 0 kl3p0K L  p 0 p 0 p 0 klpppK L  p + p - p 0 intertaginterferometry tags kpmkpm (K + K - ) stream rpirpi ( rp ) stream neu_max4gneutral events, at most 4 prompt g ’s neu_min5gneutral events, at least 5 prompt g ’s cha_min3g2 tracks, 3 or more prompt g ’s cha_ppg p + p - g tagged events clbclb ( m + m -, p + p - ) stream

DST banklist: initial proposal Raw Banks (?) 2 KB/evt w/w/o compression Allow recovery of information not in DST LRID, EVCL Event headers T0GL Global t0 TRIG (TPIE, TMON, TCAF, MOKA) Trigger summaryBased on Ntuple block, loosely discussed CTRG Sector-to-cluster CSPS (CELE, CWRK) Calorimeter hitsAlmost ready CLUS (CLPS, CLLS) Calorimeter clustersPlanned QCAL (QCAE, QWRK) QCAL hitsBased on Ntuple, CSPS; new to this proposal DC hit counts Based on Ntuple, new to this proposal DTFS w/o hit lists Fitted tracksNew to this proposal DVFS Fitted vertices TCLO Track-to-cluster ECLS, ECLO Event classification KNVO, VNVO, INVO Neutral vertices

Mini-DST size estimate Example mini-DST size estimates prepared with help of S. Miscetti ksl stream: K S  p + p -, K L  3p 0 events selected by S. DiFalco Original event size:34.4 KB CPTEST3:11.4 KB +gzip7.3 KB +dropping raw~5.5 KB Typical DST event size for kpm, ksl substreams: ~7 KB Total reduction factor not accounting for event selection: ~5 rad stream: substreams from radiant Original event size~23 KB cha_min3g7.6 KB (+gzip  4.9 KB) neu_max4g3.6 KB neu_min5g4.0 KB Typical DST event size for rad subtreams: 3-5 KB Total reduction factor not accounting for event selection: ~5 Estimated zlib compression factor: ~1.6