Q. He, K.T. McDonald  BooNe Electronics/DAQ Meeting Dec. 9,  BooNE Comments on  BooNE DAQ Challenges Qing He, Kirk T. McDonald Princeton University (November 21, 2008; updated December 8, 2008)

Q. He, K.T. McDonald  BooNe Electronics/DAQ Meeting Dec. 9,  BooNE The Analysis Challenge for a Liquid Argon TPC A liquid argon TPC is an “electronic bubble chamber”. However, no bubble chamber data was ever reconstructed by computers alone. LAr TPC data: Proton beam tracks in ICARUS T10 (1995); Cosmic ray tracks in ICARUS T300 (2000); Cosmic ray tracks in Argoneut (2008). No computerized reconstruction of tracks to date (that we are aware of). Neutrino events are more complicated that beam/cosmic ray tracks: Primary vertex; secondary vertices for  0 showers. The full advantage of LAr TPCs will only be realized if these more complex event topologies are successfully analyzed by computers.  BooNE will provide the first data with which the needed computer algorithms can be tested.  Risky to make cuts (data deletions) in a DAQ system that might compromise the effectiveness of the yet-to-be written reconstruction algorithms.  Wise to record all track information for potential neutrino interactions.

Q. He, K.T. McDonald  BooNe Electronics/DAQ Meeting Dec. 9,  BooNE Some LArTPC Facts for  BooNE Drift velocity v ~ 1.6 mm/  500 V/cm. Drift distance = 2.6 m; drift time = 1.6 ms. Active LAr volume = 2.6 m (x) x 2.6 m (y) x 12 m (z) = 81.1 m 3. 3 wire planes: U, V, X (= vertical wires) with U, V at  =  60° to the vertical. Wire length = 2.6 m for X, and 5.2 m for U/V (except for shorter U/V wires at detector ends). Nominal wire spacing d = 3 mm. Total number of wires = 12 [1/d + 2/(d / cos  )] [2 / (d / sin  )] = 28.5 / d = 9,500. [4000 X wires, 2250 each for U and V.] The induced (bipolar) signal waveforms have characteristic width  t ~ d/v ~ 2  s. (99% of frequency content below 2 MHz). Nominal time sampling is 5 MHz  10 samples per characteristic width. [Is this enough/too much?] Number of time samples = 8,000 per wire during drift time of 1.6 ms. Argon purity: Nominal electron lifetime of 1.5 ms  only 1/3 of original signal left after 2.6 m drift. Horizontal minimum ionizing track, 2.6 m from readout  ~ 6000 electron signal. Electronic noise on a 5-m-long wire (with cold front-end transistor near end of wire) ~ 500 e. S/N only 12:1 for long drift (36:1 for horizontal track close to readout plane). ADC bits: 1 (bipolar) + 5 (MIP close in) + 5 (slow protons/multiple tracks up to 32 x MIP) = 11 (i.e., 12).

Q. He, K.T. McDonald  BooNe Electronics/DAQ Meeting Dec. 9,  BooNE Cosmic Ray Rates in  BooNE [Refer to DocDB #198 for “theory”.] R H = 200 /m 2 /s of cosmic rays crossing a horizontal plane at the Earth’s surface. R V = R H /  = 63 /m 2 /s of cosmic rays crossing a vertical plane (from one side) at the Earth’s surface. Active volume V = 2.6 m (x) x 2.6 m (y) x 12 m (z) = 81.1 m 3. Total rate of muons entering this volume is R total = 11,000 / s (1 muon every 90  s).  1.1 e4 x 1.6 e-3 = 18 muons cross the detector during one readout time of 1.6 ms. The average path length of a muon crossing this volume is 2.2 m. The average projection of this path onto the long (z) axis of the TPC is about 1.1 m,  1.1 / = 367 vertical (X) wires “struck” by a muon on average. The signal from a wire from a single particle is significant over about  1 cm along the drift direction (see Argoneut events),  ~ 70 readout 5 MHz.  367 x 70 = 2.6 e4 pixels occupied by each muon, on average. Total number of X pixels during one readout time is 4000 x 8000 = 3.2 e7.  Fraction of X pixels occupied during one readout time = 18 x 2.6 e4 / 3.2 e7 = 1.5%. Occupancy of U and V pixels is about double that of X pixels, ~ 3%. Average occupancy of pixels by muons is ~ 2.5%. [An overburden of 20 feet of dirt would cut the occupancy in half, but 10’ of dirt only cuts rate by 20%. The liquid argon is also self shielding to some extent, so the above estimate is a bit high.]

Q. He, K.T. McDonald  BooNe Electronics/DAQ Meeting Dec. 9,  BooNE Triggers 1.Beam trigger. 67 ms between beam pulses; 15 Hz maximum trigger rate, < 10 Hz average. Booster beam spill ~ 1.6  s long, so an LAr scintillation trigger would fire only 1.6/90 = 1.7% due to cosmics. Rate of neutrino interactions with > 100 MeV energy ~ 1 every 10 min for 5e12 protons per booster pulse. 2.“Random” trigger for diagnostic purposes. Most beam triggers will not contain neutrino events, even if use scintillator trigger, so most beam triggers are in effect “random” triggers. For diagnostics, record “all” data. 3.Supernova trigger. Given an external trigger from some supernova watch, record “all” data in the detector for the next ~ 5 sec.

Q. He, K.T. McDonald  BooNe Electronics/DAQ Meeting Dec. 9,  BooNE Overview of Data Rates 9500 wires sampled at 5 MHz x 2 bytes/sample  100 GB/s “raw” data. [ATLAS data rate is 160 GB/s after their Level 1 trigger.] Data storage on 500 GB hard drives ($100 each) If record all “raw” data, will fill a 500 GB hard drive every 5 s,  Operating cost of $20/s, or $200M per year of 1e7 sec. To have operating cost of ~ $1M/year for hard drives, need to reduce data rate by ~ 200. Best to do this without deleting data within an “event” (= data relevant to a neutrino interaction or other physics signal),  Use “lossless” coding of data, which may reduce data rate by ~ 10;  Reduce “live time” of the experiment by factor of ~ 20.

Q. He, K.T. McDonald  BooNe Electronics/DAQ Meeting Dec. 9,  BooNE What Data Can/Should We Record? 9500 wires x 8000 times samples over full drift length x 2 bytes/sample  160 MB / trigger. Concern about effects of partial tracks from earlier muons  also record data from up to 1 drift time earlier and 1 drift time later,  480 MB / trigger during 4.8 ms. [100 GB/s to read everything.] Can use lossless (Huffman = gzip) encoding to reduce data size by ~ 10  48 MB / trigger. [With ~ 3% occupancy, barely pays to do “zero suppression” compared to Huffman coding.] If transfer this data “on the fly”, instantaneous rate = 48 MB / 4.8 ms = 10 GB/s If buffer data on detector and read out over 67 ms, rate = 48 MB / 67 ms = 720 MB/s. If 64 wires per readout channel, have 150 channels (front-end readout boards), with readout rates of 67 MB/s per channel “on the fly” (or 4.8 MB/s per channel if over 67 ms). If record all beam 10 Hz average with Huffman coding, have average data rate of 480 MB /s. [Fill a 500 GB hard drive in 15 min, if no further data processing.] If only record beam triggers with prompt LAr scintillation, average date rate is 8.2 MB/s. [Fill 500 GB hard drive every 17 hours, if no further data processing.] For supernova trigger, buffer 5 sec of data on the detector. With Huffman coding, this is 50 GB. If stored in 150 readout board memories, each is only 1/3 GB  single 512 MB chip per board. [If 8GB/board is practical limit, could store 80 sec of data in the board memories.] There is no serious technical challenge to record “all” data for all beam triggers, and for occasional supernova triggers as well. And (personal opinion), should record only beam triggers with prompt LAr scintillation to cut total data by ~ 50. [Would still have ~ 50 muons/event, 1 event every 5 sec,  10 muons/sec, i.e., 1e7/year.]

Q. He, K.T. McDonald  BooNe Electronics/DAQ Meeting Dec. 9,  BooNE Event Building: Switch + PC Farm If combine the data from 150 channels (readout boards) into events “on the fly”, need a switch that can handle rates of 10 GB/s. This is doable, but at a visible cost. If buffer data on detector and read out over 67 ms, need only a 720 MB/s switch (whose cost would hardly be visible). Option: Do event building in stages, with a small PC farm at the intermediate stage. This farm could (perhaps) makes cuts based on information from partial events to reduce the data size by ~ factor of 10. [No algorithm to do this has been suggested, and it is not so easy to imagine what it would be.] After the main switch, have full events available to individual PCs, and the main data analysis can begin. Personal view: there is no harm in organizing the switch in 2 stages with a small PC farm in between, but there is also no obvious gain in doing this.

Q. He, K.T. McDonald  BooNe Electronics/DAQ Meeting Dec. 9,  BooNE PMT DAQ Issues The LAr TPC will include a set of (cryogenic) PMTs to observe the (copious, but UV) scintillation light from ionization of liquid argon. Scintillation associated with a neutrino interaction will come within a few-ns interval during the 1.6-  s booster beam spill (possibly followed by a delayed signal of MEV energy from capture of thermalized neutrons resulting from the -Ar interaction).  While the trigger could be an analog energy sum from the PMT array, it will be good to have continuous digitization of the PMT signals, record during the entire 4.8-ms readout time of an event. This would permit detailed studies of the PMT trigger efficiency using muons whose energy deposition is separately determined by collection of the ionization electrons in the TPC. 50  /trigger  1/5 trigger/sec  10  /s  10 4  every 20 min; enough for a 1% study. A DAQ system for this is straightforward, but has been little discussed to date.

Q. He, K.T. McDonald  BooNe Electronics/DAQ Meeting Dec. 9,  BooNE Summary Recommendations Primary trigger = LAr scintillation during the 1.6-  s booster spill  trigger rate ~ 0.2 Hz. Only ~ 1% of these triggers will include neutrino interactions. Record all LAr TPC data (and PMT data) in lossless, compressed format during 4.8 ms for each trigger,  Average data rate  8 MB/s  1 PB/year (if ADC gain such that rms noise  1 count). Store all data in compressed format in 8-GB memory per front-end boards for 60 sec; Read out if desired within 60 sec as per supernova or other special trigger. Assemble data that is read out into events via a fast switch. No need for intermediate-level PCs to reduce data rate further online. No need for an external muon tagger/veto. The TPC (+ internal PMTs) will eventually tell us more than we want to know about the muons.

Q. He, K.T. McDonald  BooNe Electronics/DAQ Meeting Dec. 9,  BooNE Appendix: How Much Online Data Analysis Should We Do? Online data analysis  Deletion of some event data. Not prudent in  BooNE, which will be the first experiment ever to reconstruct neutrino interactions in a LAr TPC, for which the algorithms do not yet exist. A goal of  BooNE is to develop algorithms that could be applied to future LAr TPC experiments to delete data online while retaining excellent efficiency for reconstruction of physics events. This goal can and should be achieved via the  BooNE offline data analysis. Major analysis challenge is to deal with primary and secondary vertices, near which the data for individual wires is complex, and not well characterized by present, single track LAr TPC data. The signal for a single “hit” extends for ~ 2 cm along the drift direction, so whenever 2 tracks come within 2 cm of one another, the single wire data has forms not yet understood. Simple online deletion of data would be based on information from individual wires (“zero suppression” and “hit parameterization”), which would compromise vertex reconstruction in ways not presently known. Bottom line: We should do NO online data analysis in  BooNE, especially as it appears that a combination of lossless data coding + bean-spill triggers permits an affordable DAQ system.