Irina Makhlyueva ALICE DAQ group 28 February 2005 DATE multi-stream recorder

Irina Makhlyueva – ALICE DAQ2 28 February 2005 Contents Purpose of DATE recorders  Throughput balancing Multiple stream recording DATE mStreamRecorder  Structure  Features  Configuration files Performance Plans

Irina Makhlyueva – ALICE DAQ3 28 February 2005 Purpose of DATE online recorders To provide an interface between EBs and CASTOR:  Event Builder by itself has only limited recording functions (local disk and “online recording” to FIFOs/buffers). Anything beyond that is the task of independent processes running on the GDC hosts – online recorders Data conversion option (raw event format→ ROOT) EB/CASTOR throughput balancing  To ensure a continuous data extraction from the EB buffers and make the best use of the available CASTOR throughput

Irina Makhlyueva – ALICE DAQ4 28 February 2005 EB/CASTOR throughput balancing EB’s main task is event building. The EB load balancing is performed by EDM. The total number of GDCs is tuned to optimize the DAQ performance. In theory, the CASTOR segment dedicated to ALICE should have a sufficient (or better) aggregate throughput to match the total data rate delivered by all GDCs. Recorders must not be a bottleneck! In particular, EBs should never get blocked due to the recorder latencies. Therefore, the recorder layer needs a sufficient bandwidth reserve to provide the best possible balance between GDCs and CASTOR throughputs. Currently, CASTOR is lagging behind DATE. In this situation, the important task of the recording layer is to make the best use of the CASTOR capacity. (*) Photo non contractuelle GDC farm (*)CASTOR (*) Total GDC throughput Aggregate CASTOR throughput recorder layer loss recorder bandwidth reserve

Irina Makhlyueva – ALICE DAQ5 28 February 2005 From mono- to multi-stream recording The past DC experience showed that GDCs with single recording streams could not fully load CASTOR. However, increasing the number of GDCs is expensive. The solution: a multiple stream recording, with >1 recording channel (stream) per GDC. This is also beneficial for the GDC efficiency: should a latency in one stream occur, the other stream(s) can take over. Recording layer capacity (BW) ≈ number of ACTIVE streams = total – blocked In practice, the optimal number of streams per GDC is dictated by the requirement of CASTOR saturation → its tuning is one of the DC goals castor Bad: CASTOR is underloaded ~1 stream/GDC DAQ Not good : no BW reserve ≈2 streams/GDC OK : adequate recorder BW, losses are due limited CASTOR BW. ≥3 streams/GDC

Irina Makhlyueva – ALICE DAQ6 28 February 2005 The new DATE multi-stream recorder Was introduced in August, 2004 and has passed extensive testing since then. in DATE since release 5.0 very useful for debugging/testing of new CASTOR facilities is used in the ongoing DC (3 streams/GDC, raw data mode) Event descriptors via SimpleFifo Events via EB Consumer API Fatal condition signals (interrupts) (optional) raw/ROOT transformation Dispatching algorithm n File destination assigned to stream n Reporting to infoLogger/Stat EB buffer GDC disp ….. 1 ……….. 2N Recorder shared memory (internal logging etc) stream (1) Config file mStreamRecorder Event Builder 1 2 N GDC host stream (2) stream (N)

Irina Makhlyueva – ALICE DAQ7 28 February 2005 mStreamRecorder: features The recorder consists of disp + N × stream processes, running on a GDC host EB is configured for online recording (recording device “:” in run parameters) Dispatching process disp:  started by the DATE run control  reads/interpretes the configuration file mStreamRecorder.config;  forks and configures n identical stream processes, distributes event descriptors from the EB between the streams according to a specified dispatching algorithm: “equal-load”, “first- available” or “special” (trigger-mask etc – not yet implemented).;  handles interrupts from the streams. Recording stream process:  totally independent of other streams, communicates only with disp via FIFO (ev. descriptors), signals (fatal conditions), shared memory (status reporting) and a pipe (configuration);  retrieves the events from the EB buffer, using eventBuilder API ;  (optionally, transforms the events into ROOT objects);  Opens, closes, writes files to a specified destination: local disk, CASTOR (RFIO or ROOTd);  Reports its status to disp via shared memory. disp and streams send log messages via DATE infoLogger Both disp and streams use polling with timeout/sleep (10 ms): very low CPU load, good response

Irina Makhlyueva – ALICE DAQ8 28 February 2005 mStreamRecorder: configuration Flexible, easily scalable configuration, currently based on a traditional ASCII config file Simple case the recorders for all GDCs have the same configuration:  3 identical streams  “first-available” dispatching method  all streams have the same properties (destinations, file sizes, polling regimes, buffering). A bit more sophisticated configuration:  Separate configurations for different GDCs (e.g., different number of streams/GDC)  A variety of streams with different destinations >COMMON method=2 Nstreams=3 loglevel=1 >RECORDERS default_rec stream=default_str >OSTREAMS default_str sleep=1 fsize=1024 mxrecl=0 \ pool=alimdc6 stager=lxs5007 \ path=/castor/cern.ch/alice/mdc6/tapes DC6 >COMMON loglevel=1 method=1 dump=1 >RECORDERS default_rec stream=default_str !! 1 stream tbed0084gdc Nstreams=2 !! 2 (default) streams tbed0021gdc stream=default_str stream=public \ stream=local !! 3 streams >OSTREAMS default_str sleep=1 fsize=1024 mxrecl=0 \ pool=alimdc6 stager=lxs5007 \ path=/castor/cern.ch/alice/mdc6/tapes public sleep=2 \ path=/castor/cern.ch/user/m/makhlyui \ pool=public stager=stagepublic fsize=128 local path=/tmp =public !! write to local /tmp Demo

Irina Makhlyueva – ALICE DAQ9 28 February 2005 Recorder performance Rough start with a transition of CASTOR to the new platform → the DC delays. E.g. (end of January): very long file open latency (from seconds to ~1 hour!). Multi-stream feature of the DATE recorder helped to push the CASTOR load to the extreme. Numerous modifications done to mStreamRecorder (protections, error recovery, internal communication, reporting, bugs) → a more robust and better performing code. Ongoing DC configuration: 15 LDCs, 40 GDCs and 3 streams/GDC. Very fresh results: an average data rate of ~300 MB/s, with large momentary fluctuations (CASTOR is being debugged in flight !) Initial debugging and timing was done using local disks and the old (“production version”) CASTOR. Smooth operation with up to 100 streams/GDC and 1-2 GDCs.

Irina Makhlyueva – ALICE DAQ10 28 February 2005 Plans for near-term future ROOT recording commissioning ……….. Transition to DB-based configuration (retaining ASCII option) ………………….. Full integration in the DATE Run Control (SOR.command → SMI object) ………….. Reporting to DATE status display ……… A detailed study of the overall recording performance, as function of # streams / GDC ROOT recording commissioning ……….. Transition to DB-based configuration (retaining ASCII option) ………………….. Full integration in the DATE Run Control (SOR.command → SMI object) ………….. Reporting to DATE status display ……… A detailed study of the overall recording performance, as function of # streams / GDC in progress to be done in progress during the DC in progress to be done in progress during the DC

Irina Makhlyueva – ALICE DAQ11 28 February 2005 Iguatzu Falls (Argentina). © Long-term

Irina Makhlyueva – ALICE DAQ12 28 February 2005 Thanks and Credits R. Divia for his help with the recorder debugging, the event builder API and other aspects of DATE P. Vande Vyvre for numerous discussions B. Panzer for the idea of multiple stream recording J-D Durand, O.Barring and other members of IT/ADC – for help in struggling with CASTOR K. Schossmaier and U.Fuchs, for help with DATE and Linux installation T. Kuhr and C. Cheshkov of ALICE offline group – for the ROOT recording codes and related stuff Photographs from WWW sites: (stream) (Waterfalls) The end

Irina Makhlyueva – ALICE DAQ13 28 February 2005 LDCs GDC pool CASTOR multiple recording streams multiple recording streams Spare slide