1. 1P. Vande Vyvre - CERN/PH ALICE DAQ Progress Report Comprehensive Review IV P. Vande Vyvre – CERN/PH

2. 2P. Vande Vyvre - CERN/PH Acronyms (1) AliROOTALICE Offline sw framework based on ROOT AFFAIRA Fine Fabric and Applications Information RecorderPerformance monitoring sw ADCALICE Data ChallengeALICE DAQ/HLT/MSS/Offline integrated test BWBandwidth CASTORCERN Advanced STORage ManagerCERN developed MSS CDRCentral Data RecordingCentral facility for physics data recording CTPCentral Trigger ProcessorSystem managing TRG L0, L1, L2 DAQData Acquisition System DASDirect Attached Storage Storage accessible from one computer DATEData Acquisition and Test EnvironmentALICE DAQ sw framework DDLDetector Data LinkALICE DAQ optical link DDL DIU DDL Destination Interface UnitOptical Link receiving side (DAQ side) DDL SIU DDL Source Interface UnitOptical Link sender side (detector side) DDL FEMUDDL Front-End EmulatorFast data source for the DDL, emulating a detector readout DDL SIMUDDL SimulatorElectronic board simulating a complete DDL DLTDigital Linear TapeOpen standard for linear magnetic tape

3. 3P. Vande Vyvre - CERN/PH Acronyms (2) DSSDAQ Services ServersServers running services applications such as control,, database, logging EBDSEvent Building and Distribution SystemEvent building load balancing system EDMEvent Destination ManagerSw allocating the GDC for event-building EOREnd Of RunPhase of the DAQ control system FEROFront-End Read-OutElectronics readout interface to DDL SIU FSMFinite State MachineLogical model used for the control GDCGlobal Data CollectorCPU performing event-building HLTHigh Level TriggerSlow firmware or software trigger HWHardware I/O busInput/Output busComputer bus used for input/output L0, L1, L2Trigger levels 0,1,2 Fast Hardware trigger LDCLocal Data ConcentratorCPU performing sub-event building LTCLocal Trigger CrateLocal trigger system: interface to the central TRG and to the TTC. Also stand-alone TRG system LTOLinear Tape OpenOpen standard for linear magnetic tape LTULocal Trigger UnitBoard interfacing the central TRG to the LTC. MOODMonitoring Of Online DataData quality monitoring sw framework based on DATE and ROOT

4. 4P. Vande Vyvre - CERN/PH Acronyms (3) MSSMass Storage SystemData management software NASNetwork Attached Storage Storage accessible from a network through a server NICNetwork Interface CardComputer interface to the network NTWNetwork PCIOpen standard of PC I/O bus RCUReadout Control UnitFinal stage of the ALICE TPC readout ROOTOO software framework for I/O & visualization RORCRead-Out Received CardHost adapter for the DDL p-RORCPCI RORCPCI 32 RORC D-RORCDAQ RORCPCI 64 RORC with 2 integrated DDL channels SANStorage Area NetworkNetwork dedicated to serverless storage SMIState Manager InterfaceRun control based on distributed state machines SORStart Of RunPhase of the DAQ control system SWSoftware TDSTransient Data StorageFacility at the experimental pit to store data before migration to the PDS TRGTrigger TTCTrigger, Timing and ControlOptical broadcast system used by the TRG

5. 5P. Vande Vyvre - CERN/PH ALICE DAQ Institutes, Responsibilities, Milestones DAQ Architecture DDL and D-RORC MOOD (Data quality monitoring) DAQ Fabric Data Challenge V Installation, test and commissioning

6. 6P. Vande Vyvre - CERN/PH ALICE DAQ Institutes, Responsibilities, Milestones DAQ Architecture DDL and D-RORC MOOD (Data quality monitoring) DAQ Fabric Data Challenge V Installation, test and commissioning

7. 7P. Vande Vyvre - CERN/PH Institutes & Responsibilities Birmingham –TRG/DAQ simulation –RORC receiver cards KFKI-Budapest : –DDL: optical links and RORC receiver cards –Radiation tolerance tests (with Technical University/Budapest and Institute of Nuclear Research (ATOMKI/Debrecen) CERN: –DATE: DAQ software framework –DAQ fabric Zagreb: –TRG/DAQ simulation –AFFAIR: performance monitoring package Split: –TRG/DAQ simulation –Storage Collaborating institutes: Istanbul (Data quality monitoring)

8. 8P. Vande Vyvre - CERN/PH Milestones and outcome of CR3 LHCC Milestones Detector readout with January 2003 TDR preparation statusMarch 2003 Common TDR submission to LHCCDecember 2003 DDL pre-production for detector 1Q 2004 test and commissioning D-RORC pre-production for detector1Q 2004 test and commissioning DAQ reference system in DAQ lab: 4Q 2004 DAQ systems for surface tests and commissioning: –SXL2: mounting hall on the surface of point 2 1Q 2005 –Si. lab: ITS surface test 2005 Milestones Final DAQ –Jan 2006Final DAQ system ready with all functionalities 20 % of final performance –Nov 2006 30 % for pp and first HI run –Oct 2008100 % for second HI run (needs and budget)

9. 9P. Vande Vyvre - CERN/PH ALICE DAQ Institutes, Responsibilities, Milestones DAQ Architecture DDL and D-RORC MOOD (Data quality monitoring) DAQ Fabric Data Challenge V Installation, test and commissioning

10. 10P. Vande Vyvre - CERN/PH 25 GB/s 2.50 GB/s 1.25 GB/s Pb-Pb beam Rate Max. ev. size - Central 20 Hz86.0 MB - MB 20 Hz20.0 MB - Dimuon1600 Hz 0.5 MB - Dielectron 200 Hz 9.0 MB pp beam MB 100 Hz 2.5 MB Running modes A: DAQ B: DAQ+HLT Analysis C: DAQ+HLT Trigger Physics requirements Trigger Level 0,1 Trigger Level 2 High-Level Trigger Detector Front-end Buffer Readout Buffer Event-Building Network Storage network Local Data Concentrators (LDC) Global Data Collectors (GDC) Permanent Data Storage (PDS) Detector Data Link (DDL)

11. 11P. Vande Vyvre - CERN/PH GDC DAQ architecture CTP LTU TTC FERO LTU TTC FERO LDC BUSY Rare/All Event Fragment Sub-event Event File Storage Network TDS PDS L0, L1a, L2 262 DDLs EDM LDC Load Bal. LDC HLT Farm FEP DDL H-RORC 10 DDLs 10 D-RORC 10 HLT LDC 123 DDLs TDS DSS Event Building Network 329 D-RORC 175 Detector LDC 50 GDC 25 TDS 5 DSS

12. 12P. Vande Vyvre - CERN/PH GDC Distribution of HLT decisions inside DAQ (1) CTP LTU TTC FERO LTU TTC FERO LDC Storage Network TDS PDS L2 trigger pattern EDM LDC HLT Farm FEP TDS DSS Event Building Network L2 trigger pattern L2 trigger pattern Original LDC pattern Refined LDC pattern HLT output pattern Refined LDC pattern HLT output pattern Original LDC pattern Refined LDC pattern HLT output pattern

13. 13P. Vande Vyvre - CERN/PH Distribution of HLT decisions inside DAQ (2) LDC DDL DIUDDL SIU DATE banks readout recorder D-RORC Raw data Events fragments Event Building Network HLT Detector NIC decision agent Sub-events HLT decisions

14. 14P. Vande Vyvre - CERN/PH Key building blocks and concepts Protocol-less push-down strategy System throttling by X-on/X-Off signals Detector interface via a standard link (DDL) –Readout –Control and download Software framework (DATE) –Dataflow (data driven according to TRG generated event tags) –Control (FSM and messages) –Monitoring Distributed DAQ Fabric –LDC –GDC –DSS

15. 15P. Vande Vyvre - CERN/PH ALICE DAQ Institutes, Responsibilities, Milestones DAQ Architecture DDL and D-RORC MOOD (Data quality monitoring) DAQ Fabric Data Challenge V Installation, test and commissioning

16. 16P. Vande Vyvre - CERN/PH D-RORC Hardware Architecture APEX FPGA 64-bit/66 MHz, PCI/PCI-X Media I/F 1 Media I/F 2 Busy I/F Conf. Flash JTAG P11P12 P13P14 CMC I/F Optical I/F LVDS I/F Configuring 250 MB/s 528 MB/s

17. 17P. Vande Vyvre - CERN/PH D-RORC Hardware D-RORC with integrated DIU ports 2 DDL channels Integration with the HLT system D-RORC with plug-in DIU Read out single DDL channel Detector integration

18. 18P. Vande Vyvre - CERN/PH D-RORC and DIU PC + single-channel D-RORC CPU: 2 x Xeon 2400 MHz Kernel: 2.4.20-30.7 Testing the integration of the D-RORC and the DIU using the new library (v4.2) rorc_receive –g 3... (internal loopback) rorc_receive –g 1... (DIU loopback)

19. 19P. Vande Vyvre - CERN/PH D-RORC and DATE 2 D-RORC cards Same PCI bus Data generation: Front-end emulator cards Internal data generator

20. 20P. Vande Vyvre - CERN/PH D-RORC and DATE 1 week 220 10 6 events 98 MBytes/s

21. 21P. Vande Vyvre - CERN/PH Data splitter for DAQ/HLT interface PC + twin-channel D-RORC PC + single-channel D-RORC FEE Emulator CPU: Intel P3 800 MHz Kernel: 2.4.20-24.7 rorc_receive CPU: 2 x Xeon 2400 MHz Kernel: 2.4.20-30.7 rorc_receive DAQ HLT Detector

22. 22P. Vande Vyvre - CERN/PH ALICE DAQ Institutes, Responsibilities, Milestones DAQ Architecture DDL and D-RORC MOOD (Data quality monitoring) DAQ Fabric Data Challenge V Installation, test and commissioning

23. 23P. Vande Vyvre - CERN/PH Monitoring Of Online Data Detector Debugger (raw data visualizer) Written in C/C++ Based on ROOT Based on DATE Data quality monitoring framework for all ALICE detectors MOOD

24. 24P. Vande Vyvre - CERN/PH MOOD: Current Detector Scheme Detectors currently implemented : ITS – SDD TPC Sector HMPID Photocathode Detectors to be implemented : All detectors individually Test setups ALICE as a whole

25. 25P. Vande Vyvre - CERN/PH TPC Sector Test Pad Plane Charge Plane Event Size Distribution 3D View Charge Plane 3D Tool Bar

26. 26P. Vande Vyvre - CERN/PH HMPID Photocathode (1) Tabs Photocathode 3D View Size Distribution Event Dump Logs …

27. 27P. Vande Vyvre - CERN/PH HMPID Photocathode (2) Tabs Photocathode 3D View Size Distribution Event Dump Logs …

28. 28P. Vande Vyvre - CERN/PH ITS - SDD (1) Tabs SDD Display 3D View Size Distribution Event Dump Logs …

29. 29P. Vande Vyvre - CERN/PH ITS - SDD (2) Tabs SDD Display 3D View Size Distribution Event Dump Logs …

30. 30P. Vande Vyvre - CERN/PH ALICE DAQ Institutes, Responsibilities, Milestones DAQ Architecture DDL and D-RORC MOOD (Data quality monitoring) DAQ Fabric Data Challenge V Installation, test and commissioning

31. 31P. Vande Vyvre - CERN/PH DAQ Reference System 1 FC switch 1 GE switch 4 LDCs (Local Data Concentrator): - rackmount PCs - U1, U2, U4 height - equipped with 6 D-RORCs 6 DDLs 2 GDCs (Gobal Data Collector): - rackmount PCs - U1 height - equipped with FC cards 2 TDS (Transient Data Storage): - rackmount disk array - U2 height - IDE and FC disks 1 DSS (DAQ Services): - rackmount PC - U4 height - hot-swap SCSI disks 1 KVM switch standard LAN

32. 32P. Vande Vyvre - CERN/PH Reference Setup – Front KVM switch: Raritan Paragon UMT2161 2x LDCs: - 4U height - dual Xeon - 2x D-RORC cards  DDL 2x GDCs: - 1U height - dual Xeon - Qlogic QLA2310F cards 2x disk arrays: - Infortrend IFT-6330 - DotHill SANnet II DSS: - 4U height - quad Xeon - 3x 36GB SCSI disks L3 rack 2x GB Ethernet switch: 3COM SuperStack 3 Fibre Channel switch: Broacade SilkWorm 3800 DAQ lab Detect and address integration issues System for development and support

33. 33P. Vande Vyvre - CERN/PH Reference Setup - Rear Power distributor Cat5 cables: - Ethernet - KVM - RJ45 connectors Optical cables: - DDL - FiberChannel - 2 Gbit/s multimode - LC-LC connectors L3 rack Mounting rails (!)

34. 34P. Vande Vyvre - CERN/PH ALICE DAQ Institutes, Responsibilities, Milestones DAQ Architecture DDL and D-RORC MOOD (Data quality monitoring) DAQ Fabric Data Challenge V Installation, test and commissioning

35. 35P. Vande Vyvre - CERN/PH ADC V Hw Architecture ~ 80 CPU servers 2 x 2.4 GHz Xeon, 1 GB RAM, Intel 8254EM Gigabit in PCI-X 133 (Intel PRO/1000), CERN Linux 7.3.3 4 x 7 Disk servers 2 x 2.0 GHz Xeon 1 GB RAM Intel 82544GC 32 x GE 32 IA64 HP-rx2600 Servers 2 x 1 GHz Itanium-2 2 GB RAM Broadcom NetXtrem BCM5701 (tg3) RedHat Advanced Workstation 2.1 6.4 GB/s to memory, 4.0 GB/s to I/O 3COM 4900 16 x Gbit Enterasys E1 OAS 12 Gbit, 1 x 10 Gbit Enterasys ER16 16 slots 4/8 x Gbit or 1 x 10 Gbit/slot 3COM 4900

36. 36P. Vande Vyvre - CERN/PH Achievements (1) System size (lack of resources in LCG testbed) System scalability Performance test with ALICE data traffic – ALICE-like traffic – ALICE-like events simulated data used: Realistic (sub-)event size on tape (ALICE year 1) DATE load-balancing demonstrated and used Sustained bw to tape not achieved – Peak 350 MB/s. Sustained 280 MB/s over 1 day – Reached production-quality level only last week of test IA-64 from Openlab successfully integrated in the ADC V Simulated raw data used for performance test Data read back from CASTOR and verified

37. 37P. Vande Vyvre - CERN/PH ALICE DC: Scalability

38. 38P. Vande Vyvre - CERN/PH ALICE DC – MSS Bw

39. 39P. Vande Vyvre - CERN/PH Achievements (2) Network LDCs and GDCs: stable and scaleable including trunking Between GDCs and disk servers: Unreliable Truncking not scaling as expected Module broken and replaced twice in Enterasys 10 Gbit Eth. Backbone New generation of NIC cards (Intel Pro 1000) – NIC from Broadcom unreliable. Replaced by Intel Pro 1000. Storage –Hardware problem on the disk servers –Several last minute workarounds needed (scripts for monitoring and reconfiguring)

40. 40P. Vande Vyvre - CERN/PH Performance Goals (1) 650 MB/s

41. 41P. Vande Vyvre - CERN/PH Performance Goals (2) MB/s to Mass Storage 300 MB/s

42. 42P. Vande Vyvre - CERN/PH Open issues and future goals CASTOR: –Recovery from malfunctioning disk server –New stager –Special daemon between CPU and disk server instead of standard RFIO daemon. Needed to achieve adequate performance. Should be put back in main development. DAQ –Increase performances Network –First prototypes of 10 Gbit Eth. equipment from Enterasys unreliable –Enterasys support not effective on this case Meeting scheduled with LCG PEB to present results and address the open issues

43. 43P. Vande Vyvre - CERN/PH ALICE DAQ Institutes, Responsibilities, Milestones DAQ Architecture DDL and D-RORC MOOD (Data quality monitoring) DAQ Fabric Data Challenge V Installation, test and commissioning

44. 44P. Vande Vyvre - CERN/PH DAQ Commissioning (1) What: –DAQ itself –How the DAQ will help the commissioning of other systems Where: –ACR: ALICE Control Room. –PX24-CR1: ALICE DAQ Counting Room located in the access shaft. –SXL2: mounting hall on the surface of point 2. –UX25: experimental underground area When: –1 Q 2005: all DAQ functionalities. Hw at the final location. 20 DDLs for readout of 2 TPC sectors in SXL2 + other detectors –Jan 2006 Final DAQ system ready with all functionalities 20 % of performance –Nov 2006 30 % of performance –Oct 2008100 % of performance (needs and budget)

45. 45P. Vande Vyvre - CERN/PH DAQ Commissioning (2) Tests at the construction site of hardware elements –Verification of the DDLs and D-RORCs –Test of the cards with a test station made of a PC and DDL test software Tests at the development sites of software elements: –AFFAIR, DATE, DDL sw, CASTOR –Combined tests well before installation: test beams, Data Challenges Standalone tests in experimental area –DAQ: possibility of injecting data at every stage of the data flow. –Each segment of the dataflow first tested in isolation and then in combination with other elements DAQ integration at Point 2: –Integration with ECS –Tests with TRG, HLT, DCS –Detector test and commissioning Test with cosmic and pulser trigger –From June 2005: a TTC-based trigger to trigger autonomous DDL data sources for global tests of the DAQ involving DDLs. –From January 2006: a cosmic or pulser trigger for the commissioning of detectors involving Trigger and DAQ

46. 46P. Vande Vyvre - CERN/PH DAQ Tools for Detector Commissioning Lab test –DDL Simulator: standalone daughter-card –Detector readout with DDL and DATE Beam test –Detector readout by DDL and DATE –VME for trigger and older electronics (Silicon telescope e.g.) Point 2 –DAQ system at Point 2 in 2004 –Detector test and commissioning –System will evolve in size and performances according to the needs. –Concurrent tests of several detectors (~3 in 2004, all in 2006) –Complete capabilities since the start –Control from ACR or any computer Other tests –ITS integrated test DDL Simulator Detector Readout card under test

47. 47P. Vande Vyvre - CERN/PH DAQ Tools for Detector Commissioning Lab test –DDL Simulator: standalone daughter- card –Detector readout with DDL and DATE Beam test –Detector readout by DDL and DATE –VME for trigger and older electronics (Silicon telescope e.g.) Point 2 –DAQ system at Point 2 in 2004 –Detector test and commissioning –System will evolve in size and performances according to the needs. –Concurrent tests of several detectors (~3 in 2004, all in 2006) –Complete capabilities since the start –Control from ACR or any computer Other tests –ITS integrated test SIU (DDL)Optical Fiber to LDC Input Data from Pattern Generator

48. 48P. Vande Vyvre - CERN/PH DAQ/TOF integration 4.5 days 59 10 6 events 86 MBytes/s.

49. 49P. Vande Vyvre - CERN/PH DAQ Tools for Detector Commissioning Lab test –DDL Simulator: standalone daughter- card –Detector readout with DDL and DATE Beam test –Detector readout by DDL and DATE –VME for trigger and older electronics (Silicon telescope e.g.) Point 2 –DAQ system at Point 2 in 2004 –Detector test and commissioning –System will evolve in size and performances according to the needs. –Concurrent tests of several detectors (~3 in 2004, all in 2006) –Complete capabilities since the start –Control from ACR or any computer Other tests –ITS integrated test DDL LDC (PC/Linux) RORC DDL DIU Detector readout GDC LDC (VME/Linux) Trigger Logic Event Building Network Data Storage in Computing Center DDL SIU Silicon tel. readout DATE V4

50. 50P. Vande Vyvre - CERN/PH DAQ/Detector integration status DATEDDL data generator DDL readout MOOD Data quality monitoring SPD SSD SDD  TPC  In progress  TRD  TOF  HMPID  Muon  In progress PHOS ZDC FMD T0 V0

51. 51P. Vande Vyvre - CERN/PH DAQ Tools for Detector Commissioning Lab test –DDL Simulator: standalone daughter- card –Detector readout with DDL and DATE Beam test –Detector readout by DDL and DATE –VME for trigger and older electronics (Silicon telescope e.g.) Point 2 –DAQ system at Point 2 in 2004 –Complete capabilities since the start –Detector test and commissioning –System will evolve in size and performances according to the needs –Concurrent tests of several detectors (~3 in 2004, all in 2006) –Control from ACR or any computer Other tests –ITS integrated test

52. 52P. Vande Vyvre - CERN/PH DAQ Tools for Detector Commissioning Lab test –DDL Simulator: standalone daughter- card –Detector readout with DDL and DATE Beam test –Detector readout by DDL and DATE –VME for trigger and older electronics (Silicon telescope e.g.) Point 2 –DAQ system at Point 2 in 2004 –Complete capabilities since the start –Detector test and commissioning –System will evolve in size and performances according to the needs –Concurrent tests of several detectors (~3 in 2004, all in 2006) –Control from ACR or any computer Other tests –ITS integrated test

53. 53P. Vande Vyvre - CERN/PH DAQ for Test & Commissioning SXL HallSX Hall UX25 (Experimental Area) PX24/CR1 (DAQ) PX24/CR2 (HLT) SR Hall Access Shaft DDL LAN DDL Patch Panel (ALICE sub-detector assembly) (Networking) PX24/CR3 (DCS) PX24/CR4 (Misc.) LAN WR 1 WR 2 ACR 1 Q 2005 : DAQ System at Point 2 12 DDLs for TPC 8 DDLs for others 3 partitions 2005: similar system in Si. lab.

54. 54P. Vande Vyvre - CERN/PH Final DAQ System SXL HallSX Hall UX25 (Experimental Area) PX24/CR1 (DAQ) PX24/CR2 (HLT) SR Hall Access Shaft DDL LAN DDL Patch Panel (ALICE sub-detector assembly) (Networking) PX24/CR3 (DCS) PX24/CR4 (Misc.) WR 1 WR 2 ACR Installation staging: (% of final DAQ performance) 2006: 20% 2007: 30% 2008: 100%