1. ACADEMIC TRAINING B. Panzer – CERN/IT, F. Rademakers – CERN/EP, P. Vande Vyvre - CERN/EP Academic Training CERN

2. CERN Academic Training 12-16 May 20032P. Vande Vyvre CERN-EPOutline  Day 1 (Pierre VANDE VYVRE) Outline, main concepts Requirements of LHC experiments Data Challenges  Day 2 (Bernd PANZER) Computing infrastructure Technology trends  Day 3 (Pierre VANDE VYVRE) Trigger and Data Acquisition  Day 4 (Fons RADEMAKERS) Simulation, Reconstruction and analysis  Day 5 (Bernd PANZER) Computing Data challenges Physics Data Challenges Evolution

3. CERN Academic Training 12-16 May 20033P. Vande Vyvre CERN-EP Trigger and Data Acquisition  Dataflow, Trigger and DAQ architectures  Trigger  Data transfer  Event building  Storage  Software framework  Simulation  Conclusion

4. CERN Academic Training 12-16 May 20034P. Vande Vyvre CERN-EP Online dataflow Trigger Level 0,1 Trigger Level 2 Event-Build. Netw. High-Level Trigger Storage network Transient storage Detector Digitizers Front-end Pipeline/Buffer Decision Readout Buffer Decision Subevent Buffer Event Buffer Permanent storage Decision

5. CERN Academic Training 12-16 May 20035P. Vande Vyvre CERN-EP GDC TRG/DAQ/HLT CTP LTU TTC FERO LTU TTC FERO LDC FEP LDC FEP LDC FEP LDC FEP LDC FEP Event Building Network BSY Rare/All HLT Farm Event Fragment Sub-event Event File Storage Network TDS PDS L0, L1a, L2 DDL RORC FE FERO EDM LDC FEP RORC Load Balancing

6. CERN Academic Training 12-16 May 20036P. Vande Vyvre CERN-EPALICE Levels 4 LV-1 rate500 Hz Readout 25GB/s Storage 1250 MB/s

7. CERN Academic Training 12-16 May 20037P. Vande Vyvre CERN-EP DATAFLOWDATAFLOW EVBEVB R/O S Y S T E M HLTHLT LV L1 D E T R/O ROD LVL2 TriggerDAQ ATLAS 2.5  s ~ 10 ms 40 MHz 75 kHz ~2 kHz ~ 200 Hz Calo MuTrCh Other detectors ROB IOM SFI SFO RRC RRM EBN EFN FE Pipelines Read-Out Drivers ROD-ROB Connection Read-Out Buffers ROD-ROS Merger I/O Manager Dataflow Manager Sub-Farm Input Sub-Farm Output Event Filter N/work ROIB L2P L2SV L2N Event Filter DFM EFP RoI Builder L2 Supervisor L2 N/work L2 Proc Unit RoI RoI data = 2% RoI requests Lvl2 acc = ~2 kHz Event Building N/work ~ sec Lvl1 acc = 75 kHz 40 MHz 120 GB/s ~ 300 MB/s ~3+3 GB/s Event Filter Processors 120 GB/s ~3 GB/s EFacc = ~0.2 kHz Levels 3 LV-1 rate100 kHz Readout 100GB/s Storage 100 MB/s

8. CERN Academic Training 12-16 May 20038P. Vande Vyvre CERN-EPCMS Levels 2 LV-1 rate100 kHz Readout 100GB/s Storage 100 MB/s

9. CERN Academic Training 12-16 May 20039P. Vande Vyvre CERN-EPLHCb Levels 3 LV-1 rate1 MHz Readout 4GB/s Storage 40 MB/s

10. CERN Academic Training 12-16 May 200310P. Vande Vyvre CERN-EPTrigger Multi-level trigger system Reject background Select most interesting collisions Reduce total data volume

11. CERN Academic Training 12-16 May 200311P. Vande Vyvre CERN-EP Multi-level trigger Multi-level trigger system to op timize  Rate and granularity  System speed and size  Technology required

12. CERN Academic Training 12-16 May 200312P. Vande Vyvre CERN-EPTrigger  Trigger Level 0 Custom logic  Trigger Level 1 Custom logic Special architectures Computing farm  Trigger Level 2 Special architectures Computing farm  High Level Trigger (HLT) Computing farm  HEP specific Home-made development Custom building blocks Fast but rigid Programmable by “a few experts”  General-purpose Home-made software Commodity building blocks Slow but flexible Programmable by “all”

13. CERN Academic Training 12-16 May 200313P. Vande Vyvre CERN-EP Trigger & Timing distribution  Transfer from TRG to electronics  One to many  Massive broadcast (100’s to 1000’s)  Optical, Digital HEP-specific components HEP developments

14. CERN Academic Training 12-16 May 200314P. Vande Vyvre CERN-EP Trigger & Timing distribution

15. CERN Academic Training 12-16 May 200315P. Vande Vyvre CERN-EP Detector & Readout Data Link (1)  Transfer from detector to DAQ  Point-to-point  Massive parallelism (100’s to 1000’s)  Interface detector/readout  Analog HEP-specific components  Digital HEP developments based on commodity components Fiber Channel or Gigabit Ethernet 2.1 or 2.5 Gb/s

16. CERN Academic Training 12-16 May 200316P. Vande Vyvre CERN-EP Detector & Readout Data Link (2) SERDESOT PM ID APEX20KE OSC RX TX TX_CLK RX_CLK PLL IF_CLK OT – optical transceiver PM – power monitor circuit OSC – crystal oscillator SERDES – serializer/de-serializer ID – identification memory DDL Interface I2C SPI TLK2501 PROTOCOL device 2.5 Gb/s 250 MB/s 850nm 50/125um MMF (1-L P )250 MB/s

17. CERN Academic Training 12-16 May 200317P. Vande Vyvre CERN-EP Optical link source

18. CERN Academic Training 12-16 May 200318P. Vande Vyvre CERN-EP Links Adapters  Transfer from 1 or several links to I/O bus of the memory or the computer  Many-to-one  Massive parallelism (100’s to 1000’s)  Interface detector/readout  Physical interface realized by Custom chip IP core (VHDL code synthesized in FPGA)

19. CERN Academic Training 12-16 May 200319P. Vande Vyvre CERN-EP PCI evolution  Initiative of Intel  Public from the start, “imposed” to industry  Industry de-facto standard for local I/O: PCI (PCI SIG) 1992: origin32 bits 33 MHz133 MBytes/s 1993: V2.032 bits 1994: V2.1 1996: V2.2 64 bits 66 MHz512 MBytes/s 1999: PCI-X 1.0 64 bits133 MHz1 GBytes/s 2002: PCI-X 2.064 bits512 MHz4 Gbytes/s

20. CERN Academic Training 12-16 May 200320P. Vande Vyvre CERN-EP Optical link destination & PCI adapter

21. CERN Academic Training 12-16 May 200321P. Vande Vyvre CERN-EP Link and adapter performance (1) Example of ALICE DDL and RORC PCI 32 bits 33 MHz interface with custom chip No local memory. Fast transfer to PC memory DDL saturated for block size above 5 kBytes: – 101 Mbytes/sec Event rate saturated for block size below 5 kBytes: – 35’000 events/sec – RORC handling overhead in LDC: 28 µsec

22. CERN Academic Training 12-16 May 200322P. Vande Vyvre CERN-EP Link and adapter performance (2) PCI 32 bits 66 MHz with commercial IP core No local memory. Fast transfer to PC memory Reach 200 MB/s for block size above 2 kBytes. Total PCI load: 92 % Data transfer PCI load: 83 %

23. CERN Academic Training 12-16 May 200323P. Vande Vyvre CERN-EP Subevent & event buffer  Baseline: Adopt commodity component (PC) Develop fast dual-port memories  Key parameters: Cost/performance Performance: memory bandwidth

24. CERN Academic Training 12-16 May 200324P. Vande Vyvre CERN-EP PC Memory Bandwidth www.cs.virginia.edu Stream v4.0 with gcc 2.96-103 AMD modules Xeon machine } Pentium II & III machines

25. CERN Academic Training 12-16 May 200325P. Vande Vyvre CERN-EP Event Building Network (1)  Baseline: Adopt broadly exploited standards Switched Ethernet (ALICE, ATALS, CMS) Adopt a performant commercial product Myrinet (CMS)  Motivations for switched Ethernet: Performance of Gigabit Ethernet switches currently available already adequate for most DAQ @ LHC Use of commodity items: network switches and interfaces Easy (re)configuration and reallocation of resources  Same technology also used for DAQ services

26. CERN Academic Training 12-16 May 200326P. Vande Vyvre CERN-EP Event Building Network (2) GDC 4TDS 1GDC 1 Sector 35-36 Switch 18 Pixel - Strips Switch 19 Sector 1-2 Switch 1 Muon-PMD-TRG Switch 21 1 14... 211 224 1 10... 31 40 200 MB/s 2500 MB/s 60 MB/s 600 MB/s 2500 MB/s 1250 MB/s 1 141 8 1 10 60 MB/s Data Link to computing center TPC LDC C 0 TRD LDC GDCLDC Drift Switch 20 1 9 TOF-HM-PHOS Switch 22 LDC 1 7 TDS 2 21 20 TDS

27. CERN Academic Training 12-16 May 200327P. Vande Vyvre CERN-EP Event Building Network (3)  Baseline: Adopt broadly exploited standards Transport protocol: TCP/IP (ALICE event building) Adopt efficient protocol Transport protocol: raw packets (LHCb TRG L1)  Motivations for TCP/IP: Reliable and stable transport service: Flow control handling Lost packet handling Congestion control Can be verified during the ALICE Data Challenges Industry mainstream: Guaranteed support from present and future industrial providers: operating systems, switches, interfaces Constant improvements

28. CERN Academic Training 12-16 May 200328P. Vande Vyvre CERN-EP Ethernet NIC’s Performance  Fast Ethernet copper  Intel 82557, 82550, 82559 with eepro100 driver, mostly on-board  around 11 MB/s, stable  3Com 3C980 *, 3C905 with 3c59x driver, mostly on-board  around 11 MB/s, stable  Gigabit Ethernet  NetGear GA620 with acenic driver  up to 78 MB/s  3Com 3C996 with bcm5700 or tg3 driver  up to 88 MB/s (150% of one CPU)  Intel Pro/1000 * (82545EM) with e1000 driver  up to 95 MB/s (56% -> 75% of one CPU)

29. CERN Academic Training 12-16 May 200329P. Vande Vyvre CERN-EP ADC IV: DATE Event Building (1)

30. CERN Academic Training 12-16 May 200330P. Vande Vyvre CERN-EP ADC IV: DATE Event Building (2) Event building No recording 5 days non-stop 5 days non-stop 1750 MBytes/s sustained (goal was 1000) 1750 MBytes/s sustained (goal was 1000)

31. CERN Academic Training 12-16 May 200331P. Vande Vyvre CERN-EP Transient Data Storage  Transient Data Storage at point 2 before archiving (migration to tape), if any, in the computing center  Several options being tested by ALICE DAQ Technologies Disk attachment:  DAS: IDE (commodity), SCSI  NAS: disk server  SAN: Fiber Channel RAID-level Key selection criteria: cost/performance & bandwidth/box

32. CERN Academic Training 12-16 May 200332P. Vande Vyvre CERN-EP Storage: file & record size (file cache active)

33. CERN Academic Training 12-16 May 200333P. Vande Vyvre CERN-EP Storage: file & record size (file cache inactive)

34. CERN Academic Training 12-16 May 200334P. Vande Vyvre CERN-EP Storage: effect of connectivity

35. CERN Academic Training 12-16 May 200335P. Vande Vyvre CERN-EP Storage: effect of SCSI RAID

36. CERN Academic Training 12-16 May 200336P. Vande Vyvre CERN-EP Transient Data Storage  Disk storage highly non scalable  To achieve high bandwidth performance 1 stream, 1 device, 1 controller, 1 bus With these conditions: 15-20 MB/s with 7.5 kRPM IDE disks 18-20 MB/s with 10 kRPM SCSI disks  To obtain 1.25 GB/s with commodity solutions Footprint too big Infrastructure cost too high  Investigate ways to obtain more compact performance RAID (Redundant Array of Inexpensive Disks) RAID 5, large caches, intelligent controllers HP 3 SCSI devices: 30 MB/s with 10 kRPM disks HP 6 SCSI devices: 40 MB/s with 10 kRPM disks EMC 7 FCS: 50 MB/s with 10 kRPM disks (4 U) IBM 5 FCS: 70 MB/s with 15 kRPM disks Dot Hill SANnet II: 90 MB/s with 15 kRPM disks (2 U)

37. CERN Academic Training 12-16 May 200337P. Vande Vyvre CERN-EP Permanent Data Storage (1)  Infinite storage  At very low cost  Must be hidden by a MSS  Critical area Small market Limited competition Not (yet) commodity  Solution demonstrated since ‘02

38. CERN Academic Training 12-16 May 200338P. Vande Vyvre CERN-EP Permanent Data Storage (2) Tape Drive STK 9940A10 MB/s 60 GB/Volume SCSI STK 9940B30 MB/s 200 GB/Volume Fibre Channel Tape Library Several tape drives of both generations

39. CERN Academic Training 12-16 May 200339P. Vande Vyvre CERN-EP Permanent Data Storage (3)

40. CERN Academic Training 12-16 May 200340P. Vande Vyvre CERN-EP DAQ Software Framework  DAQ Software Framework Common interfaces for detector-dependant applications Target the complete system from the start  ALICE DATE (Data Acquisition and Test Environment) Complete ALICE DAQ software framework: Data-flow: detector readout, event building System configuration, control (100’s of programs to start, stop, synchronize) Performance monitoring Evolving with requirements and technology  Key issues Scalability Very small configurations (1 PC). Used in test beams Verified for scale of the final system (100’s of PCs) during the DC Support and documentation

41. CERN Academic Training 12-16 May 200341P. Vande Vyvre CERN-EP Run Control (1)

42. CERN Academic Training 12-16 May 200342P. Vande Vyvre CERN-EP Run Control (2) State of one node

43. CERN Academic Training 12-16 May 200343P. Vande Vyvre CERN-EP Performance monitoring - AFFAIR LDC Evt. Build. Switch GDC Disk Server DATE ROOT I/O CASTOR Tape Server ROOT I/O performances Files Round Robin DB DATE performances Fabric monitoring ROOT DB ROOT Plots ROOT Plots for Web CASTOR performances

44. CERN Academic Training 12-16 May 200344P. Vande Vyvre CERN-EP Control Hierarchy DAQ Run Control ECS Trigger Control Sys. Detector Control Sys. TPCMuonPixelTPCMuonPixelTPCMuonPixel TPC LTC LDC 1 LDC 2 LDC 216 GASHV

45. CERN Academic Training 12-16 May 200345P. Vande Vyvre CERN-EP ECS operators ECS functions Configuration and booking Synchronize subsytems Operator console Automated procedures config operators State Machines Command/Status TPCPixelMuonTPCPixelMuonTPCPixelMuon DCSDAQTRG DAQTRGDCS Experiment Control System

46. CERN Academic Training 12-16 May 200346P. Vande Vyvre CERN-EP Detector ADetector B Physics Run Partition LDC/FEP Partition: Physics Run CTP LDC/FEP GDC LDC/FEP RORC DDL DIU GDC Storage LTC TTC DDL SIU RORC DDL TTC TTC Rx DDL SIU Det. RO LDC/FEP LTC TTC DDL TTC Event Building Network

47. CERN Academic Training 12-16 May 200347P. Vande Vyvre CERN-EP Physics Run Partition Standalone Partition LDC/FEP 2 Partitions: Physics Run & Standalone CTP LDC/FEP GDC LDC/FEP RORC DDL DIU GDC Storage LTC TTC DDL SIU RORC DDL TTC TTC Rx DDL SIU Det. RO LDC/FEP LTC TTC DDL TTC Event Building Network

48. CERN Academic Training 12-16 May 200348P. Vande Vyvre CERN-EP ADC IV: DATE Scalability test

49. CERN Academic Training 12-16 May 200349P. Vande Vyvre CERN-EP Simulation conditions/results Conditions: 8000 Hz tot: 1600 Hz CE,MB,EL,MU EL, MU considered rare 50 % rejection at LDC HLT rejects 80 % of EL Realistic event sizes, distributions, buffer numbers, transfer rates Original count After P/F considerations Final count Huge and unacceptable decrease of EL and MU triggers due to detector busy

50. CERN Academic Training 12-16 May 200350P. Vande Vyvre CERN-EP The problem DAQ ~50 GB/s (after P/F and time to read events into detector buffers) ~25 GB/s (limited by DDL rates) ~1.25 GB/s (after compression by 0.5) Huge reduction of the original rare decays (Electron-EL and Muon-MU) due to various backpressures PDS Detectors EL MU CE,MB Would like to accept all rare decays

51. CERN Academic Training 12-16 May 200351P. Vande Vyvre CERN-EP The proposed solution  High/low level at LDC to inform the CTP to block “non important” high bandwidth triggers ( CE, MB) to prevent multi-event buffer getting full any At high level any LDC buffer for any detector blocks CE,MB CTP L0 CE/MB high low all LDCs all CE,MB restarted when all LDCs fall below low level CTP high low all LDCs L0 CE/MB

52. CERN Academic Training 12-16 May 200352P. Vande Vyvre CERN-EP The proposed solution  High/low level at LDC to inform the CTP to block “non important” high bandwidth triggers ( CE, MB) to prevent multi-event buffer getting full  Result: almost no losses after P/F

53. CERN Academic Training 12-16 May 200353P. Vande Vyvre CERN-EP Promises of the future  Industry has learned to do switches for Telco: Silicon has been developed  Exponential development of Internet has lead to commodity networking. Same revolution as WS in ’90s  Switches are better  switches everywhere !  Industry is mastering wireless technology  Mobility for all !

54. CERN Academic Training 12-16 May 200354P. Vande Vyvre CERN-EP I/O and system busses I/O System

55. CERN Academic Training 12-16 May 200355P. Vande Vyvre CERN-EPInfiniband  Techno 2.5 Gbit/s line rate 1, 4 or 12 lines giving 0.5, 2, 6 GB/S Switch-based system Transport: reliable connection and datagram, unreliable connection and datagram, IPV6, ethertype  Common link architecture and components with Fibre Channel and Ethernet  Chips: Cypress, IBM, Intel, LSI logic, Lucent, Mellanox, Redswitch  Products: Adaptec, Agilent

56. CERN Academic Training 12-16 May 200356P. Vande Vyvre CERN-EPInifiniband CPU HCA Mem Ctrl Switch Host Channel Adapter (HCA) Target Channel Adapter (TCA) Fibre Channel TCATCA SCSI TCATCA Gigabit Ethernet TCATCA Host

57. CERN Academic Training 12-16 May 200357P. Vande Vyvre CERN-EP Inifiniband: multiple hosts CPU HCA Mem Ctrl Switch FCFC TCATCA SCSI TCATCA Gig. Eth. TCATCA CPU HCA Mem Ctrl Switch SCSI TCATCA TCATCA Host 1 Host 2 Router Internet

58. CERN Academic Training 12-16 May 200358P. Vande Vyvre CERN-EP Rapid I/O  The RapidIO Interconnect Architecture: Chip-to-chip and board-to-board communications at performance levels scaling to ten Gigabits per second and beyond. High-performance, packet-switched, interconnect technology Switches on the board  The RapidIO Trade Association: Non-profit corporation controlled by its members Direct the future development For networking products: increased bandwidth, lower costs, and a faster time-to-market than other more computer-centric bus standards. Steering Committee: Alcatel, Cisco Systems, EMC Corporation, Ericsson, Lucent Technologies, Mercury Computer Systems, Motorola, and Nortel Networks

59. CERN Academic Training 12-16 May 200359P. Vande Vyvre CERN-EPConclusions  Trigger and Data Acquisition systems Large and complex systems 100s of components Most are commodity Some HEP developments to adapt to special requirements  Integration is THE BIG ISSUE Testing Software framework Data flow Control Data Challenges to verify the performances and the integration  Simulation to verify the overall system behavior with a detector in the beam !

60. CERN Academic Training 12-16 May 200360P. Vande Vyvre CERN-EPTomorrow  Day 1 (Pierre VANDE VYVRE) Outline, main concepts Requirements of LHC experiments Data Challenges  Day 2 (Bernd PANZER) Computing infrastructure Technology trends  Day 3 (Pierre VANDE VYVRE) Trigger and Data acquisition  Day 4 (Fons RADEMAKERS) Simulation, Reconstruction and analysis  Day 5 (Bernd PANZER) Computing Data challenges Physics Data Challenges Evolution