Bernd Panzer-Steindel, CERN/IT ALICE data flow, T0 buffer system, performance issues 27.March 2006 Bernd Panzer-Steindel, CERN/IT

Bernd Panzer-Steindel, CERN/IT Considerations for two scenarios : stable running first 2 years of ‘consolidation’ Activities : Data recording -- copy from the DAQ -- copy to tape Data export -- copy to and from the different Tiers Data processing -- calibration using calibration streams and using selected RAW data -- processing of RAW data and ESD production -- ESD data analysis -- end-user analysis Facility T0 CAF (CERN Analysis Facility) T0 and CAF are logical concepts, physical differences constrain the flexible resource re-allocation 27.March 2006 Bernd Panzer-Steindel, CERN/IT

Bernd Panzer-Steindel, CERN/IT ALICE generic data flow scenario Analysis Farm Calibration Farm USER Data CAL ESD CAL RAW RAW ESD CAL Tape Storage DAQ Disk Buffer Disk Buffer RAW RAW ESD ESD file aggregation layers ? Reconstruction Farm Tier 1 Data Export 27.March 2006 Bernd Panzer-Steindel, CERN/IT

Bernd Panzer-Steindel, CERN/IT Calibration Farm Analysis Farm ALICE data flow scenario USER Data CAL Activities User Disk Buffer ESD Disk Buffer 2 CAL CAL ESD RAW RAW Tape Storage ESD Disk Buffer 3 DAQ Disk Buffer Disk Buffer 1 RAW Reconstruction Farm file aggregation layers ? Tier 1 Data Export 27.March 2006 Bernd Panzer-Steindel, CERN/IT

Bernd Panzer-Steindel, CERN/IT Boundary conditions for data flow design Network topology and connectivity Node IO performance -- network -- disk controller -- disks -- file system -- OS Storage management system Application scheduling system Experiment data management software Cost 27.March 2006 Bernd Panzer-Steindel, CERN/IT

Bernd Panzer-Steindel, CERN/IT Network boundary conditions Force10 router HP 3400 switches CPU nodes Disk server HP 3400 switch 10 Gbit 1Gbit Tape server Current Network Blocking factors : 2.4 for the disk servers = 24 server share one 10 Gbit uplink = 50 MB/s per server 2.0 for the tape server = 20 server share one 10 Gbit uplink = 60 MB/s per server 19.2 for the CPU server = 192 server share one 10 Gbit uplink = 6 MB/s per server + ~5% of CPU nodes with a b locking factor of 2.4 no magic here! throughput figures can be changed : more uplinks per switch, different daisy-chaining numbers, more switches  money issue, not architecture 27.March 2006 Bernd Panzer-Steindel, CERN/IT

Bernd Panzer-Steindel, CERN/IT Some ALICE numbers and calculations (I) CPU resource growth rate (installed capacity) (T0 + CAF) 0.65 + 0.39 MSI2000 1.3 + 1.7 MSI2000 3.3 + 5.5 MSI2000 today one CPU = 1 KSI2000, move to multi-core 1.5 KSI2000/core assume to get 4-way in 2007 and 8-way in 2008 (cores per node) jump in technology during 2006, later ‘normal’ growth rate (SI2000/core and year) 8.8 MSI2000 in 2008 == ~1300 nodes == ~ 6000 cores (mixture of 2-way, 4-way and 8-way) Reconstruction pp event size = 1.0 MB proc effort = 5400 SI2000/ev  300 KB/s IO/core Chaotic analysis pp event size = 0.04 MB proc effort = 500 SI2000/ev  120 KB/s IO/core Reconstruction HI event size = 12.5 MB proc effort = 68000 SI2000/ev  280 KB/s IO/core Chaotic analysis HI event size = 2.5 MB proc effort = 7500 SI2000/ev  500 KB/s IO/core parameter for calibration not clear (5% of the CPU resources, but no IO values available) 27.March 2006 Bernd Panzer-Steindel, CERN/IT

Bernd Panzer-Steindel, CERN/IT Some ALICE numbers and calculations (II) Disk space resource growth rate (installed capacity) (T0 + CAF) 2006 48 + 183 TB 2007 95 + 526 TB 238 + 1447 TB Today one disk is about ~400 GB and a disk server has 22 disks == ~ 5TB RAID5 distributed over three file systems extrapolating to 2008:  1685 TB == ~180 server with ~540 file systems and ~4000 disks (3000 ‘active’ + 1000 parity/spare) 6000 jobs with one input and 1-2 output streams access 180 server with 540 file systems 70-100 streams per server, 20-30 streams per file system ‘overload’ figures depend on the access model (page 9) 3000 cores running RAW reconstruction needs an aggregate of ‘only’ 1 GB/s what is the calibration doing ? much larger calibration effort in the beginning !? what are the performance deviations during the first 2 years ? 27.March 2006 Bernd Panzer-Steindel, CERN/IT

Bernd Panzer-Steindel, CERN/IT Disk server, file system performances overall network speed 120 MB/s equal performance sharing of read streams write has strong preference over read streams sharing of high speed streams and many low speed streams complicated guaranteed >= 60 MB/ tape streams ……… 27.March 2006 Bernd Panzer-Steindel, CERN/IT

Bernd Panzer-Steindel, CERN/IT Interfaces for the disk pools access 4 different data transfer activities : DAQ buffer to the T0 buffer  RFIO rfcp T0 buffer to the tape drives  RFIO rtcopy T0 buffer to the T1 sites  gridFTP SRM FTS T0 buffer to the CPU nodes  xrootd point 4 has depends actually on the work flow model -- data files are accessed (opened) from the application directly on a disk server -- data files are copied from the disk server to the local disk of the worker node Application Disk server CPU server xrootd rootd, xrootd, rfio 27.March 2006 Bernd Panzer-Steindel, CERN/IT

Bernd Panzer-Steindel, CERN/IT disk pool as a base unit to provide policies and performance values, only one data management software can have control of the pool Characterization of application activities: number of concurrent physical streams mixture ratio of read and write operations (+seeks) number of different users priority schemes between users aggregate tape writing performance guaranteed total throughput guaranteed performance per stream file size distribution access pattern, sequential versus random access data replication policies, performance implications complexity of priority and policy schemes versus performance overheads due to extra data copies 27.March 2006 Bernd Panzer-Steindel, CERN/IT

Bernd Panzer-Steindel, CERN/IT Calibration Farm Analysis Farm ALICE data flow scenario USER Data CAL User Disk Buffer ESD Disk Buffer 2 CAL CAL ESD RAW RAW Tape Storage ESD Disk Buffer 3 DAQ Disk Buffer Disk Buffer 1 RAW Reconstruction Farm file aggregation layers ? Tier 1 Data Export 27.March 2006 Bernd Panzer-Steindel, CERN/IT

Bernd Panzer-Steindel, CERN/IT  disk pool design is complex -- access patterns with boundary conditions -- flexibility -- space not ‘really’ an issue clear separation of activities (and responsibilities) performance management disk pool  defined activity mapping disentanglement reduced complexity for easier understanding/debugging cost and support issues merging later is easier 27.March 2006 Bernd Panzer-Steindel, CERN/IT