RAL Site Report John Gordon HEPiX/HEPNT Catania 17th April 2002

John Gordon - HEPiX- n° 2 Outline u Recent hardware changes n CPU n Disk n Tape n Network & Wireless n Videoconferencing u Recent service changes n TierA Centre for BaBar n EDG Testbed1 u Issues

John Gordon - HEPiX- n° 3 Recent hardware changes u CPU u Disk u Tape u Network

John Gordon - HEPiX- n° 4 CPU u Recently started buying 1u racked dual cpu boxes n 14 dual 1GHz for EDG Testbed late 2001 n 156 dual 1.4GHz PIII in March 2002 u Large increase on existing 250 cpus n Speeds from 450MHz to 1GHz

John Gordon - HEPiX- n° 5 CPU u Recent purchase from Compusys u Result of competitive EU Tender u Chose PIII when we defined the spec last October n Worried about P4 compiler support and performance u Kickstarted into existing infrastructure so running within days

John Gordon - HEPiX- n° 6

John Gordon - HEPiX- n° 7 CPU u GHz Pentium III Tualatin cpus u 1GB of memory/dual u Internal 40GB Maxtor Viper disk u Tyan S2518 Serverworks LE Motherboard u 100Mbit Ethernet NIC n Connected to switch with multiple Gb uplinks

John Gordon - HEPiX- n° 8 Recent Tender for Disk u Delivered March 2002 so no running experience yet u Compusys the supplier u Chose SCSI/IDE RAID solution n IDE disk in RAID controller n SCSI connection to Host n No modification to host operating system n Quicker to replace host, reconfigure, add extra arrays u 26 rack-mounted Linux servers with 52 RAID Arrays n = ~ 50,000GB raw, 45TB RAID5

John Gordon - HEPiX- n° 9

John Gordon - HEPiX- n° 10 RAID Arrays & Servers RAID ControllerZero-D X-3I-R Drives Per Controller12 Drive Size/Manufacturer80GB Maxtor Viper Speed7200rpm Processors2x1.266GHz MotherboardTyan S2518 Serverworks LE Memory1GB ECC PC133 SDRAM NICIntel pro/1000T RAID Servers

John Gordon - HEPiX- n° 11 Performance u Benchmarked several of the offered solutions (IOZONE shown) n Can’t disclose benchmark results of other suppliers u Zero-D RAID controller showed the best performance, n particularly in sequential read where we have the most pressing requirement. u Infotrends 6300 also offers good performance. u The remaining systems offer only tolerable or poor performance u Benchmarking was most illuminating regarding the suppliers knowledge of the equipment, technical expertise, ability to cope under pressure and ability to provide support on their product. n This information was also fed into the tender evaluation.

John Gordon - HEPiX- n° 12 Benchmark Results SINGLE THREAD SEQUENTIAL READ (KB/s) at Varying Record Size 1K K K K THROUGHPUT TEST (aggregate KB/S) at Fixed Record Size (32K) 1 Thread Thread Thread Thread Thread Thread Read Single Array Stride Throughput Test 32K Read 15 Skip 1 Thread Threads Threads Threads Threads Threads Random IO Read

John Gordon - HEPiX- n° 13 RAID0 across 2 RAID5 Arrays Single Thread Record Size Write Read 1K K K K

John Gordon - HEPiX- n° 14 Throughput test (aggregate KB/s) at Fixed Record Size (32K) Threads Write Read

John Gordon - HEPiX- n° 15 Stride Test CMS Data read under Objectivity mimic 1000MB file 32K read. 15 Record Skip Threads Read Stride Reader (30 Record Skip) 6958 Random I/O test 1000MB file 32K read Write Read 1 Random I/O

John Gordon - HEPiX- n° 16 IDE RAID u dual 1GHz PIII system u two 3ware Escalade 7810 controllers (64bit/66MHz PCI) u 8 x 100GB maxtor disks per controller 7 per RAID 5 set 1 hot spare giving 1.1TB usable filespace in total u Bonnie performance (kernel ) MB/s RAID level Block write Block Read u Good RAID5 read performance but not good write performance

John Gordon - HEPiX- n° 17 Disk Services u Still need to investigate how best to run new disk servers u RAID0 vs RAID5 vs RAIDn u Filesystems? u Will probably try different setups for data files/databases/scratch

John Gordon - HEPiX- n° 18 Tape u Single STK Powderhorn Robot u 5632 slots – not full u 5 IBM 3590 drives (10GB) u 5 STK 9940 drives (60GB) u Currently: 3160x x9940 = 81TB u If full of 9940s = 330TB

John Gordon - HEPiX- n° 19 Network u Nortel Gbit infrastructure for whole site u 3xSummit 7i for HEP services u WAN 622Mb to local WAN, 2.5GB UK backbone u 2.5GB to RAL summer 2002 when backbone goes 10Gb u 100s of cpus access 10s of disk servers (Gb) needs:- n Very big switch as interconnect – OR n Clustering of disk servers and cpu clusters

John Gordon - HEPiX- n° 20 Wireless Networking u Installing wireless access ports in conference rooms u DHCP gives IP numbers on separate class C ‘visitors network’ n Outside firewall u Staff and other approved people can use PPTP to enter RAL site from wireless

John Gordon - HEPiX- n° 21 Video Conferencing u Conference Room ISDN u Personal VRVS n UK reflector at RAL n Ad-hoc use by bigger meeting u H.323 n Regular use in UK HEP. n Most groups have Zydacron room-based systems or Polycom ViaVideo personal systems n No production MCUs yet, using several test services

John Gordon - HEPiX- n° 22 Other Services I have concentrated on changes, we still continue to run u 2 Compaq AlphaSC systems with Quadrics switches u AMD-based Beowulf clusters u Sun cpu and disk for BaBar u IBM batch farm and fibre-based disk server for CDF

John Gordon - HEPiX- n° 23 Recent service changes u TierA Centre for BaBar u EDG Testbed1 u More about this on Friday

John Gordon - HEPiX- n° 24 Issues u Network layout cpu vs disk n Big switch with high-end backplane vs localised use u Disk Reliability n Many problems with IBM 75GB IDE n Got a batch of 60 replaced by supplier u AFS n How long can we get away with AFS? n What platform should we use for OpenAFS?