TRIUMF Site Report for HEPiX, JLAB, October 9-13, 2006 – Corrie Kost TRIUMF SITE REPORT Corrie Kost & Steve McDonald Update since Hepix Spring 2006

TRIUMF Site Report for HEPiX, JLAB, October 9-13, 2006 – Corrie Kost Update since Hepix Spring 2006 LHC Optical Private Network Map (Sep 13/2006)

TRIUMF Site Report for HEPiX, JLAB, October 9-13, 2006 – Corrie Kost Update since Hepix Spring 2006

TRIUMF Site Report for HEPiX, JLAB, October 9-13, 2006 – Corrie Kost Update since Hepix Spring 2006 Summary of TRIUMF WAN Connections Added 2 additional 1Gb wavelengths for ATLAS Canada Tier2 sites Expect TRIUMF-CERN 10Gb lightpath Nov 1/2006 Brings total to six (6) 1Gb wavelengths & one (1) 10Gb wavelength from TRIUMF to BCnet gigapop (regional area network) Multiple wavelengths are harder to debug! New CWDM: nm Old CWDM: nm Problem with loss being wavelength sensitive (see OTDR plot below)

TRIUMF Site Report for HEPiX, JLAB, October 9-13, 2006 – Corrie Kost Update since Hepix Spring 2006

TRIUMF Site Report for HEPiX, JLAB, October 9-13, 2006 – Corrie Kost Update since Hepix Spring 2006

TRIUMF Site Report for HEPiX, JLAB, October 9-13, 2006 – Corrie Kost Update since Hepix Spring 2006 TRSHARE 4TB RAID5 (14*300GB SCSI) Dell Storage for TRSHARE Colubris for site wireless

TRIUMF Site Report for HEPiX, JLAB, October 9-13, 2006 – Corrie Kost AoE - ATA over Ethernet CORAID SR SATA EtherDrive Storage - 15 EtherDrive 3U blades, currently 8 with 750GB SATA drives - Cost ~ $4k (shell) + $4k for 8*750GB drives - 7 drives as RAID5, 1 spare - Seen by Linux as block devices References:

TRIUMF Site Report for HEPiX, JLAB, October 9-13, 2006 – Corrie Kost AoE - ATA over Ethernet Comments: - ideal for non-critical / low-cost storage - easy to configure (although web interface missing!) - handles Jumbo frames ( ifconfig ethx mtu 9000 up ) - R/W(XFS) ~ 60MB/sec (blockdev --setra 8192 /dev/etherd/eth0.0) (without above – kernels have setra 256 and achieve ~ 5MB/sec !) -ethX.Y where Y:slot 0-14 and X:chassis so…max 61,425 disks Current limit: 61425*750GB ~ 44PB for about $45million (before volume discounts & Moore’s law)

TRIUMF Site Report for HEPiX, JLAB, October 9-13, 2006 – Corrie Kost Atlas at TRIUMF CFI funded -Major purchase of cluster: Spring 2007 Oracle RAC (Real Application Cluster): Sep 2006 Oracle replication of Tier0 ATLAS data : Oct/2006 More RAC nodes & storage arrays in 2008 & 2009

TRIUMF Site Report for HEPiX, JLAB, October 9-13, 2006 – Corrie Kost Atlas at TRIUMF Power for Blades Two HP Proliant DL380 MSA20 Storage Array -12*500GB SATA HP MSA (Modular Smart Array) U 14 Blade IBM Dual CPU/Dual Core 3.0GHz Woodcrest with 8GB Memory 7U 10 Blade Dell Dual CPU/Dual Core 3.0GHz Woodcrest with 8GB Memory Two (redundant) HP StorageWorks FC Switches LCG Compute Element (Scheduler…) dCache nodes MON (RGMA) dCache Pool Nodes LFC VOBOX FTS ADMIN SRM dCache dCache Pool node Dual drive SDLT-I Dual drive SDLT-II

TRIUMF Site Report for HEPiX, JLAB, October 9-13, 2006 – Corrie Kost Atlas (ORACLE RAC) at TRIUMF RAC: Real Application Cluster Two HP Proliant DL380 Gen 5 Single CPU Dual-Core Woodcrest HP MSA(Modular Smart Array) 1500 (Fibre Channel I/O Controller MSA20 Storage Array -12*500GB Sata Two (redundant) HP StorageWorks Sanswitch 4/8 Brocade SilkWorm 200E FC Switches

TRIUMF Site Report for HEPiX, JLAB, October 9-13, 2006 – Corrie Kost Atlas at TRIUMF Move to SL4 (Oct 2006) TRIUMF (unofficial) Tier2 Members (Sep 13/2006) SFU U of Toronto U of Montreal U of Alberta U of Victoria ATLAS Milestones: Calibration Data ChallengeNovember 2006 Full Dress RehearsalSummer 2007 LHC First CollisionsNovember 2007

TRIUMF Site Report for HEPiX, JLAB, October 9-13, 2006 – Corrie Kost Atlas at TRIUMF The agreed fractions and the rates of the data to be distributed to Tier1s are as follows: Tier-1LocationFraction.RAWESDAODm1Total rate BNLBrookhaven SARAAmsterdam CCIN2P3Lyon FZKKarlsruhe RALDidcot ASGCTaipei CNAFBologna NDGF(distributed) PICBarcelona TRIUMFVancouver Total % MB/s

TRIUMF Site Report for HEPiX, JLAB, October 9-13, 2006 – Corrie Kost Atlas at TRIUMF Need to test full re-import from T0 (from a possible h/w and or s/w problem) Schedule full recovery test for applications. Perform streams re-sync recovery procedures Perform (corrupt) database recovery Details: LCG 3D Sep 13/14 workshop NEAR TERM TESTS

TRIUMF Site Report for HEPiX, JLAB, October 9-13, 2006 – Corrie Kost Atlas (hardware) at TRIUMF Blade solution favored over “pizza box” ● share a common infrastructure (chassis) ● space saving ( ~ 50% less) ● power saving ( ~ 35% less) ● cabling (power & networking, ~ 70% less) ISAC-II facility Available Floor Space: 40’ x 22’ (880sq-ft) No false floor – use of hot/cold aisles Power Estimates – to end of 2009 CPU “blades” ~ 175kW Disks ~ 95kW Tape, servers, network, etc ~ 30kW

TRIUMF Site Report for HEPiX, JLAB, October 9-13, 2006 – Corrie Kost Possible Cluster Configuration (up to 15 units / DS4500) CE: Compute Element FTS: File Transfer Service LFC: LCG File Catalog RGMA: Accounting Facility SRM: Storage Resource Manager VOBOX:ATLAS Data Management

TRIUMF Site Report for HEPiX, JLAB, October 9-13, 2006 – Corrie Kost Repeated reads on same set of (typically 16) files (at ~ 600MB/sec) – during ~ 300 days ~ 15 PB (total since started ~20PB – no reboot for >300 days)

TRIUMF Site Report for HEPiX, JLAB, October 9-13, 2006 – Corrie Kost Update since Hepix Spring 2006 Sony HD Camcorder (HDR-HC3) replaces video presenter 4 MegaPixel Stills HDV or DC 1920*1080i

TRIUMF Site Report for HEPiX, JLAB, October 9-13, 2006 – Corrie Kost Update since Hepix Spring 2006 Still: 2304 x 1768 Full image Cropped images

TRIUMF Site Report for HEPiX, JLAB, October 9-13, 2006 – Corrie Kost Misc Material….