UVic Advanced Networking Day 28 November 2005 University of Victoria Research Computing Facility Colin Leavett-Brown

UVic Advanced Networking Day 28 November 2005 Introduction Established in 1999 Supports National Laboratories on Vancouver Island –NRC Herzberg Institute for Astrophysics –NRCan Pacific Forestry Centre –NRCan Pacific Geoscience Centre Broad range of research –Earth Science, Physical Sciences, Engineering –Social Science, Medical Sciences Users across Canada, US and Europe Centre of Grid Activity (Forestry and Grid Canada)

UVic Advanced Networking Day 28 November 2005 Minerva HPC (1999 CFI Award) 8 node 8 way IBM SP Upgraded to 8 node 16 way in th in the TOP 500 (highest ranking of any C3 facility?) Funded in part with an IBM Shared University Research (SUR) Award for $830,000 Primary users: cosmology, climate simulations, engineering, and geosciences Still operational and utilized today.

UVic Advanced Networking Day 28 November 2005 Storage and Cluster (2001 CFI Award) 4 year project; online in March 2003 Currently 80 TB disk and 440 TB tape capacity Primary users: Astronomy, Forestry, Particle Physics –Grid enabled for Forestry Data Grid IBM Beta Test Site for LTO2 (Jan- Apr 2003) Upgraded to LTO3 (March 2005)

UVic Advanced Networking Day 28 November 2005 Mercury & Mars Xeon Clusters Operational in March Dual Xeon blades (2.4 & 2.8 GHz) Top500 in 2003 Now have 154 Dual Xeon blades (including 3.2GHz) Primary users: particle physics, engineering, geoscience Back-filled with particle physics simulations –1 TB of data shipped to Stanford (SLAC) each week Grid-enabled and connected to Grid Canada –Particle physics and other simulation applications –Goal is to back-fill other sites for simulation production

UVic Advanced Networking Day 28 November 2005 TAPOR Project (2001 CFI Award) TAPor (Text-Analysis Portal) project will help Canadian researchers in online textual research Six leading humanities computing centres in Canada, including UVic, will contribute collections of rare tapes and documents. –ISE, Robert Graves Diary, Cowichan Dictionary – Database servers are integrated into the Storage Facility

UVic Advanced Networking Day 28 November 2005 Kikou Vector Computational Facility

UVic Advanced Networking Day 28 November 2005 Kikou Vector Computational Facility $2.42 M CFI Award (2004) for a Vector Computational Facility Purchased NEC-SX6 –$5.3M –Commissioned March 2005 –4 Nodes, 32 Processors, GFlops, 32/64GB, 10TB. –Primary use for Climate Modelling (A.Weaver)

UVic Advanced Networking Day 28 November 2005 Llaima Opteron Cluster 2004 CFI/BCKDF awards. 40 Node, Dual Opteron (2.0 Ghz), 4GB. 20/40 Commissioned September Will have direct SAN attachment. Primary use is for Astronomy (S. Ellison)

UVic Advanced Networking Day 28 November 2005 Networking Campus Networks are primarily optical networks with Cisco 6500 series routers. Connections to the access layer are generally twisted pair copper. Both a general use and a dedicated Research Network are provided (GE), interconnected via access layer. Research network used for I/O, node interconnect, and direct connections BCNet/CA*Net4.

UVic Advanced Networking Day 28 November 2005 Networking External connections: commodity network (100E), ORAN/CA*Net4 (optical & GE) through BCNet. CA*Net4/ to other national networks –Abilene, NLR, ESNET (US), GEANT (EUR), JANET (UK), JAXA (JPN), KREONET (KOR), AARNET (AUS) Lightpaths near future (Jan 06) –~1GB going to ~10GB –Optical Switching: BCNet, Member Sites, TXs. –Applications: HEPNet/C$ Testbed, CCCma, Babar/SLAC, ATLAS/TRIUMF, WG2

UVic Advanced Networking Day 28 November 2005 Infrastructure Challenges March 2002, 43KW Research Load: –March 2000, Minerva, 6KW –March 2003, Storage, 30KW –March 2003, Mercury & Mars, 40KW –April 2004, TAPoR, 5kw –March 2005, Kikou, 45 KW –September 2005, Llaima, 20KW November 2005, 150KW

UVic Advanced Networking Day 28 November 2005 Infrastructure Challenges Ran out of space, power, & A/C. $700K renovation. Increased room by 32m ². Added 44 Tons of A/C. Added 225KW UPS. Still growing, time for another renovation.

UVic Advanced Networking Day 28 November 2005 Support Managed by the University of Victoria Computer Centre –Under the Research and Administrative Systems Support Group –3 FTE dedicated to the Research Computing –Augmented by approximately 9 other sysadmins, network and hardware technicians –24/7 almost. C3 TASP Support –Position was used to assist with parallel programming support very successfully. –Position is currently vacant.

UVic Advanced Networking Day 28 November 2005 User community Minerva (~150 user accounts) –NRC, Queens, UBC, SFU, NRCan, Alberta, Washington, Dalhousie Storage Facility –HIA and PFC use facility to distribute images to their user community –TAPOR will provide a database service for social science Cluster (~60 user accounts not including Grid Canada) –NRC (Ottawa), NRCan (Geoscience and Forestry) –Montreal, UBC, TRIUMF, Alberta, NRC –Germany, UK, US (SLAC)

UVic Advanced Networking Day 28 November 2005 Plans for 2006 Storage –Currently at the beginning of 4th year –200 TB disk and 1000 TB Tape –Higher bandwidth, remove single points of failure. –Minimise ongoing maintenance costs. Mercury Cluster –50 blades per year Llaima –Commission remaining 20 nodes. –Upgrade to multi-core processors? Kikou –Another 8-way node?

UVic Advanced Networking Day 28 November 2005 Future plans Next CFI competition –Renewal of Minerva HPC –Continued support of Storage –Modest expansion of the Mercury Cluster –Upgrade of TAPoR –Part of WG2 Venus and Neptune Projects –$50 Million CFI projects to instrument the floor of the Pacific shelf –Venus (inland strait) will take data in 2005 –Neptune begins data taking in

UVic Advanced Networking Day 28 November 2005 Summary UVic RCF supports a broad range of users –A large fraction of the users are not part of the C3 community Wide range of computational and storage requirements –HPC, Cluster, Storage, Database, Grid Very successful –Operational very quickly and efficiently –2 machines in the TOP500 –Large storage repository –Willingness to experiment: Beta test site, 2 Grid modes