Planning the LCG Fabric at CERN openlab TCO Workshop November 11 th 2003 CERN.ch.

Slides:



Advertisements
Similar presentations
IBM Software Group ® Integrated Server and Virtual Storage Management an IT Optimization Infrastructure Solution from IBM Small and Medium Business Software.
Advertisements

Distributed Data Processing
Distributed Processing, Client/Server and Clusters
Hardware & the Machine room Week 5 – Lecture 1. What is behind the wall plug for your workstation? Today we will look at the platform on which our Information.
Introduction to Storage Area Network (SAN) Jie Feng Winter 2001.
MUNIS Platform Migration Project WELCOME. Agenda Introductions Tyler Cloud Overview Munis New Features Questions.
Introduction to DBA.
Chapter 5: Server Hardware and Availability. Hardware Reliability and LAN The more reliable a component, the more expensive it is. Server hardware is.
LHC experimental data: From today’s Data Challenges to the promise of tomorrow B. Panzer – CERN/IT, F. Rademakers – CERN/EP, P. Vande Vyvre - CERN/EP Academic.
6/2/2015Bernd Panzer-Steindel, CERN, IT1 Computing Fabric (CERN), Status and Plans.
Network Management Overview IACT 918 July 2004 Gene Awyzio SITACS University of Wollongong.
12. March 2003Bernd Panzer-Steindel, CERN/IT1 LCG Fabric status
Storage area Network(SANs) Topics of presentation
Copyright 2009 FUJITSU TECHNOLOGY SOLUTIONS PRIMERGY Servers and Windows Server® 2008 R2 Benefit from an efficient, high performance and flexible platform.
Server Platforms Week 11- Lecture 1. Server Market $ 46,100,000,000 ($ 46.1 Billion) Gartner.
The CERN Computer Centres October 14 th 2005 CERN.ch.
Router Architectures An overview of router architectures.
Virtual Network Servers. What is a Server? 1. A software application that provides a specific one or more services to other computers  Example: Apache.
Router Architectures An overview of router architectures.
11 SERVER CLUSTERING Chapter 6. Chapter 6: SERVER CLUSTERING2 OVERVIEW  List the types of server clusters.  Determine which type of cluster to use for.
Simplify your Job – Automatic Storage Management Angelo Session id:
EU funding for DataGrid under contract IST is gratefully acknowledged GridPP Tier-1A Centre CCLRC provides the GRIDPP collaboration (funded.
CHAPTER 11: Modern Computer Systems
Bob Thome, Senior Director of Product Management, Oracle SIMPLIFYING YOUR HIGH AVAILABILITY DATABASE.
Online Systems Status Review of requirements System configuration Current acquisitions Next steps... Upgrade Meeting 4-Sep-1997 Stu Fuess.
ASGC 1 ASGC Site Status 3D CERN. ASGC 2 Outlines Current activity Hardware and software specifications Configuration issues and experience.
CHAPTER 11: Modern Computer Systems
IT Infrastructure Chap 1: Definition
IMPROUVEMENT OF COMPUTER NETWORKS SECURITY BY USING FAULT TOLERANT CLUSTERS Prof. S ERB AUREL Ph. D. Prof. PATRICIU VICTOR-VALERIU Ph. D. Military Technical.
Building A Computer Centre HEPiX Large Cluster SIG October 21 st 2002 CERN.ch.
1 Computer and Network Bottlenecks Author: Rodger Burgess 27th October 2008 © Copyright reserved.
LAN Switching and Wireless – Chapter 1
Physical Infrastructure Issues In A Large Centre July 8 th 2003 CERN.ch.
RAL Site Report Andrew Sansum e-Science Centre, CCLRC-RAL HEPiX May 2004.
RAL Site Report John Gordon IT Department, CLRC/RAL HEPiX Meeting, JLAB, October 2000.
10/22/2002Bernd Panzer-Steindel, CERN/IT1 Data Challenges and Fabric Architecture.
CERN - IT Department CH-1211 Genève 23 Switzerland The Tier-0 Road to LHC Data Taking CPU ServersDisk ServersNetwork FabricTape Drives.
JLAB Computing Facilities Development Ian Bird Jefferson Lab 2 November 2001.
Computer Centre Upgrade Status & Plans Post-C5, June 27 th 2003 CERN.ch.
CERN IT Department CH-1211 Genève 23 Switzerland Introduction to CERN Computing Services Bernd Panzer-Steindel, CERN/IT.
S.Jarp CERN openlab CERN openlab Total Cost of Ownership 11 November 2003 Sverre Jarp.
CERN.ch 1 Issues  Hardware Management –Where are my boxes? and what are they?  Hardware Failure –#boxes  MTBF + Manual Intervention = Problem!
11 CLUSTERING AND AVAILABILITY Chapter 11. Chapter 11: CLUSTERING AND AVAILABILITY2 OVERVIEW  Describe the clustering capabilities of Microsoft Windows.
Managing the CERN LHC Tier0/Tier1 centre Status and Plans March 27 th 2003 CERN.ch.
Phase II Purchasing LCG PEB January 6 th 2004 CERN.ch.
IDE disk servers at CERN Helge Meinhard / CERN-IT CERN OpenLab workshop 17 March 2003.
VMware vSphere Configuration and Management v6
Cluster Configuration Update Including LSF Status Thorsten Kleinwort for CERN IT/PDP-IS HEPiX I/2001 LAL Orsay Tuesday, December 08, 2015.
1 LHCC RRB SG 16 Sep P. Vande Vyvre CERN-PH On-line Computing M&O LHCC RRB SG 16 Sep 2004 P. Vande Vyvre CERN/PH for 4 LHC DAQ project leaders.
CERN Computer Centre Tier SC4 Planning FZK October 20 th 2005 CERN.ch.
23.March 2004Bernd Panzer-Steindel, CERN/IT1 LCG Workshop Computing Fabric.
Building and managing production bioclusters Chris Dagdigian BIOSILICO Vol2, No. 5 September 2004 Ankur Dhanik.
David Foster LCG Project 12-March-02 Fabric Automation The Challenge of LHC Scale Fabrics LHC Computing Grid Workshop David Foster 12 th March 2002.
CNAF Database Service Barbara Martelli CNAF-INFN Elisabetta Vilucchi CNAF-INFN Simone Dalla Fina INFN-Padua.
Computer Centre Upgrade Status & Plans Post-C5, October 11 th 2002 CERN.ch.
1 CEG 2400 Fall 2012 Network Servers. 2 Network Servers Critical Network servers – Contain redundant components Power supplies Fans Memory CPU Hard Drives.
Vault Reconfiguration IT DMM January 23 rd 2002 Tony Cass —
Virtual Server Server Self Service Center (S3C) JI July.
Dominique Boutigny December 12, 2006 CC-IN2P3 a Tier-1 for W-LCG 1 st Chinese – French Workshop on LHC Physics and associated Grid Computing IHEP - Beijing.
26. Juni 2003Bernd Panzer-Steindel, CERN/IT1 LHC Computing re-costing for for the CERN T0/T1 center.
CERN Disk Storage Technology Choices LCG-France Meeting April 8 th 2005 CERN.ch.
Bernd Panzer-Steindel CERN/IT/ADC1 Medium Term Issues for the Data Challenges.
CHAPTER 11: Modern Computer Systems
CASTOR: possible evolution into the LHC era
LHC Computing re-costing for
GridPP Tier1 Review Fabric
Bernd Panzer-Steindel CERN/IT
OffLine Physics Computing
Web Server Administration
The Problem ~6,000 PCs Another ~1,000 boxes But! Affected by:
Presentation transcript:

Planning the LCG Fabric at CERN openlab TCO Workshop November 11 th 2003 CERN.ch

CERN.ch 2 Fabric Area Overview Infrastructure Electricity, Cooling, Space Infrastructure Electricity, Cooling, Space Network Batch system (LSF, CPU server) Batch system (LSF, CPU server) Storage system (AFS, CASTOR, disk server) Storage system (AFS, CASTOR, disk server) Purchase, Hardware selection, Resource planning Purchase, Hardware selection, Resource planning Installation Configuration + monitoring Fault tolerance Installation Configuration + monitoring Fault tolerance Prototype, Testbeds Benchmarks, R&D, Architecture Benchmarks, R&D, Architecture Automation, Operation, Control Coupling of components through hardware and software GRID services !?

CERN.ch 3 Agenda  Building Fabric  Batch Subsystem  Storage subsystem  Installation and Configuration  Monitoring and control  Hardware Purchase

CERN.ch 4 Agenda  Building Fabric  Batch Subsystem  Storage subsystem  Installation and Configuration  Monitoring and control  Hardware Purchase

CERN.ch 5 Building Fabric — I  B513 was constructed in the early 1970s and the machine room infrastructure has evolved slowly over time. –Like the eye, the result is often not ideal…

CERN.ch 6 Current Machine Room Layout Problem: Normabarres run one way, services run the other…. Services

CERN.ch 7 Building Fabric — I  B513 was constructed in the early 1970s and the machine room infrastructure has evolved slowly over time. –Like the eye, the result is often not ideal…  With the preparations for LHC we have the opportunity to remodel the infrastructure.

CERN.ch box PCs 105kW U PCs 288kW 324 disk servers 120kW(?) Future Machine Room Layout 18m double rows of racks 12 shelf units or 36 19” racks 9m double rows of racks for critical servers Aligned normabarres

CERN.ch 9 Building Fabric — I  B513 was constructed in the early 1970s and the machine room infrastructure has evolved slowly over time. –Like the eye, the result is often not ideal…  With the preparations for LHC we have the opportunity to remodel the infrastructure. –Arrange services in clear groupings associated with power and network connections. »Clarity for general operations plus ease of service restart should there be any power failure. –Isolate critical infrastructure such as networking, mail and home directory services. –Clear monitoring of planned power distribution system.  Just “good housekeeping”, but we expect to reap the benefits during LHC operation.

CERN.ch 10 Building Fabric — II  Beyond good housekeeping, though, there are building fabric issues that are intimately related with recurrent equipment purchase. –Raw power: We can support a maximum equipment load of 2.5MW. Does the recurrent additional cost of blade systems avoid investment in additional power capacity? –Power efficiency: Early PCs had power factors of ~0.7 and generated high levels of 3 rd harmonics. Fortunately, we now see power factors of 0.95 or better, avoiding the need to install filters in the PDUs. Will this continue? –Many sites need to install 1U or 2U rack mounted systems for space reasons. This is not a concern for us at present but may become so eventually. »There is a link here to the previous point: the small power supplies for 1U systems often have poor power factors.

CERN.ch 11 Agenda  Building Fabric  Batch Subsystem  Storage subsystem  Installation and Configuration  Monitoring and control  Hardware Purchase

CERN.ch 12 Fabric Architecture Level of complexity Batch system, load balancing, Control software, Hierarchical Storage Systems Hardware Software CPU Physical and logical coupling Disk PC Storage tray, NAS server, SAN element Storage tray, NAS server, SAN element Motherboard, backplane, Bus, integrating devices (memory,Power supply, controller,..) Operating system, driver Network (Ethernet, fibre channel, Myrinet, ….) Hubs, switches, routers Cluster World wide cluster Grid middleware Wide area network

CERN.ch 13

CERN.ch 14

CERN.ch 15 Batch Subsystem  Looking purely at batch system issues, TCO is reduced as the efficiency of node usage increases. What are the dependencies? –The load characteristics –The batch scheduler –Chip technology –Processors/box –The operating system –Others?

CERN.ch 16 Batch Subsystem  Looking purely at batch system issues, TCO is reduced as the efficiency of node usage increases. What are the dependencies? –The load characteristics »Not much we in IT can do here! –The batch scheduler –Chip technology –Processors/box –The operating system –Others?

CERN.ch 17 Batch Subsystem  Looking purely at batch system issues, TCO is reduced as the efficiency of node usage increases. What are the dependencies? –The load characteristics –The batch scheduler »LSF is pretty good here, fortunately. –Chip technology –Processors/box –The operating system –Others?

CERN.ch 18 Batch Subsystem  Looking purely at batch system issues, TCO is reduced as the efficiency of node usage increases. What are the dependencies? –The load characteristics –The batch scheduler –Chip technology »Take hyperthreading, for example. Tests have shown that, for HEP codes at least, hyperthreading wastes 20% of the system performance running two tasks on a dual processor machine. There are no clear benefits to running with hyperthreading enabled when running three tasks. What is the outlook here? –Processors/box –The operating system –Others?

CERN.ch 19 Batch Subsystem  Looking purely at batch system issues, TCO is reduced as the efficiency of node usage increases. What are the dependencies? –The load characteristics –The batch scheduler –Chip technology –Processors/box »At present, a single 100baseT NIC would support the I/O load of a quad processor CPU server. Quad processor boxes would halve the cost of networking infrastructure—but they come at a hefty price premium (XEON MP vs XEON DP, heftier chassis, …). What is the outlook here? u And total system memory becomes an issue. –The operating system –Others?

CERN.ch 20 Batch Subsystem  Looking purely at batch system issues, TCO is reduced as the efficiency of node usage increases. What are the dependencies? –The load characteristics –The batch scheduler –Chip technology –Processors/box –The operating system »Linux is getting better, but things such as processor affinity would be nice. u Relationship to hyperthreading… –Others?

CERN.ch 21 Batch Subsystem  Looking purely at batch system issues, TCO is reduced as the efficiency of node usage increases. What are the dependencies? –The load characteristics –The batch scheduler –Chip technology –Processors/box –The operating system –Others?

CERN.ch 22 Agenda  Building Fabric  Batch Subsystem  Storage subsystem  Installation and Configuration  Monitoring and control  Hardware Purchase

CERN.ch 23 Storage subsystem Processors  “desktop+” node == CPU server CPU server + larger case + 6*2 disks == Disk server CPU server + Fiber Channel Interface + tape drive == Tape server  Simple building blocks:

CERN.ch 24

CERN.ch 25

CERN.ch 26 Storage subsystem — Disk Storage  TCO: Maximise available online capacity within fixed budget (material & personnel). –IDE based disk servers are much cheaper than high end SAN servers. But are we spending too much time on maintenance? »Yes, at present, but we need to analyse carefully the reasons for the current load. u Complexities of Linux drivers seem under control, but numbers have exploded. And are some problems related to batch of hardware? –Where is the optimum? Switching to fibre channel disks would reduce capacity by factor of ~5. »Naively, buy, say, 10% extra systems to cover failures. Sadly, this is not as simple as for CPU servers; active data on the servers must be reloaded elsewhere. »Always have duplicate data? => purchase 2x required space. Still cheaper than SAN? How does this relate to …

CERN.ch 27 Storage System — Tapes  The first TCO question is “Do we need them?”  Disk storage costs are dropping…

CERN.ch 28 Disk Price/Performance Evolution

CERN.ch 29 Storage System — Tapes  The first TCO question is “Do we need them?”  Disk storage costs dropping… But –Disk servers need system administrators, idle tapes sitting in a tape silo don’t. –With disk only solution, we need storage for at least twice the total data volume to ensure no data loss. –Server lifetime of 3-5 years; data must be copied periodically. »Also an issue for tape, but the lifetime of a disk server is probably still less than the lifetime of a given tape media format.  Assumption today is that tape storage will be required.

CERN.ch 30 Storage System — Tapes  Tape robotics is easy. –Bigger means better cost/slot.

CERN.ch 31

CERN.ch 32 Storage System — Tapes  Tape robotics is easy. –Bigger means better cost/slot.  Tape drives: High end vs LTO –TCO issue: LTO drives are cheaper than high end IBM and STK drives, but are they reliable enough for our use? »c.f. the IDE disk server area.  Real problem, though is tape media. –Vast portion of the data is accessed rarely but must be stored for long period. Strong pressure to select a solution that minimises an overall cost dominated by tape media.

CERN.ch 33 Storage System — Managed Storage  Should CERN build or buy software systems?  How to measure the value of a software system? –Initial cost: »Build: Staff time to create required functionality »Buy: Initial purchase cost of system as delivered plus staff time to install and figure for CERN. –Ongoing cost »Build: Staff time to maintain system and add extra functionality »Buy: License/maintenance cost plus staff time to track releases. u Extra functionality that we consider useful may or may not arrive.  Choice: –Batch system: Buy LSF. –Managed storage system: Build CASTOR.  Use this model as we move on to consider system management software.

CERN.ch 34 Agenda  Building Fabric  Batch Subsystem  Storage subsystem  Installation and Configuration  Monitoring and control  Hardware Purchase

CERN.ch 35 Installation and Configuration  Reproducibility and guaranteed homogeneity of system configuration is a clear method to minimise ongoing system management costs. A management framework is required that can cope with the numbers of systems we expect.  We faced the same issues as we moved from mainframes to RISC systems. Vendor solutions offered then were linked to hardware—so we developed our own solution.  Is a vendor framework acceptable if we have a homogeneous park of Linux systems? –Being honest, why have we built our own again?

CERN.ch 36 Installation and Configuration  Installation and configuration is only part of the overall computer centre management:

CERN.ch 37 ELFms architecture Node Configuration System Monitoring System Installation System Fault Mgmt System

CERN.ch 38 Installation and Configuration  Installation and configuration is only part of the overall computer centre management:  Systems provided by vendors cannot (yet) be integrated into such an overall framework.  And there is still a tendency to differentiate products on the basis of management software, not raw hardware performance. –This is a problem for us as we cannot ensure we always buy brand X rack mounted servers or blade systems. –In short, life is not so different from the RISC system era.

CERN.ch 39 Agenda  Building Fabric  Batch Subsystem  Storage subsystem  Installation and Configuration  Monitoring and control  Hardware Purchase

CERN.ch 40 Monitoring and Control  Assuming that there are clear interfaces, why not integrate a commercial monitoring package into our overall architecture?  Two reasons: –No commercial package meets (met) our requirements in terms of, say, long term data storage and access for analysis. »This could be considered self serving: we produce requirements that justify a build rather than buy decision. –Experience has show, repeatedly, that monitoring frameworks require effort to install and maintain, but don’t deliver the sensors we require. »Vendors haven’t heard of LSF, let alone AFS. »A good reason!

CERN.ch 41 Hardware Management System  A specific example of the integration problem. Workflows must interface to local procedures for, e.g., LAN address allocation. Can we integrate a vendor solution? Do complete solutions exist?

CERN.ch 42 Console Management  Done poorly now:

CERN.ch 43  We will do better: TCO issue: Do the benefits of a single console management system outweigh costs of developing our own? How do we integrate vendor supplied racks of preinstalled systems? Console Management

CERN.ch 44 Agenda  Building Fabric  Batch Subsystem  Storage subsystem  Installation and Configuration  Monitoring and control  Hardware Purchase

CERN.ch 45 Hardware Purchase  The issue at hand: How do we work within our purchasing procedures to purchase equipment that minimises our total cost of ownership?  At present, we eliminate vast areas of the multi- dimensional space by assuming we will rely on ELFms for system management and Castor for data management. Simplified[!!!] view: –CPU: White box vs 1U vs blades; install or ready packaged –Disk: IDE vs SAN; level of vendor integration  HELP!  Can we benefit from management software that comes with ready built racks of equipment in a multi-vendor environment?