1. Cluster computing facility for CMS simulation work at NPD-BARC Raman Sehgal

2. Outline of the talk Example of typical cluster. Cluster types. Setup up of the Cluster Components required to make up a cluster. Selection of different Components for cluster. Overview of selected components.

3. NODE-01 NODE-02NODE-03NODE-04NODE-05NODE-06NODE-07NODE-08 NODE-09 NODE-10NODE-11NODE-12NODE-13NODE-14NODE-15NODE-16 FILE SERVER FAST / GIGABIT ETHERNET SWITCH CAT-5/6 CABLE A Typical Cluster

4. Cluster Types High Performance clusters Parallel computers and jobs Optimized for better job performance Emphasis on better interconnects High Throughput clusters Large numbers of sequential jobs Emphasis on better storage and I/O Load Balancing High Availability clusters Clusters to provide reliable service Web servers, database servers etc. Emphasis on better redundancy

5. user1user2user3 user4 Head Node (UI,PBS,NIS) Gigabit Switch Worker Nodes Cat 5/6 Cable To CMS Grid Node1 Node2 Node3 Node 4 Node 5 Node 6 Node 7 Node 8 Node 9 Node 10 Node 11 Node 12 Node 13 Node14 Node 15 20 TB Storage Box Infiniband Public N/w Private N/w Infiniband Cable Cluster Setup

6. Components that make up a cluster Nodes Compute (Worker) nodes : For running jobs Service nodes (Head Node) : Management, monitoring, User interface Storage nodes : Central storage area for users and all I/P and O/P files Network Switches Cables Support equipment Racks : House nodes and network equipment KVM switches, Terminal servers : For console sharing Power distribution units : Power sequencing and control Software Operating System Networking protocols Applications

7. Selection of Node Computing power of a cluster depends on the power of a compute node Choice of processor (Xeon, Opteron, Itanium …), cache, frequency Memory Single/Dual/Quad Processor Network ports (Single/Dual) Expansion slots (PCI, PCIe) Management (IPMI …) Form Factor (Tower, Rack mountable, Chassis, Power supply) Heat Dissipation, Cooling OS Support Desktop nodeRack Mount server

8. Selection of network switch Selection of a good interconnect is an important part in the design of any cluster. Gigabit Ethernet with copper is the standard for cluster interconnects Infiniband provides high throughput and low latency. Infiniband is useful when there is high I/O and communication overhead. Choice of 24 and 48 port stackable switches Chassis (core) switches available for large configurations (upto 672 ports) Stackable Fixed Port SwitchChassis based switch

9. Storage Box Central storage server(s) is (are) needed to store all users’ directories and all I/P and O/P files Should have high capacity. Ensure reliability using RAID. Automated backup Console Sharing Access to each node’s console (keyboard, mouse, display) is required during installation KVM Switches share a single Keyboard, Mouse, Video among multiple nodes

10. Operating System Majority of clusters nowadays run some distribution of Linux Robust Open source solution Cost-effective Support for clusters (auto installers, cluster management tools) Widely used distributions: Redhat, SuSe and Debian Other Linux distributions : Mandriva and Gentoo Redhat based distributions : Scientific Linux Cern, Centos Selection of a distribution for a cluster depends mainly on compatiability of code to be run, so for our purpose the selected distribution is Scientific Linux Cern 4.0 or higher Code to be run : CMSSW

11. Overview of selected components 1)Worker Node and Head Node Processor: Dual Intel Quad Core Xeon speed @ 3.0 GHz or higher Harpertown series with 12 MB cache or higher with 1600 MHz FSB. Memory: 16 GB 800 MHz DDR2 memory Hard Disk: 2-3 750 GB or higher Enterprise SATA II / SAS hard disks Network: 1. Two or more PXE boot compliant on-board 1000BaseT Gigabit Ethernet Ports 2. One Infiniband 4x DDR (20 Gbps) port on PCI Express x8 or higher 3. Form Factor: 1 U Rackmountable

12. 2).Storage Server : 2 Number Processor: Dual Intel Quad Core Xeon @ 2.5 GHz or higher with 12 MB cache or higher with 1333 MHz FSB, 80W per processor Memory: 16 GB, 667 MHz DIMM DDR-2 Capacity : 24 x 450 GB = 10.8 TB with SAS disks of 15 K rpm. Network:  Two PXE boot complaint on-board 1000BaseT Gigabit Ethernet Ports  Dual 10G Ethernet multimode fiber port on PCI Express  One Infiniband 4x DDR(20 Gbps) port on PCI Express Form Factor :3 / 4 U rackmountable. RAID level : Support for RAID levels 0,1,10,5,6 Server Management : Using Intelligent Platform Management Interface ( IPMI)

13. Networking Switches 3). Infiniband Switch with following specifications : No. of Ports : 24 port InfiniBand 4X DDR (20Gbps) ports Bandwidth : 960 Gbps Chassis: 1 U Rack Mountable chassis Management protocols : SNMP, Telnet, SSH, HTTP, FTP Infiniband cables : 4xDDR infiniband CX4, 30 AWG Passive cable 4).Gigabit Ethernet Switch (a) 24 port autosensing, unmanaged Gigabit Switch (b) 24 port autosensing, managed Gigabit Switch

14. 5). Rack Mountable KVM switch with following specifications No. of ports: 16 ports with PS/2 and USB keyboard and mouse support Features: Integrated with 15 inch LCD monitor, Keyboard and touch pad Form Factor: 1U with LCD panel folded 6) Uninterruptible Power Supply (UPS) Capacity:6KVA Form Factor: 3 U Rack Mountable with rail kit and internal batteries Nominal Output Voltage: 230 V @ 50 Hz Backup time: 30 minutes at half load with internal battery or higher

15. THANKS