17th October 2013Graduate Lectures1 Oxford University Particle Physics Unix Overview Pete Gronbech Senior Systems Manager and GridPP Project Manager

17th October 2013Graduate Lectures2 l Strategy l Local Cluster Overview l Connecting to it l Grid Cluster l Computer Rooms l How to get help

17th October 2013Graduate Lectures3 Particle Physics Strategy The Server / Desktop Divide Win 7 PC Linux Desktop Desktops Servers General Purpose Unix Server Group DAQ Systems Linux Worker nodes Web Server Linux File Servers Win 7 PC Ubuntu PC Approx 200 Desktop PC’s with Exceed, putty or ssh/X windows used to access PP Linux systems Virtual Machine Host NIS Server torque Server

17th October 2013Graduate Lectures4 Particle Physics Linux l Unix Team (Room 661): n Pete Gronbech - Senior Systems Manager and GridPP Project Manager n Ewan MacMahon – Grid Systems Administrator n Kashif Mohammad – Grid and Local Support n Sean Brisbane – Local Server and User Support l General purpose interactive Linux based systems for code development, short tests and access to Linux based office applications. These are accessed remotely. l Batch queues are provided for longer and intensive jobs. Provisioned to meet peak demand and give a fast turnaround for final analysis. l Systems run Scientific Linux which is a free Red Hat Enterprise based distribution. l The Grid & CERN are just migrating to SL6. The local cluster is following and currently has one interactive node with a growing set of worker nodes available from "pplxint8". l Most cluster systems are still currently running SL5. These can be accessed from pplxint5 and 6. l We will be able to offer you the most help running your code on the newer SL6. Some experimental software frameworks still require SL5.

17th October 2013Graduate Lectures5 Current Clusters l Particle Physics Local Batch cluster l Oxfords Tier 2 Grid cluster

pplxwnnn 8 * Intel 5420 cores 17th October 2013 PP Linux Batch Farm pplxwn9 Scientific Linux 5 pplxint6 pplxint5 8 * Intel 5420 cores Interactive login nodes pplxwn10 8 * Intel 5420 cores pplxwnnn 8 * Intel 5420 cores pplxwnnn 8 * Intel 5420 cores pplxwnnn 8 * Intel 5420 cores 6Graduate Lectures pplxwn25 pplxwn26 pplxwn27 pplxwn28 pplxwn31 pplxwn32 pplxwn41 pplxwn42 16 * E cores 16 * Intel 5650 cores 16 * AMD Opteron 6128 cores Users log in to the interactive nodes Pplxint5 & 6, the home directories and all the data disks (/home area or /data/group ) are shared across the cluster and visible on the interactive machines and all the batch system worker nodes. Approximately 300 Cores each with 4GB of RAM memory. pplxwnnn 8 * Intel 5420 cores

17th October 2013 PP Linux Batch Farm Scientific Linux 6 pplxint8 Interactive login nodes pplxwn49 16 * Intel 2650 cores pplxwn50 16 * Intel 2650 cores pplxwnnn 16 * Intel 2650 cores 7Graduate Lectures Migration to SL6 ongoing. New SL6 interactive node pplxint8. Use this by preference. Worker nodes will be migrated from the SL5 cluster to SL6 over the next month. Currently four servers with 16 cores each with 4GB of RAM memory per core but more will arrive as required. ie 64 job slots. pplxwnnn 16 * Intel 2650 cores

17th October 2013 PP Linux Batch Farm Data Storage pplxfsn 9TB pplxfsn 40TB Data Areas pplxfsn 19TB 8 Graduate Lectures NFS Servers Home areas Data Areas NFS is used to export data to the smaller experimental groups, where the partition size is less than the total size of a server. The data areas are too big to be backed up. The servers have dual redundant PSUs, RAID 6 and are running on uninterruptible powers supplies. This safeguards against hardware failures, but does not help if you delete files. The home areas are backed up to by two different systems nightly. The OUCS HFS service and a local back up system. If you delete a file tell us a soon as you can when you deleted it and it’s full name. The latest nightly backup of any lost or deleted files from your home directory is available at the read-only location "/data/homebackup/{username} The home areas are quota’d but if you require more space ask us. Store your thesis on /home NOT /data. pplxfsn 30TB Data Areas

Particle Physics Computing Lustre MDSLustre OSS02Lustre OSS03 18TB 44TB SL5 Node SL6 Node Lustre OSS01 44TB Lustre OSS04 df -h /data/atlas Filesystem Size Used Avail Use% Mounted on /lustre/atlas 244T 215T 18T 93% /data/atlas df -h /data/lhcb Filesystem Size Used Avail Use% Mounted on /lustre/lhcb 95T 82T 8.5T 91% /data/lhcb 17th October Graduate Lectures The Lustre file system is used to group multiple file servers together to provide extremely large continuous file spaces. This is used for the Atlas and LHCb groups.

17th October 2013Graduate Lectures10

17th October 2013Graduate Lectures11 Strong Passwords etc l Use a strong password not open to dictionary attack! n fred123 – No good n Uaspnotda!09 – Much better l Better to use ssh with a passphrased key stored on your desktop.

17th October 2013Graduate Lectures12 Connecting with PuTTY Question: How many of you are using Windows? & Linux? On the desktop Demo 1. Plain ssh terminal connection 2. With key and Pageant 3. ssh with X windows tunnelled to passive exceed 4. ssh, X windows tunnel, passive exceed, KDE Session

17th October 2013Graduate Lectures13

Puttygen to create an ssh key on Windows 17th October 2013Graduate Lectures14 Paste this into ~/.ssh/authorized_keys on pplxint Enter a secure passphrase then save the public and private parts of the key to a subdirectory of your h: drive

Pageant l Run Pageant once after login to load your (windows ssh key) 17th October 2013Graduate Lectures15

17th October 2013Graduate Lectures16 SouthGrid Member Institutions l Oxford l RAL PPD l Cambridge l Birmingham l Bristol l Sussex l JET at Culham

Current capacity l Compute Servers n Twin and twin squared nodes –1300 CPU cores l Storage n Total of ~700TB n The servers have between 12 and 36 disks, the more recent ones are 3TB capacity each. These use hardware RAID and UPS to provide resilience. 17th October 2013Graduate Lectures17

17th October 2013Graduate Lectures18 Get a Grid Certificate Must remember to use the same PC to request and retrieve the Grid Certificate. The new UKCA page uses a JAVA based CERT WIZARD

17th October 2013Graduate Lectures19 Two Computer Rooms provide excellent infrastructure for the future The New Computer room built at Begbroke Science Park jointly for the Oxford Super Computer and the Physics department, provides space for 55 (11KW) computer racks. 22 of which will be for Physics. Up to a third of these can be used for the Tier 2 centre. This £1.5M project was funded by SRIF and a contribution of ~£200K from Oxford Physics. The room was ready in December Oxford Tier 2 Grid cluster was moved there during spring All new Physics High Performance Clusters will be installed here.

17th October 2013Graduate Lectures20 Local Oxford DWB Physics Infrastructure Computer Room Completely separate from the Begbroke Science park a computer room with 100KW cooling and >200KW power has been built. ~£150K Oxford Physics money. Local Physics department Infrastructure computer room. Completed September This allowed local computer rooms to be refurbished as offices again and racks that were in unsuitable locations to be re housed.

Cold aisle containment 21 17th October 2013Graduate Lectures

17th October 2013Graduate Lectures22 The end for now… l Sean will give more details of use of the clusters next week l Help Pages n n physics/particle-physics-computer-support physics/particle-physics-computer-support l n l Questions…. l Network Topology

17th October 2013Graduate Lectures23 Network l Gigabit JANET connection to campus July l Second JANET gigabit connection Sept l JANET campus connection upgraded to dual 10 gigabit links August 2009 l Gigabit Juniper firewall manages internal and external Physics networks. l 10Gb/s network links installed between Tier-2 and Tier-3 clusters in l Physics-wide wireless network. Installed in DWB public rooms, Martin Wood, AOPP and Theory. New firewall provides routing and security for this network.

17th October 2013Graduate Lectures24 Network Access Campus Backbone Router Super Janet 4 2* 10Gb/s with Janet 6 OUCS Firewall depts Physics Firewall Physics Backbone Router 1Gb/s 10Gb/s 1Gb/s 10Gb/s Backbone Edge Router depts 100Mb/s 1Gb/s depts 100Mb/s Backbone Edge Router 10Gb/s

17th October 2013Graduate Lectures25 Physics Backbone desktop Server switch Physics Firewall Physics Backbone Router 1Gb/s 100Mb/s 1Gb/s Particle Physics desktop 100Mb/s 1Gb/s 100Mb/s Clarendon Lab 1Gb/s Linux Server Win 2k Server Astro 1Gb/s Theory 1Gb/s Atmos 1Gb/s Server switch 10Gb/s Linux Server 10Gb/s Linux Server 1Gb/s

17th October Future Physics Backbone desktop Server switch Physics Firewall Physics Backbone Switch Dell 8024F 10Gb/s 1Gb/s Particle Physics Dell 8024F desktop 1Gb/s Clarendon Lab Dell 8024F 10Gb/s Win 2k Server Astro Dell 8024F 10Gb/s 1Gb/s Theory Dell 8024F 10Gb/s Atmos Dell 8024F 10Gb/s Server Switch S Gb/s Linux Server 10Gb/s Linux Server 10Gb/s Super FRODO Frodo 10Gb/s 1Gb/s Graduate Lectures