ARCHER Advanced Research Computing High End Resource Nick Brown nick.brown@ed.ac.uk

Website Location http://www.archer.ac.uk support@archer.ac.uk

Machine overview About ARCHER ARCHER (a Cray XC30) is a Massively Parallel Processor (MPP) supercomputer design built from many thousands of individual nodes. There are two basic types of nodes in any Cray XC30: Compute nodes (4920) These only do user computation and are always referred to as “Compute nodes” 24 cores per node, therefore approx 120,000 cores Service/Login nodes (72/8) Login nodes – allow users to log in and perform interactive tasks Other misc service functions Serial/Post-Processing Nodes (2)

Interacting with the system User guide Users do not log directly into the system. Instead they run commands via an esLogin server. This server will relay commands and information via a service node referred to as a “Gateway node” Compute node LNET Nodes Gateway esLogin Lustre OSS Cray Aries Interconnect Cray XC30 Cabinets Cray Sonnexion Filesystem External Network Infiniband links Ethernet Serial node

Job submission example Quick start guide #!/bin/bash --login #PBS -l select=2 #PBS -N test-job #PBS -A budget #PBS -l walltime=0:20:0 # Make sure any symbolic links are resolved to absolute path export PBS_O_WORKDIR=$(readlink -f $PBS_O_WORKDIR) aprun -n 48 -N 24 ./hello_world Test-job.o50818 Test-job.e50818 my_job.pbs PBS QUEUE Compute node Compute node Compute node Compute node Compute node Compute node nbrown23@eslogin008:~> qsub my_job.pbs 50818.sdb nbrown23@eslogin008:~> nbrown23@eslogin008:~> qstat –u $USER 50818.sdb nbrown23 standard test-job 29053 2 48 -- 00:20 R 00:00 nbrown23@eslogin008:~> qstat –u $USER 50818.sdb nbrown23 standard test-job -- 2 48 -- 00:20 Q --

Compute node architecture and topology ARCHER Layout Compute node architecture and topology

Cray XC30 node The XC30 Compute node features: Cray XC30 Compute Node NUMA Node 1 NUMA Node 0 Intel® Xeon® 12 Core die Aries Router Aries NIC 32GB PCIe 3.0 Aries Network QPI DDR3 The XC30 Compute node features: 2 x Intel® Xeon® Sockets/die 12 core Ivy Bridge 64GB in normal nodes 128GB in 376 “high memory” nodes 1 x Aries NIC Connects to shared Aries router and wider network

XC30 Compute Blade Cray Proprietary

Cray XC30 Rank1 Network Chassis with 16 compute blades 128 Sockets This is the All-to-all rank-1 topology. The little animation shows that for a message going from a node from Aries 3 to a node on Aries 11, the shortest route is one hop. It also illustrates how each packet can be routed adaptively on a non-minimal, 2-hop route. Point out that packets are at most 64 bytes of data and routing decisions are made on a packet-by-packet basis. Chassis with 16 compute blades 128 Sockets Inter-Aries communication over backplane Per-Packet adaptive Routing

Cray XC30 Rank-2 Copper Network 4/26/2017 Cray XC30 Rank-2 Copper Network 2 Cabinet Group 768 Sockets 6 backplanes connected with copper cables in a 2-cabinet group: This is a slide that helps you build up the topology of Dragonfly within a group. I would point out that each black wire actually represents 3 routes between the Aries. That is because the copper cables carry 3 router-tiles worth of traffic. 16 Aries connected by backplane Active optical cables interconnect groups 4 nodes connect to a single Aries

Copper & Optical Cabling Connections Copper Connections

ARCHER Filesystems Brief Overview

Nodes and filesystems Compute Nodes Login/PP Nodes RDF /home /work

ARCHER Filesystems /home (/home/n02/n02/<username>) User guide /home (/home/n02/n02/<username>) Small (200 TB) filesystem for critical data (e.g. source code) Standard performance (NFS) Fully backed up /work (/work/n02/n02/<username>) Large (>4 PB) filesystem for use during computations High-performance, parallel (Lustre) filesystem No backup RDF (/nerc/n02/n02/<username>) Research Data Facility Very large (26 PB) filesystem for persistent data storage (e.g. results) High-performance, parallel (GPFS) filesystem Backed up via snapshots

Research Data Facility RDF guide Mounted on machines such as: ARCHER (service and PP nodes) DiRAC Bluegene/Q (frontend nodes) Data Transfer Nodes (DTN) Jasmin Data Analytic Cluster (DAC) Run compute, memory, or IO intensive analyses on data hosted on the service. Nodes are specifically tailored for data intensive work with direct connections to the disks. Separate from ARCHER but very similar architecture

ARCHER Software Brief Overview

Cray’s Supported Programming Environment Programming Languages Fortran C C++ Programming models Distributed Memory (Cray MPT) MPI SHMEM PGAS & Global View UPC (CCE) CAF (CCE) Chapel Shared Memory OpenMP 3.0 OpenACC Compilers CrayPat Cray Apprentice2 Tools Environment setup Debuggers Modules Allinea (DDT) lgdb Debugging Support Tools Abnormal Termination Processing Performance Analysis STAT Scoping Analysis Reveal Optimized Scientific Libraries LAPACK ScaLAPACK BLAS (libgoto) Iterative Refinement Toolkit Cray Adaptive FFTs (CRAFFT) FFTW Cray PETSc (with CASK) Cray Trilinos (with CASK) I/O Libraries NetCDF HDF5 Cray Compiling Environment (CCE) GNU 3rd Party Compilers Intel Composer Python This is the PE Software that is packaged and shipped today. Third party compilers, such as the Intel and PGI compilers also work on the Cray, but users have to get the compiler directly from the provider. Other Libraries and tools, such as Valgrind, Vampir, work on the Cray system, but are not packaged and distributed with the Cray PE Cray developed Licensed ISV SW 3rd party packaging Cray added value to 3rd party Cray Inc. ConfidentialCray Proprietary Cray Inc. ConfidentialCray Proprietary 17

Module environment Software is available via the module environment Best practice guide Software is available via the module environment Allows you to load in different packages and different versions of packages Deals with potential library conflicts This is based around the module command List currently loaded modules: module list List all modules: module available Load a module: module load x Unload a module: module unload x

Service Administration https://www.archer.ac.uk/safe ARCHER SAFE Service Administration https://www.archer.ac.uk/safe

SAFE SAFE user guide SAFE is an online ARCHER management system which all users have an account on Request machine accounts Reset passwords View resource usage Primary way in which PIs manage their ARCHER projects Management of project users Track user’s project usage Email users of the project

Project resources Machine usage is charged in kAUs. Disk quotas User guide Machine usage is charged in kAUs. This is time running your jobs on each compute node, 0.36 kAUs for a node hour. There is no usage charge for time spent working on the login nodes, post processing nodes or RDF DAC You can track usage via the SAFE or the budgets command (calculated daily.) Disk quotas There is no specific charge made for disk usage, but all projects have quotas If you need more disk space then contact the PI or us if you manage the project

To conclude…. You will be using ARCHER during this course If you have any questions then let us know The documentation on the archer website is a good reference tool Especially the quick start guide In normal use if you have any questions or can not find something then contact the helpdesk support@archer.ac.uk