1. Advanced Computing Facility Introduction
Dr. Feng Cen
09/16/16
Modified: Yuanwei Wu, Wenchi Ma
09/12/18 1

2. Overview The Advanced Computing Facility (ACF)
houses High Performance Computing (HPC) resources dedicated to scientific research
458 nodes, 8568 processing cores and TB memory
20 nodes have over 500GB memory per node
13 nodes have 64 AMD cores per node and 109 node have 24 Intel cores per node
Coprocessor: Nvidia TitanXp, Nivida Tesla P100, Nvidia K80: 52, Nvidia K40C: 2, Nvidia K40m: 4, Nvidia K20m: 2, Nvidia M2070:1
Virtual machine operation system: Linux

3. Cluster Usage Website 3

4. Useful Links ACF Cluster computing resources
Advanced Computing Facility (ACF) documentation main page
Cluster Jobs Submission Guide
Advanced guide
ACF Portal Website
Cluster Usage Website 4

5. ACF Portal Website 5

6. ACF Portal Website Monitor jobs View cluster loads Download files
Upload files
... 6

7. Access Cluster System via Linux Terminal
Access cluster in Nichols hall
1. Login to login server → 2. Submit cluster jobs or start an interactive session from the login server .
Cluster will create a virtual machine to run your job or for your interactive session.
Access cluster from off campus
Use the KU Anywhere VPN first : anywhere-0
login1 server
or login2 server 7

8. Access Cluster System via Linux Terminal
Login to login server
Use “ssh” to directly connect to the cluster login servers: login1 or login2
Examples:
ssh login1 # login with your default linux account
ssh -X login1 # “-X” access login server with X11 forwarding
ssh # login with a different linux account
ssh -X
Login server is an entry point to the cluster and cannot support computationally intensive tasks 8

9. Access Cluster System via Linux Terminal: updated by Yuanwei @ 9/12/18
Request GPU resources on cluster
1. Reference document:
2. The GPU resources on cluster
g002, 4 k20 (4G memory per k20)
g003, 2 k k40 (4G memory per k20, 12G memory per k40)
g015, 4 k80 (12G memory per k80)
g017, 4 P100 (16G memory per P100)
g018, 4 P100 (16G memory per P100), might be reserved
g019, 2 Titanxp (12G memory per Titanxp) + 1T SSD (I saved the ImageNet12 images here for experiments)
g020, 2 Titanxp (12G memory per Titanxp)
g021, 2 Titanxp (12G memory per Titanxp) 9

10. Access Cluster System via Linux Terminal
Request GPU resources on cluster
3. The steps of requesting GPU from your local ITTC
3.1 login to 'login1' node at cluster
3.2 Load the slurm module
3.3 Load the right version of CUDA, cuDNN, python, matlab modules you want 10

11. Access Cluster System via Linux Terminal
Request GPU resources on cluster
3. The steps of requesting GPU from your local ITTC
3.4 check the usage of GPU on cluster 11

12. Access Cluster System via Linux Terminal
Request GPU resources on cluster
3. The steps of requesting GPU from your local ITTC
3.5 request GPU from cluster
Meaning of options (check the ittc cluster documentation for more details):
--gres=”gpu:gpu_name_u_request:gpu_num”
-w: select the gpu node
--mem: this specifies the requested CPU memory per node
-t: the requested time for using this GPU source to run your job, format is D- HH:MM:SS
-N: This sets the number of requested nodes for the interactive session (you can add if you want)
-n: Specifies the number of tasks or processes to run on each allocated node (you can add if you want) 12

13. Access Cluster System via Linux Terminal
Request GPU resources on cluster
3. The steps of requesting GPU from your local ITTC
3.6 check the usage on your requested GPU (or use 'watch -n 0.5 nvidia-smi' to dynamically watch the usage on GPU) 13

14. Access Cluster System via Linux Terminal
Request GPU resources on cluster
3. The steps of requesting GPU from your local ITTC
3.7 quit your GPU job
Ending by 9/12/18 14

15. Thank you ! 15