Introduction to Boston University’s Shared Computing Cluster (SCC)

Introduction to Boston University’s Shared Computing Cluster (SCC)
Aaron D. Fuegi Research Computing Services Information Services & Technology Boston University Introduce myself, Katia, and RCS (previously SCV). Distribute brochures, slide printouts, and Getting Started PDF printouts. Do you currently have an SCC account? Are you familiar with Linux? Who in the class is using a laptop and do you need help with setup to access the SCC?

Outline What is the Shared Computing Cluster (SCC)?
Information Services & Technology 11/8/2018 Outline What is the Shared Computing Cluster (SCC)? Getting an Account on the SCC Connecting to the SCC Using the SCC (Hands-On) Questions? Questions are welcome at any time. Please just raise your hand. This tutorial is scheduled for two hours but that should leave lots of time for questions. The latter part only of the tutorial will be hands-on and I will stop when we get to that point to make sure everyone can connect to the SCC from their workstation or laptop.

Information Services & Technology
11/8/2018 What Is The SCC? A Linux cluster with over 7800 processors and 240 GPUs. Currently over 2 Petabytes of disk. Located in Holyoke, MA at the Massachusetts Green High Performance Computing Center (MGHPCC), a collaboration between 5 major universities and the Commonwealth of Massachusetts. Went into production in June, 2013 for Research Computing. Continues to be updated/expanded. URLs with many slides providing much more detailed information.

Why Holyoke? – MGHPCC Benefits
Information Services & Technology 11/8/2018 Why Holyoke? – MGHPCC Benefits Green, environmentally friendly design. Low cost, clean and renewable energy source. Space on-site for building expansion (years 10-20). Opportunities for shared facilities and services. Opportunities for collaboration with other institutions. BU “Far West” – Two 10Gigabit/second Ethernet connections from BU to the MGHPCC.

11/8/2018 MGHPCC - Photo

Service Models – Shared and Buy-In
Information Services & Technology 11/8/2018 Service Models – Shared and Buy-In Many of the elements of the SCC are paid for by BU and university-wide grants and are free to the entire BU Research Computing community. Other elements (over 65% of the processors currently) are purchased by individual faculty or research groups through the Buy-In program with priority access for the purchaser.

SCC Architecture File Storage Login Nodes Compute Nodes
Information Services & Technology 11/8/2018 SCC Architecture File Storage Public Network SCC1 SCC2 GEO SCC4 VPN Only Login Nodes Login nodes. Special considerations of SCC4. Private Network Compute Nodes

11/8/2018 Storage Research projects are automatically granted 50GB of backed-up spaced (/project/PROJNAME) and 50GB of not-backed-up space (/projectnb/PROJNAME). These numbers can be increased for free to 200GB/800GB. Project groups can either purchase or “rent” additional storage. All users have a Home Directory with a 10GB quota.

11/8/2018 Storage Space (in GBs)

Storage – What files should go where?
Information Services & Technology 11/8/2018 Storage – What files should go where? Home Directory – Personal files, custom scripts. /project – Source code, files you can’t replace. /projectnb – Output files, downloaded data sets. Large quantities of data that you could recreate in the incredibly unlikely event of a disastrous data loss. /stash – Manual backup of vital /projectnb data.

Storage - Restricted (dbGaP) Data
Information Services & Technology 11/8/2018 Storage - Restricted (dbGaP) Data Some projects, mostly those on the BU Medical Campus, require dbGaP security measures: /restricted/project/PROJNAME – backed up space for dbGaP data /restricted/projectnb/PROJNAME – not backed up space for dbGaP data Only accessible through scc4.bu.edu and compute nodes

Storage – Scratch Space
Information Services & Technology 11/8/2018 Storage – Scratch Space Each node (login or compute) has a directory called /scratch stored on a local hard drive. This can be used by batch jobs to quickly write temporary files. If you wish to keep these files, you should copy them to your own space when the job completes. Scratch files are kept for 30 days, with no guarantees.

Snapshots – Recovering lost files
Information Services & Technology 11/8/2018 Snapshots – Recovering lost files Available for Home Directories, all Project Disk Space, and STASH. Backups made daily at Midnight. ~]$ cd .snapshots ~]$ ls 140613/ / ~]$ cd ~]$ ls –l -rw-r--r-- 1 adftest2 scv May 29 19:41 myfile ~]$ cp myfile ../../

Accounting – CPU Hours/SUs
Information Services & Technology 11/8/2018 Accounting – CPU Hours/SUs No monetary charges for CPU use on the SCC. Usage is tracked in Service Units (SUs). If a project exceeds its allocation, the project leader (LPI) must submit a request for additional resources. Reports on usage are mailed out monthly to all users and project leaders. Large requests require approval by the Large Allocation Review Committee (LARC). SU charge rates for SCC nodes were ‘renormalized’ on January 6, 2016, adjusting the charge rate for most SCC nodes from 2.6 to Details on this are at:

Software (Tutorial this semester)
Information Services & Technology 11/8/2018 Software (Tutorial this semester) Programming Languages: C, FORTRAN, Python, CUDA, Perl Math, Data Analysis, and Plotting: MATLAB, Mathematica, IDL, MAPLE Statistics: R, Rstudio, SAS, Stata Visualization: VTK, ParaView, VMD, Maya Domain Specific Packages: Bioinformatics, Engineering, Geographic Information Systems (GIS) Parallel: MPI, MATLAB PCT, OpenMP, OpenAcc Advertise other RCS tutorials, including the “Advanced Usage of the SCC” tutorial.

11/8/2018 GPU Computing Fast computation using GPUs (graphics processing units). 100x speedups possible for some codes. 240 GPUs available. Programming: C++ and FORTRAN - CUDA, OpenACC Software Packages: MATLAB PCT, R If interested, take one or more of our GPU tutorials.

Getting an Account on the SCC
Information Services & Technology 11/8/2018 Getting an Account on the SCC Using tutorial accounts today. These should not be used after today. All users of the SCC must be on a Research Project headed up by a full-time BU Faculty member. Exception: 3 month trial accounts for students/tutorial attendees. if interested. If you want a trial account to try out the system for up to 3 months, just let me know.

Alternative: Linux Virtual Lab
Information Services & Technology 11/8/2018 Alternative: Linux Virtual Lab Available to any BU community member that needs access to a Linux system. Send to to get access. Advantages: Permanent account Full access to SCC software via scc-lite.bu.edu Disadvantages: No batch system access Limited disk space

11/8/2018 Connecting to the SCC Windows - MobaXterm Macintosh – Built-in Terminal application Linux – Terminal application

Connecting - Details Software you need:
Information Services & Technology 11/8/2018 Connecting - Details Software you need: SSH Client – To log in to the SCC machines, such as scc1.bu.edu X Forwarding – Display graphics for those programs with a GUI interface (such as MATLAB) or that otherwise display images. File Transfer – Transferring files between the SCC and your local machine. VNC – Advanced users only. Faster graphics: For Windows, MobaXterm does all of the first three tasks in one free program. For the Mac, the Xforwarding client you need is XQuartz, which is free and dowloadable at

Questions so Far Questions on the Shared Computing Cluster so far?
Information Services & Technology 11/8/2018 Questions so Far Questions on the Shared Computing Cluster so far? Remainder of the tutorial will be hands-on getting a feel for using Linux and the SCC. If you are already familiar with Linux, this section may be slow for you.

Using the SCC (Hands-On)
Information Services & Technology 11/8/2018 Using the SCC (Hands-On) Linux “Command Line” Environment – No menus or graphics unless in specific software packages. Login Nodes - Interactive use, code development. General: scc1.bu.edu, scc2.bu.edu Earth & Environment Dept. Users: geo.bu.edu BUMC and Restricted Data Users: scc4.bu.edu Compute Nodes – Run “Batch Jobs” on, both single and multi-processor. Names like scc-bc5.bu.edu

11/8/2018 Using the SCC - Basics This tutorial is going to cover the very basics of Linux on the SCC. Please consider taking a fuller Linux tutorial from us or online if you end up using the SCC significantly. We have on our web site some material for new users of Linux and the SCC at:

11/8/2018 Using the SCC – ssh From your ssh/terminal application on your tutorial workstation or your laptop or on a machine at home: ssh -l adftest2 scc1.bu.edu “ssh” is the command you are issuing “-l adftest2” is a “command line option” to specify your login name on the SCC “scc1.bu.edu” is a “parameter” of the command Make sure to hit the “Enter” key after every command Show running MobaXterm to connect to the SCC and to do an example file transfer here or with the file transfer slide later. Help people connect to the SCC.

Using the SCC - Logging In
Information Services & Technology 11/8/2018 Using the SCC - Logging In Windows/MobaXterm local_prompt% ssh Mac local_prompt% ssh –Y Linux local_prompt% ssh –X The extra –Y for most Macintosh users and –X for most Linux users enables X Forwarding and thus the display of graphics.

SFTP File Transfer to/from the SCC
Information Services & Technology 11/8/2018 SFTP File Transfer to/from the SCC Graphical Applications Windows – MobaXterm (Free), WinSCP (Free) Mac – FileZilla (Free), Fetch (BU site license) Command Line Applications rsync scp

File Transfer Issues – dos2unix
Information Services & Technology 11/8/2018 File Transfer Issues – dos2unix Windows, Macs, and Linux in text files define “end of line” differently. To solve this issue, there is a utility called dos2unix. This is not an issue with binary files. Transfer text file “example.txt” from Windows to Linux. Rewrite “example.txt” as a Linux style file. ~]$ dos2unix example.txt Use “file example.txt” before and after to show the difference. More information at For Macs, this should only be needed for files generated using a Mac OS prior to OS X.

Using the SCC – the “prompt”
Information Services & Technology 11/8/2018 Using the SCC – the “prompt” You should now see something like: ~]$ This is what is called the “prompt” and indicates the system (the bash “shell” in particular) is ready to accept commands from you. “adftest2” is your login name. “scc1” is the machine you are on. “~” is the directory you are in – in Linux “~” is a shorthand for a person’s home directory. Differences in shells, bash vs tcsh – see Demonstrate arrow keys to access/edit previous commands and tab key for filename completion.

Using the SCC - X-Forwarding (Graphics)
Information Services & Technology 11/8/2018 Using the SCC - X-Forwarding (Graphics) Run the command xclock to see if graphics are working for you. ~]$ xclock A window similar to the image on the right should come up. Click the X in the upper right to close this window.

11/8/2018 Using the SCC – pwd Show the current “full path”, the directory you are in with its parent and all levels of grandparents up to the root directory (/). Items you type will be shown in bold: ~]$ pwd /usr2/collab/adftest2 Here the command pwd “returns” (prints to your screen) the result “/usr2/collab/adftest2”

11/8/2018 Using the SCC – man The man (short for “manual”) command is used to look up information about a Linux command. ~]$ man pwd PWD(1) User Commands PWD(1) NAME pwd - print name of current/working… SYNOPSIS pwd [OPTION]... … For some commands command_name –help may also be useful.

11/8/2018 Using the SCC – man cont. For some commands, such as if you run man cd, you will get a general manual page for the bash shell and not such a particular page as for pwd. You can page through the manual page for a command a screenful at a time using the “spacebar”, a line at a time using the “Enter” key, and quit out of the page by typing q.

11/8/2018 Using the SCC – mkdir Create a new directory: ~]$ mkdir newdir Creates a new directory (folder) to store files in within your home directory.

11/8/2018 Using the SCC – ls List the contents of a directory: ~]$ ls newdir Or with a command line option, asking for more details: ~]$ ls -l total 0 drwxr-xr-x 3 adftest2 adftest 512 Oct 28 16:03 newdir

Using the SCC – File Permissions
Information Services & Technology 11/8/2018 Using the SCC – File Permissions From the previous slide: drwxr-xr-x 3 adftest2 adftest 512 Oct 28 16:03 newdir “drwxr-xr-x” gives the “permissions” for this directory (or file). The “d” indicates this is a directory. There are then three sets of three characters for “user” (u), “group” (g), and “other” (o) access levels. “r” indicates a file/directory is readable, “w” writable, and “x” executable. A “-” indicates no such permission.

11/8/2018 Using the SCC - chmod Change the permissions on the directory “newdir” so that members of your group can write to it: ~]$ chmod g+w newdir and note the difference: ~]$ ls -l total 0 drwxrwxr-x 3 adftest2 adftest 512 Oct 28 16:03 newdir Use “chmod go-rx newdir” to take away read and execute permissions for the group and other/world.

Using the SCC – chmod cont.
Information Services & Technology 11/8/2018 Using the SCC – chmod cont. The chmod command also works with the following mappings, readable=4, writable=2, executable=1, which are combined like so: ~]$ ls –l newdir drwxrwxr-x 3 adftest2 adftest 512 Oct 28 16:03 newdir ~]$ chmod 750 newdir ~]$ ls -l newdir drwxr-x--- 3 adftest2 adftest 512 … (4+2+1=7) (0+0+0=0) (4+0+1=5)

11/8/2018 Using the SCC - cd Change directory to “newdir”: ~]$ cd newdir You can also move to other directories by giving a “full path” (a path starting with the / character) such as: newdir]$ cd /usr/local/bin/ Type just cd anytime to go back to your home directory.

Using the SCC – cp (Start C Example)
Information Services & Technology 11/8/2018 Using the SCC – cp (Start C Example) We will now begin a sequence of commands to compile and run a very simple C code. We start by copying the code from our “examples” directory into the current directory, which can be abbreviated by the . (period) character: newdir]$ cp /project/scv/examples /c/examples/ex01-helloworld/helloWorld.c . Show Examples directory on the SCC at /project/scv/examples/ or via the web at Various examples including “Hello World” examples for many popular packages.

11/8/2018 Using the SCC - more Look at the contents of the C source code file we just copied using the more command: newdir]$ more helloWorld.c #include <stdio.h> int main(int argc, char *argv[]) { /* print message */ printf("Hello, World!\n"); return (0); } Show using tab for filename completion.

11/8/2018 Using the SCC - gcc Compile the source code file we just copied into the binary file hello using the Gnu C compiler gcc: newdir]$ gcc -o hello helloWorld.c The “-o hello” option causes the output file to be named “hello”. Without this, it would be named “a.out” regardless of the name of your source code file.

Using the SCC – File Execution
Information Services & Technology 11/8/2018 Using the SCC – File Execution Note that the compiled file is automatically made “executable”: newdir]$ ls -l hello -rwxr-xr-x 1 adftest2 adftest 6430 Oct 28 15:49 hello Now we run the command from the current directory: newdir]$ hello Hello, World! With an interpreted language like Perl, you would need to make your code file executable manually using ‘chmod’.

Using the SCC – qsub and qstat
Information Services & Technology 11/8/2018 Using the SCC – qsub and qstat Use the Open Grid Scheduler (OGS) command qsub to submit our compiled program to the batch system: newdir]$ qsub -b y hello Your job ("hello") has been submitted If you are quick, you can monitor this job using qstat: newdir]$ qstat –u adftest2 job-ID prior name user state submit/start at queue … … hello adftest2 qw 09/02/ :44: …

Using the SCC – qsub output
Information Services & Technology 11/8/2018 Using the SCC – qsub output The job should run soon and produce an output file: newdir]$ cat hello.o hello, world There will also be an error file which should be empty: newdir]$ cat hello.e For a longer job, you can run “qsub ~aarondf/bin/sleepsub” which is a Perl script which simply sleeps for 60 seconds. Jobs can be deleted with the “qdel” command.

Using the SCC – qsub Details
Information Services & Technology 11/8/2018 Using the SCC – qsub Details Submit non-interactive batch jobs using qsub qsub [options] command [arguments] Setting default qsub options using a .sge_request file: You can set “default” qsub settings by creating a .sge_request file. An explanation of doing this is at:

Using the SCC – qsub options
Information Services & Technology 11/8/2018 Using the SCC – qsub options

Using the SCC – qsub options cont.
Information Services & Technology 11/8/2018 Using the SCC – qsub options cont.

Interactive Batch Jobs
Information Services & Technology 11/8/2018 Interactive Batch Jobs Used for doing interactive work, such as in MATLAB, that takes more than 15 minutes of CPU time. newdir]$ qsh Your job ("INTERACTIVE") has been submitted waiting for interactive job to be scheduled Your interactive job has been successfully scheduled. New window comes up after a little while: newdir]$ matlab Mention ‘qrsh’ alternative which does not bring up a new window. For MATLAB specifically, one should probably run matlab with the “-singleCompThread” option to avoid MATLAB automatically attempting to do parallelization which can get a user’s job killed by using more processors on the node than you requested.

Using the SCC - .bashrc file
Information Services & Technology 11/8/2018 Using the SCC - .bashrc file You have a .bashrc (.cshrc for tcsh users) in your home directory. Commands in this file are automatically executed every time you log in. Do not put commands like “echo” in this file. Modify this file to change your default system behaviors or automatically run certain commands when you log in. Be careful modifying this file or you could make it impossible for yourself to log in to the system; contact us if that happens. If you would prefer to use the Tcsh rather than the Bash shell, go to: Having commands like “echo” that do screen output in your .bashrc file may cause problems with doing file transfers; as such do not put any such commands in the file.

Using the SCC – gedit GUI Editor
Information Services & Technology 11/8/2018 Using the SCC – gedit GUI Editor Try launching a graphical application, such as gedit: newdir]$ gedit ~/.bashrc & Assuming you have X Forwarding set up, this should bring up in a separate window the simple editor gedit to enable you to edit your source code file. Other editors such as emacs and vi are also available.

Using the SCC – Editing your .bashrc
Information Services & Technology 11/8/2018 Using the SCC – Editing your .bashrc Add the following line in your .bashrc file. alias dir=‘ls-al’ Save the new .bashrc file and run in your shell window: newdir]$ source ~/.bashrc Test out the new command: newdir]$ dir hello -rwxr-xr-x 1 adftest2 adftest 6494 Jan 22 11:22 hello

Using the SCC – Modules (1)
Information Services & Technology 11/8/2018 Using the SCC – Modules (1) Modules – Used to load applications not automatically loaded by the system, including alternative versions of applications. See if I have access to the envi program: newdir]$ which envi /usr/bin/which: no envi in (/usr/local/apps/pgi-13.5/bin:/usr/java/default/jre/bin:/usr/java/default/bin:/usr/lib64/qt3.3/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/usr2/collab/adftest2/bin)

Using the SCC – Modules cont. (2)
Information Services & Technology 11/8/2018 Using the SCC – Modules cont. (2) See what other modules are available to me: newdir]$ module avail /usr/local/Modules/versions … envi/4.8 envi/5.0 envi/5.0_sp3 …

Using the SCC – Modules cont. (3)
Information Services & Technology 11/8/2018 Using the SCC – Modules cont. (3) Load the module I need from the earlier long list: newdir]$ module load envi/5.0 See if I now have access to the envi program: newdir]$ which envi alias envi='/project/earth/packages/exelis-5.0/envi50/bin/envi‘ /project/earth/packages/exelis-5.0/envi50/bin/envi Run module list to show module has been loaded. Mention default pgi/13.5 module.

11/8/2018 Using the SCC - grep grep is a useful command for searching for a text string in a file, such as: newdir]$ grep –i hello * Binary file hello matches hello.o :hello, world helloWorld.c: printf("hello, world\n"); * is a special “wildcard” character that matches all filenames. Can also limit it by doing, for example, *.c

11/8/2018 Using the SCC - Pipes You can also create a series of commands with the output of one being the input of the next through a series of “pipes” such as: newdir]$ cat helloWorld.c | grep print /* print message */ printf( "Hello, World!\n" ); You can also redirect the output of a command to a file using > myfilename

Using the SCC – R Example (1)
Information Services & Technology 11/8/2018 Using the SCC – R Example (1) Similar example to the C example earlier but using R. Copy the needed files from the examples directory: newdir]$ cp /project/scv/examples/r/examples/ex01-helloworld/* . Run this code in a variety of ways.

Using the SCC – R Example cont. (2)
Information Services & Technology 11/8/2018 Using the SCC – R Example cont. (2) Run R: newdir]$ R Within R, run the sourcecode file we just copied: > source("helloWorld.R") [1] "Hello, World!“ [1] "R version ( )“

Information Services & Technology 11/8/2018 Using the SCC – R Example cont. (3) Quit R: > q() Save workspace image? [y/n/c]: n Run the source code directly from the shell: newdir]$ R CMD BATCH helloWorld.R

Information Services & Technology 11/8/2018 Using the SCC – R Example cont. (4) Look at the generated output file: newdir]$ more helloWorld.Rout … > # This is a simple R code > > print("Hello, World!") [1] "Hello, World!“ > > # Print current R version > print(version$version.string) [1] "R version ( )“ …

Information Services & Technology 11/8/2018 Using the SCC – R Example cont. (5) Submit the prewritten script file which calls this R code to the batch system using qsub: newdir]$ qsub –P tutorial Rjob Your job ("helloWorld") has been submitted Check on the status of this job: newdir]$ qstat –u adftest2 job-ID prior name user state submit/start at queue slots ja-task-ID helloWorld adftest2 qw 08/28/ :43:

Information Services & Technology 11/8/2018 Using the SCC – R Example cont. (6) View the output file when the job is complete: newdir]$ cat helloWorld.o > # This is a simple R code > > print("Hello, World!") [1] "Hello, World!“ > > > # Print current R version > print(version$version.string) [1] "R version ( )“ > > A <- rnorm(100,0,1) > > pdf(file="mypdf.pdf") > hist(A) > dev.off() null device …

Additional Web Resources
Information Services & Technology 11/8/2018 Additional Web Resources Research Computing Support Pages Technical Summary of SCC Resources SCC Updates – Latest SCC News Code Examples for Popular Software Packages

11/8/2018 Questions All SCC questions welcome. Those I can’t answer I will make every effort to get you an answer for later. Addresses: Aaron Fuegi – General help using the SCC – Our staff of 15 people are available to assist you in porting codes, parallelization, and other aspects of using the SCC. Contact us.

Tutorial Survey Please open a web browser and go to:
Information Services & Technology 11/8/2018 Tutorial Survey Please open a web browser and go to: to fill out the tutorial survey. Thanks for coming. Tutorials slides available on the web from: My Contact Information: or (617)

Introduction to Boston University’s Shared Computing Cluster (SCC)

Similar presentations

Presentation on theme: "Introduction to Boston University’s Shared Computing Cluster (SCC)"— Presentation transcript:

Similar presentations

About project

Feedback

Log in

Auth with social network:

Introduction to Boston University’s Shared Computing Cluster (SCC)

Similar presentations

Presentation on theme: "Introduction to Boston University’s Shared Computing Cluster (SCC)"— Presentation transcript:

Similar presentations

About project

Feedback