CCPR Workshop Introduction to the Cluster July 13, 2006

Outline What is a cluster? How can CCPR’s cluster help you? Using the cluster Connecting from Windows Unix commands Submitting jobs Managing jobs

What is a cluster? Think of a cluster as having two parts: Hardware: a collection of computers Cluster management system: software for coordinating hardware with a collection of computational tasks (batch jobs)

CCPR’s Cluster: Management System Jobs are submitted to the cluster via the cluster management system An advanced batch system Efficiently allocates resources to jobs Remembers job requirements Remembers node specs and availability Schedules jobs according to requested resources and priority Users can easily view jobs

How does this help your research? Easy access to software and data Faster processors More RAM Easy to share data, programs, etc. with colleagues via the cluster More advanced users: parallel processing, compilers, etc. Your pc is available for other work when you submit a job to the cluster Submit a job and forget about it

Using the Cluster Connecting from Windows Unix Commands Submitting jobs Managing jobs

Secure Shell Secure Shell software (2 pieces) 1. SSH Secure Shell Client (Telnet) 2. SSH Secure File Transfer Client (FTP). Download software from  Computing  Security

Secure Shell: Connecting to lexis from Windows Connect to lexis via SSH Secure Shell Client Startup SSH Secure Shell/File Transfer Click Quick Connect Enter: Host Name: lexis.ccpr.ucla.edu User Name: your lexis account name Port Number: 22 Click connect and enter password Can add “profile” for repeated use Same process for SSH Secure File Transfer Client

Samba: Transferring files from pc to lexis Analagous to “mapping a drive” Allows user to view lexis files via Windows Explorer interface Must use vpn when you’re off campus. Download from Step-by-step instructions:

Samba: Transferring files from pc to lexis Open Windows Explorer on the local machine. Select Tools->Map Network Drive In the "Folder" text box enter: \\lexis.ccpr.ucla.edu\your-user-name Click "Reconnect at login" if you want the drive to be mounted each time the computer starts up. Click the "different user name" link to enter your user name and password for the connection. Click "OK" then click the "Finish" button on the Map Network Drive window.

Unix commands passwdChange lexis password smbpasswdChange samba password pwdPrint working directory cdChange directory mv filename1 filename2Rename/move filename1 to filename2 cp filename1 filename2Copy filename1 to filename2

More Unix Commands cat fileDisplay contents of file to screen cat file1 >> file2Concatenate file1 and file2 more fileDisplay contents of file to screen lsList files ls –lahList files in current directory (options l=LONG, a=ALL, h = HUMAN- READABLE

More Unix commands du –hDisk usage (option h=HUMAN- READABLE).forwardforward mail to another account gzipCompress file gunzipExpand compressed file ln –s actual linkCreate symbolic link which namesearch for name

Getting help in Unix man commandlist help for command man –k command keyword search for command whatis commandbrief description of command apropos keyword list commands with keyword in their description

Unix Commands: Using Aliases Alias Create an alias for commonly used commands Example: alias ls=“ls –lah” Anytime you type ls, you will see the ls command with options lah (l=long, a=all, h=human-readable) Edit.aliases file to create and remove aliases

Unix commands: Pipes and grep historydisplays a “history” of previously entered commands history | morehistory, but one page at a time history | grep catsends output of history to commnd “grep cat”, which displays only lines containing “cat” history | wcCounts number of newlines, words, and bytes contained in output of history

Editing Files on Lexis Pico – easiest editor. Open and type. Commands listed at bottom of editor. Vi – more advanced. You can move around quickly, do copy/paste, search, etc, but keystrokes required for everything. Emacs – most advanced

Submitting Stata and SAS jobs Submit jobs by typing qjob, qstata, qsas, qmatlab at Unix prompt You will then be prompted for: filename memory (number in megs, ex. 1050) if memory > 1 gig -> Over 2 hours (y/n), job type (for qjob only) submit (y/n) Alternatively, submit all options without being prompted: qstata sample.do 500 y

Files produced by qstata filename.log - This is the log file from Stata. The output of your entire Stata job is caputured in this file. filename.do.cmd - This is the Command file that is submitted to the cluster. filename.do.joblog - This file will capture the output of your CMD file and any errors that may occur (Stata output is not captured by this log.)

Job Management RAMCompute time requested Job sent to… <1 GBn/aAny node, no time constraints >2 GBn/a>4-GB node 1-2 GB<2 hours“2-hour” node, terminates after 2 hours 1-2 GB>2 hours>4-GB node

Job Management qstat or qstatus to view overall cluster usage qstat –j jobnumber for information particular to your job qdel jobnumber to kill your job

Working interactively - qinteractive To work interactively, type “qinteractive” at prompt Default RAM allotted is 256 Request x megs by typing “qinteractive x” Please use interactive nodes responsibly Limited number of spaces Testing only Logoff promptly

Questions/Comments