1. Introduction to UNIX/Linux Exercises Dan Stanzione

2. Logging In To access remote systems, you need a Secure Shell (ssh) client for your system – Linux and Mac systems have these already – For Windows users, need to download a client: For this lab, I recommend “putty”, a free client (Google “putty” - fast, free, easy) – Download and install a client now

3. Logging In Once you have a client, you are ready to connect. – For this first lab, login to “ranger.tacc.utexas.edu”, a Sun/Linux supercomputer at TACC Use your assigned training ID If you don’t have an ID, tell me now. – Try logging in now… For MAC/Linux users, from the terminal type: “ssh trainXXX@ranger.tacc.utexas.edu” where XXX is your assigned accounttrainXXX@ranger.tacc.utexas.edu Windows users, click “connect” and follow the directions

4. Basic File Manipulations Exercises: – List your current files – List your files, showing their size, and all hidden files – Determine your current working directory

5. Basic File Manipulations Exercises: – Make two new files by typing the following commands: Login3$ touch foo login3$ echo “This is my data” > bar – Check the size and modified date of your new files. – Delete the file “foo” – Now, let’s create a new file “foo” with your favorite text editor… Add the contents, “This is also my data”

6. Pipes and Redirects The power of UNIX is the ability to chain commands together to do powerful things. –login3$ cat foo bar >foobar –login3$ echo “More Data” >> foobar What’s in the file “foobar”? Find instances of the word “data” –Login3$ grep data foobar What’s missing? Why? What if I want to know *how many* times the word data appears in the file? –login3$ grep data foobar | wc -l Grep can be used with Word Count to do complex functions with simple tools.

7. Simple Scripting What about “Data”? –login3$ grep -i data foobar OK, now let’s get wild… find instances of the word “data”, case insensitive, in every file in the directory: –login3$ for i in `ls`; do grep -i data $i; done – One pretty fancy line, lots to type. – Try this… type the “for I in `ls`” part, then press enter(return). ! login3$ for i in `ls` Input > do Input > grep -i data $i Input > done So, now you’re putting a whole list of commands together… would be handy to be able to save these and use them again and again… this is called scripting.

8. Simple Scripting Use your text editing skills to put all these commands in a file. Call it “datafinder” #!/bin/bash for i in `ls` do grep -i data $i done Now, let’s run it… first make it executable: ! general > chmod +x datafinder Now run it… ! general >./datafinder Now, use it in a pipe to count how many times “data” is in all your files. ! general >./datafinder | wc -l

9. Simple Scripting This ability to combine commands, through scripts and pipes, let’s you take elegant, simple tools, and build arbitrarily complex structures for manipulating files and data. No amount of windows clicks can match it (and in fact, windows has scripts too). Of course, many of the simple ones are done for you… datafinder could be replaced by a wildcard : –login3$ grep -i data * | wc -l But often, you want something more complex… Try modifying your datafinder script to print the name of the file before printing the output of “grep” on it.

10. Want to try running a job on a supercomputer? To this point, you’ve been using the Ranger login node as a simple UNIX computer. To really take advantage of it, you need to put a job on a compute node… this requires using the batch queue. To do this, you will need to write a job script (like any other script). www.iplantcollaborative.org

11. Sample Job Script #!/bin/bash #$ -N myMPI # Job Name #$ -j y # Combine stderr and stdout #$ -o $JOB_NAME.o$JOB_ID # Name of the output file #$ -pe 12way 24 # Requests 12 tasks/node, 24 cores total #$ -q normal # Queue name "normal" #$ -l h_rt=01:30:00 # Run time (hh:mm:ss) - 1.5 hours # Put the commands you want to run here./datafinder

12. Running this script To execute this job script: – Save it in a file named something like “jobscript.sh” – (or copy sample_job_script.sh from ~dstanzi/) cp ~dstanzi/sample_job_script.sh./jobscript.sh –login3$ qsub jobscript This will run the job. www.iplantcollaborative.org

13. Then what? Check your job status: – “showq” – (this is a good time to use those “grep” skills. Note your job has a number! – Find your output file in a file named: Jobname.jobnumber – Each time you run it, you will get a unique output file. – You may have to wait a while if the machine is busy (Ranger only has 62,976 processors). www.iplantcollaborative.org

14. Notes In a real job script, you would add a line to bill to your project, especially if you have more than one (not needed today). – #$ -A IPLANT You don’t really need 24 processors to run “datafinder” – In a real large scale job, you will use the MPI launcher; instead of the “./datafinder” line use: ibrun./my_program www.iplantcollaborative.org

15. How do I learn More? The User Guide – http://services.tacc.utexas.edu/index.php/lonestar- user-guide http://services.tacc.utexas.edu/index.php/lonestar- user-guide Training Classes – Check full listings at: http://www.tacc.utexas.edu/user-services/training/ http://www.tacc.utexas.edu/user-services/training/ Online Training – http://www.cac.cornell.edu/ranger/ http://www.cac.cornell.edu/ranger/ Advanced User Support

16. How do I get help? The User Portals – https://portal.tacc.utexas.edu/gridsphere/grids phere https://portal.tacc.utexas.edu/gridsphere/grids phere – https://portal.teragrid.org/ https://portal.teragrid.org/ – For most routine requests, use the portal to submit a ticket; you will be contacted quickly (tickets monitored 24x7). Password help, general inquiries, software installs, requests for consulting, allocation questions etc.