– Introduction to the Shell 1/21/2016 Introduction to the Shell – Session Introduction to the Shell – Session 3 · Job control · Start, stop and monitor running processes · Monitoring disk usage · Command line tools · grep, cut, sort, uniq, paste, fold, tr, head, tail… · Command line Perl · When command line tools don’t quite do it

– Introduction to the Shell 1/21/2016 Introduction to the Shell – Session 3 2 Seeing Running Processes: top · See a constantly updated list of what’s running with · Sorted by CPU use · Also shows available memory and processors · Processes can be controlled based on the PID (process ID) top Overall memory use Overall CPU use Process memory use Process CPU use I’m not running all that! Many processes are run by the OS (as user “root”) in the background. These processes generally have very low PIDs.

– Introduction to the Shell 1/21/2016 Introduction to the Shell – Session 3 3 Seeing Running Processes: ps · See a complete list of what’s running with · By default only shows what you are running (no root or other user’s processes) · Can select which processes to show: eg. Uby user pby PID ronly running processes · See full list of everything running with ps auxw ps

– Introduction to the Shell 1/21/2016 Introduction to the Shell – Session 3 4 Seeing Running Jobs: jobs · See currently running jobs with · These are processes that the current user is running in this terminal (does not show other user’s processes, or jobs started from other terminal windows) · Useful for job control…. jobs

– Introduction to the Shell 1/21/2016 Introduction to the Shell – Session 3 5 Putting Processes in the Background: &, bg and fg · What if you’re running a long job (eg. BLAST), but you want to use your prompt at the same time? · Put the job in the background by starting it with & · Or, once the job is running, stop it with Cntl-Z (interrupts but does not kill), then use · To put the job back in the foreground (and lose your prompt), use · To specify which job to fg or bg, use the PID or % fg bg Job ID

– Introduction to the Shell 1/21/2016 Introduction to the Shell – Session 3 6 Ending Jobs: Cntl-C, kill · To end (without saving anything) a job currently running in the terminal, use Cntl-C · To kill a job that is running in the background or from another terminal, use · If it really won’t die, try kill -9 but only as a last resort! Killing applications If you are running applications (eg. word processors, web browsers) from the command line, do NOT kill them unless you cannot exit the application normally – this can corrupt files or cause application errors. kill

– Introduction to the Shell 1/21/2016 Introduction to the Shell – Session 3 7 Sharing the CPU: nice · Jobs are assigned priority (the lower the number, the higher the priority) which determines sharing of CPU · Set the priority of your job lower when running intensive jobs so that others can use the CPU too with · Or, for running jobs, with nice renice Priority Niced

– Introduction to the Shell 1/21/2016 Introduction to the Shell – Session 3 8 Memory and Swap · Nice does not control memory use · When running memory-intensive jobs, avoid requiring swap (virtual memory – actually disk space) as this will slow all jobs on that computer

– Introduction to the Shell 1/21/2016 Introduction to the Shell – Session 3 9 Checking Disk Location and Usage: df · To confirm that the disk you are on has enough space for you to create large files: -h human-readable df

– Introduction to the Shell 1/21/2016 Introduction to the Shell – Session 3 10 Determining Disk Space Usage: du · To check total space taken by a directory structure, use · Find out what is taking up all your disk space! -hhuman-readable -cproduce total --max-depthonly print totals for this deep in directory tree du Saving space Cut down on disk usage by using du and find to find large, old files, and use gzip to compress them.

– Introduction to the Shell 1/21/2016 Introduction to the Shell – Session 3 11 Working with Text · Simple text processing is a large part of bioinformatics · Converting between program formats · Data processing · Command line tools are a fast and easy way to do a lot of text processing · Usually faster than Perl · Don’t require any coding!

– Introduction to the Shell 1/21/2016 Introduction to the Shell – Session 3 12 Searching for Text: grep · Search for text in a file with · Prints matching lines · Can do regular expression matching (slightly different from Perl) · Useful options: -icase-insensitive -wtext must appear as a “word” (surrounded by white space) -vfind non-matching lines -ccount matching lines -B print # lines before matching lines -A print # lines after matching lines -f find lines matching patterns listed in grep

– Introduction to the Shell 1/21/2016 Introduction to the Shell – Session 3 13 Sorting Text: sort · Sort a file using · Useful options: -fcase-insensitive -gsort numerically (i.e. 11<100) -rreverse sort -k sort on column # -u only print one copy of unique lines sort

– Introduction to the Shell 1/21/2016 Introduction to the Shell – Session 3 14 Working with Columns: cut and paste · Get particular columns from a file with · Have to specify which field (column) to get · Useful options: -f fields to output -dfield delimiter · Put columns back together with cut paste

– Introduction to the Shell 1/21/2016 Introduction to the Shell – Session 3 15 Finding Unique or Repeated Lines: uniq · Determine unique lines with · Collapses repeated lines · Useful options: -icase insensitive -ccount copies of each line -dprint only duplicated lines -uprint only unique lines -f do not compare the first # fields uniq Pipe it! The UNIX text tools are particularly powerful in combination – use pipes to do a series of processing commands on your text.

– Introduction to the Shell 1/21/2016 Introduction to the Shell – Session 3 16 Counting: wc · Count lines, words, and characters with · Good for checking if files have correct number of records · Useful in combination with other commands wc linesword s bytes (characte rs) How many SAGE tags match more than one gene

– Introduction to the Shell 1/21/2016 Introduction to the Shell – Session 3 17 Replacing Characters: tr · Replace characters with · Like Perl’s tr/// · Useful options: -ddelete -sreplace repeats with single occurrence -cuse complement (replace everything EXCEPT what is specified) tr Input to tr Instead of reading from a file, tr reads from standard input. If you want it to read from a file, redirect that file to standard input with “<“, or use cat and pipe the output to tr. Format comma-separated list Capitalize

– Introduction to the Shell 1/21/2016 Introduction to the Shell – Session 3 18 Formatting Text: fold · To wrap long lines in a file, use · Good for formatting before printing text files · Useful options: -wwidth to wrap to -sbreak at spaces fold

– Introduction to the Shell 1/21/2016 Introduction to the Shell – Session 3 19 Getting Parts of Files: head and tail · Print the first 10 lines with · Print the last 10 lines with · Specify how many lines to get with head eg. head > gets first 100 lines · Good for making smaller “test” files head tail Divide a file into three Make a test file

– Introduction to the Shell 1/21/2016 Introduction to the Shell – Session 3 20 Comparing files: diff · Find differences between lines in files with · Can also compare directories · Useful options: -bignore differences in white space -iignore changes in case diff Confirm file copying

– Introduction to the Shell 1/21/2016 Introduction to the Shell – Session 3 21 More advance processing: sed and awk · Simple programming languages · Designed for command-line use · Good for more advanced text processing · But I prefer….

– Introduction to the Shell 1/21/2016 Introduction to the Shell – Session 3 22 Command Line Perl · Perl can be run on a script, or directly on text input at the command line (a very short Perl script that does not have to be saved to disk) · Input file with cat or < · Options to use command line: -efind code on command line not in file -nrun command line script on each line of input · Check out other Perl command line options with man perlrun “What?” ‘Quotes?’ When running Perl on the command line, use single quotes (‘) so that the shell does not try to interpret the Perl commands. Within these quotes, you can then use double quotes (“) for eg. print statements.

– Introduction to the Shell 1/21/2016 Introduction to the Shell – Session Introduction to the Shell – Session 3 · Now you know…. · How to control your processes · How to keep up on system usage · How to play with text on the command line · How to run Perl from the command line

– Introduction to the Shell 1/21/2016 Introduction to the Shell – Session Further Readings · Job control · Linux Cookbook Ch. 4.3 · Learning the UNIX Operating System Ch. 6 · Command line goodies · Linux Cookbook Ch.13.1, · Command line Perl · Linux Cookbook Ch. 13.5, 15.5