Introduction to Linux Part 1

Content 1.Basic overview of Linux 2.The Linux ‘terminal’ and structure of commands 3.Everyday Linux commands and working with files and directories 1.The Linux survival ‘cheat-sheet’ 2.Simple exercises

What is Linux? Simply an operating system, e.g. – Windows  ? – Mac OS X Controls a computer system’s resources Allows us to use those resources while making it difficult enough to ‘break’ things

Why Linux? (Computing Perspective) FREE - can be downloaded in its entirety Portable to almost any hardware platform Made to keep running for years Secure, versatile, scalable Very few bugs and loopholes (if any) survive more than a few hours In this case, free = better quality

Why Linux? (Bioinformatics Perspective) Software developed first for Linux/Unix Convenient built-in system utilities that can be strung together for doing complex analyses Security + Scalability The ‘command line’ – best bioinformatics software first released without fancy interfaces Control + Flexibility Automation - easy to ‘pipeline’ processes

The Linux Filesystem Just like with any window-based systems, you MUST make sub- directories in your home directory to organise your work

The ‘commandline’ One of 2 user interfaces (other is the GUI) Textual ‘shell’ for issuing commands to the system and for viewing/capturing outputs Very powerful way of getting exactly what you want Working remotely on a server is typically via a command line shell Bit of a learning curve, but once it ‘clicks’ you rarely want to click anymore

Commands General structure of Linux commands: $ command -options targets Can automatically save to a file: $ command -options targets > output_file Can let multiple commands ‘play tag’ $ command -options targets | command2 | command3 > output_file NOTES: Spaces are NOT optional RARELY do we view or grab things from non-text formatted files

‘pwd’ & ‘ls’ In pure command line mode, you can’t see where you are in the file system. pwd prints your current working directory ls lists the contents of the working directory – it is the equivalent of double-clicking a folder to see what is inside

rm removes a file from the computer Typing rm myfile removes the file called ‘myfile’ NOTE: there is no ‘Trash’ or ‘Recycle Bin’, so once a file is gone, its gone for good. Very rarely will you need to do rm *, which deletes ALL files in the current folder, so always try to convince yourself NOT to

Getting information about directories pwd (print working directory) ls (list directory contents) ls -l (long listing of directory contents) ls -lh (file size units: KB, MB, GB, etc) ls -lt (sort by time created/edited) ls -ltr (same as above, but reversed) ‘Wildcards’ allowed, eg. ls -ltrh *.txt

Getting basic information about files wc counts the number of characters, words and lines in a file If used without extra options it reports all 3 For specific parts only, use: – wc -c filename (characters) – wc -w filename (words) – wc -l filename (lines) (We’ll see examples in the practical)

Creating, renaming, moving and deleting directories mkdir directory (creates a new directory) mv directory_name new_name – renames if new_name doesn’t exist – moves directory into new_name if new_name exists (NB: Will not ask first) rmdir directory_name (deletes directory ONLY if empty)

Navigating the directory structure ‘/’ is the root of the directory tree cd directory (changes your current working directory to the new directory) – cd / takes you to the root directory – cd takes you home – cd.. takes you one step ‘up’ (closer to root) – cd full-path-to-directory will take you to that directory regardless of where you currently are eg. cd /home/john/music/jack_parow/

Navigation tips Use ‘pwd’ a LOT to get your bearings A plain ‘cd’ will always take you home ‘cd -’ (minus) takes you to the previous directory you were working in Use ‘cd..’ a lot to become used to navigating from memory eg. cd../../microrarray_data/affymetrix/

Copying cp source-file(s) destination, where destination is: – another directory, eg. copying files from an archive to a working directory – a new filename, eg. cp mainfile workingfile Copying entire directories also possible – cp -fr exome1/ exome1_Mar2012/ (identical)

Working with files Rename: mv a.fastq sample_1_1.fq Moving: mv sample_1_1.fq ngs/raw/ Deleting: rm a.fastq.backup Wildcards work (but be careful!) – rm * (removes every file in a directory) – rm *.backup (only operates on files with.backup) – mv *.fastq.backup ngs/backups/

Viewing (text) file contents cat – Short for concatenate – displays contents of file(s) one after the other – writes file contents to terminal (screen) ‘Problem’: if files are bigger than 1 screen’s worth, text just goes flying past and you don’t get to see anything useful

Viewing (text) file contents (2) more – like a cat with pause – still writes to the screen, but in seqments more 1.fasta – has a pattern finding facility, which enables you to jump to first occurrence of the search string sequence: more 1.fasta, /string, for next hit

Viewing (text) file contents (3) nano – simple terminal-based text viewer/editor – basic scrolling and search functions – USAGE: nano 1.fasta – CTRL-W (to find) – CTRL-O to save – CTRL-X to exit CAREFUL working with large files…

Cheat sheets df This covers everyday Linux commands, but… The best cheatsheet is the one you create and keep adding to as you learn new ‘tricks’

3 good options for using Linux 1.Existing Windows computer: – Virtual machine – Dual boot – ‘Linux on a stick’ (See pendrivelinux.com) 2.Dedicated desktop (commodity hardware) 3.Remote server e.g. get an account at your institute or Center for High Performance Computing:

Accessing a remote server Assumes you have an account on said machine Requires a ‘client’ to talk to remote machine Linux and Mac – simply use terminal Windows – download a special program – PuTTY is a good option: –

ssh Network protocol for secure data communication between two computers that connects, via a secure channel over an insecure network Connection is between a SSH server and a SSH client Perfect for shell account access Replaces telnet (never use) which sends passwords in plaintext that can be intercepted

Example ssh session sapphire-5:~ junaid$ ssh password: Linux rocket.sanbi.ac.za server #35-Ubuntu SMP Sat Oct 16 22:02:33 UTC 2010 x86_64 GNU/Linux Welcome to the Ubuntu Server! Last login: Mon May 27 10:25: from ls 20Apr2011-Afr-vs-Rest.zipgene.rc rvm-installer bin/ KGP.bed target_intervals.bed.gz Afr-vs-All-1000G-coding-20Apr2011.txt CE Certification Policy.doc.PDF snpEfftmp Afr-vs-All-HapMap3-coding-20Apr2011.txt tophat_sample2 more Afr-vs-All-1000G-coding-20Apr2011.txt C T rs DDX11L G A rs OR4F G A rs LOC A G rs LOC T C rs LINC T C rs LINC A G rs LINC G A rs LINC T C rs LINC A G rs LINC00115

Moving files across servers Can be between your 1) personal machine and a server or 2) between two remote servers 1.Two methods Commandline (personally preferred) - ftp or sftp Graphical client, eg. – FileZilla: or – WinSCP: 2.Commandline – 2 step ssh into remote server ftp or sftp into second server and initiate transfers

Exercise (Use your notes and ‘cheatsheet’) 1.Open a terminal 2.Check that you are in your home directory 3.Create a directory called ‘exercises’ 4.Can you move this directory to the same level as your home directory? 5.Copy testfiles.zip into the new folder

6.Go into exercises (change working directory ) and uncompress the file called ‘testfiles.zip’ – $ unzip testdata.zip 7.Do a directory listing to see what happened 8.Notice a new directory? Go into it. 9.Do the following and note the differences:  cat gene_expression.diff.txt  more gene_expression.diff.txt (hit ENTER a few times, then hit the SPACEBAR)  nano gene_expression.diff.txt NOTE: copy testfiles.zip into ‘exercises’

10. Concatenate all the files as follows: cat *.txt > joined.txt 11.Make a copy of joined.txt and name it anything you want 12.Do a ‘long’ directory listing to compare sizes of old files and new file. Do the numbers add up? 13.Go back one directory level (cd..) 14. Now copy entire testdata directory as newdata

15.Compare contents of newdata and testdata 16.Try to delete testdata using rmdir – what happens? Make it work. 17.Let’s ‘walk’ through the filesystem a bit. Go into: – newdata – then straight into the root directory (single command) – then back to your home directory (single command)