1. 1 Applied CyberInfrastructure Concepts ISTA 420/520 Fall 2015 1 Nirav Merchant (nirav@email.arizona.edu) Bio Computing & iPlant Collaborative Eric Lyons (ericlyons@email.arizona.edu) Plant Sciences & iPlant Collaborative University of Arizona http://goo.gl/p4j3mhttp://goo.gl/p4j3m or https://sites.google.com/site/appliedciconcepts/ Will Computers Crash Genomics? Science Vol 331 Feb 2011

2. Tasks for today  Log into shell.u.arizona.edu (ssh) also learn how to transfer files old school way  Shell what is it good for ?  Navigating in the shell  Working with GNU core utils  Data analysis on the command line  Building your Big Data tool kit

3. LINUX fundamentals  Ssh to shell.arizona.edu  Directory structure  Permissions  Listing, >, <, | and use of ‘ and “ and ;

4. ssh keys and managing them  Quick Intro to keys (public, private)  Where will you use these keys ?  Lets create keys to allow easier login to shell.u  Use linux/mac tutorial https://www.digitalocean.com/community/tutorial s/how-to-set-up-ssh-keys--2 https://www.digitalocean.com/community/tutorial s/how-to-set-up-ssh-keys--2  Windows users visit putty: – https://www.digitalocean.com/community/tutorials/h ow-to-create-ssh-keys-with-putty-to-connect-to-a-vps https://www.digitalocean.com/community/tutorials/h ow-to-create-ssh-keys-with-putty-to-connect-to-a-vps – https://www.digitalocean.com/community/tutorials/h ow-to-use-pageant-to-streamline-ssh-key- authentication-with-putty https://www.digitalocean.com/community/tutorials/h ow-to-use-pageant-to-streamline-ssh-key- authentication-with-putty

5. Process management  Use of &  bf, fg  Kill  nice  renice  detaching and why you need tmux (or screen)

6. GNU Core utils  http://www.gnu.org/software/coreutils/manual/ht ml_node/index.html#toc_Introduction http://www.gnu.org/software/coreutils/manual/ht ml_node/index.html#toc_Introduction  Commands we will work with  cat: Concatenate and write files  tac: Concatenate and write files in reverse  nl: Number lines and write files  head: Output the first part of files  tail: Output the last part of files  split: Split a file into pieces.  csplit: Split a file into context-determined pieces  6.1 wc: Print newline, word, and byte counts  6.2 sum: Print checksum and block counts  6.3 cksum: Print CRC checksum and byte counts  6.4 md5sum: Print or check MD5 digests

7. Hands on  http://blog.comsysto.com/2013/04/25/data- analysis-with-the-unix-shell/ http://blog.comsysto.com/2013/04/25/data- analysis-with-the-unix-shell/  How are you going to get the data from git ?  What is missing in this data set ? (how to fix ?)  Do you have access to gnuplot ?  Make a plot described in this exercise – Save the plot as pdf output – How are you going to view the pdf – Run this without interactive prompts i.e straight from command line

8. Preview pieces of toolbox http://datascienceatthecommandline.com/ We will work though Step 5 and go straight to commands We will work with csvkit today – http://csvkit.readthedocs.org/ http://csvkit.readthedocs.org/ – Download the sample data set from city country – Install pip and then install csvkit – Explore the multiple commands

10. Next class  Please practice your command line skills  Get a github account