1. Be comfortable with Linux
Linux Fundamentals
Be comfortable with Linux

2. Agenda Connectivity to the servers Profile and Environment Variables
Manipulating Files and Managing Directories
File and Directory Permissions
Important commands
Basics of vi editor
Regular Expressions
Piping output between commands
Standard Input, Output and Error
Standard Directories
Shell Scripting
Scheduling Jobs

3. What is daemon process?

4. What is default port number for SSH?

5. Connectivity to the servers
We typically use SSH or SSH based tools to connect to remote servers
Also we might have to connect from one linux server to another linux server
How SSH works?
SSH password less login

6. Setting up password less login
On the machine from where you want to connect to remote machines
Generate keys for each category of servers (ssh-keygen)
Enable passphrase for all critical categories such as production
Run ssh-copy-id (use “-i <COMPLETE_PATH_OF_KEY_FILE>” - if it is not default)
Run ssh to connect to remote machine (use “-i <COMPLETE_PATH_OF_KEY_FILE>” - if it is not default)
Default key file names are id_rsa for private key and id_rsa.pub for pulic key

7. Troubleshooting connectivity issues
Troubleshooting ssh (using telnet or nc)
Common Issues
DNS alias might have typos
DNS alias might not be mapped to correct IP address
DNS alias or IP address might be blocked by firewall
SSH service might be down
File permissions
On the source machine
.ssh should be 700
Private key should be 600
On the target machine
Also authorized_keys should be 600

8. Other uses of SSH Copying files (back and forth between servers) scp
rsync (very powerful and important)

9. Profile and Environment Variables
Command to see environment variables – env
Profile will set the defaults for a session (.profile or .bash_profile)
History of commands will be preserved in hidden file ends with history
We can use export to set environment variables such as *_HOME or update PATH for a given session
If we add same export commands to .*profile files, they will be available for all sessions

10. Manipulating Files Creating files Reading files
Empty file or update properties – touch
Use cat command to append or redirect output to a file
Using editors such as vi
Using output of a command while creating files
Reading files
view can be used to open files in read only mode
Read complete file – cat
Preview data from file – use head or tail
Copying or moving/renaming files
Copy file – cp
Move or rename file – mv
Moving only mapping between file name and location and hence it is faster
Deleting files
Delete file – rm

11. ManagingDirectories Creating Directories
mkdir
Reading contents of directories
ls, -l list, -t sort by time, -r sort in reverse
find - very important command
Copying or moving/renaming directories
Copy – cp -rf
Move or rename – mv -f
Deleting directories
Delete – rm -rf

12. File and Directory permissions
Understanding permissions – owner, group and other
Changing permission – chmod
Changing owner – chown
Better control – ACLs (Access control lists)

13. Important commands – File details
Getting help
help
man
Listing files – ls
Searching files – find

14. Important commands – Manipulating data
Filter for content from a file – grep and egrep
Sorting data – sort
Get unique records – uniq (data need to be pre-sorted to eliminate duplicates)
Delimit structured data – cut

15. Important Commands – System Details
Get details about CPUs – lscpu
Get details of memory – free
Get details of storage
Details for each mount point – df
Details for usage of a directory – du
IP Addresses configured – ip addr (OS specific)

16. Important Commands – management
To get load average – uptime
To get details about current usage of resources – top
To get process details – ps command

17. Important Commands – manipulating dates
Format options
Purpose of Option
Output
date +%a
Displays Weekday name in short (like Mon, Tue, Wed)
Thu
date +%A
Displays Weekday name in full short (like Monday, Tuesday)
Thursday
date +%b
Displays Month name in short (like Jan, Feb, Mar )
Feb
date +%B
Displays Month name in full short (like January, February)
February
date +%d
Displays Day of month (e.g., 01) 7
date +%D
Displays Current Date; shown in MM/DD/YY
02/07/13
date +%F
Displays Date; shown in YYYY-MM-DD
07/02/13
date +%H
Displays hour in (00..23) format 23
date +%I
Displays hour (01..12) format 11
date +%j
Displays day of year ( ) 38
date +%m
Displays month (01..12) 2
date +%M
Displays minute (00..59) 44
date +%S
Displays second (00..60) 17
date +%N
Displays nanoseconds ( )
date +%T
Displays time; shown as HH:MM:SS
23:44:17
Note: Hours in 24 Format
date +%u
Displays day of week (1..7); 1 is Monday 4
date +%U
Displays week number of year, with Sunday as first day of week (00..53) 5
date +%Y
Displays full year i.e. YYYY
2013
date +%Z
alphabetic time zone abbreviation (e.g., EDT) IS

18. Important Commands – miscellaneous
To get list of environment variables – env
Present working directory – pwd
To get list of active users – who
Current user – whoami
Fully qualified host name – hostname -f
Changing password – passwd (do not change password on the labs)
Killing a process – kill
Submitting a process to the background – nohup

19. Basics of vi editor Insert and Command mode Navigating Manipulating
Press esc to go to command mode (default)
Press i for insert mode or a for append mode
Navigating
Left (h), right (l), up (k) and down (j) (one at a time)
Beginning of line (^) and end of line ($) – very important
Page by page
Manipulating
Deleting one character – x and X
Replacing – r (one character), R (series of characters as we type)
Copying – yy (yank)
Deleting bulk – dd (one line), dw (one word)
Pasting – P

20. Basics of vi editor Searching Opening and managing multiple files
Saving (esc and then :x or :wq)
Discard (esc and then :!q)

21. Regular Expressions Used to filter output of commands Very wide usage
Simple commands such as ls
Search commands such as find

22. Regular Expressions Regular Expressions are used in string processing
There is significant overlap between regex and pattern matching
^ is used for beginning of the line and $ is used for end of the line
Click here for the cheatsheet.

23. Piping output between commands
Let us do some examples where data need to be piped to other commands
Typically the output of one command will be passed as input to another command
Filtering the output
Sorting
Getting unique records
And more
xargs is a special command to build execution pipelines. Output will be passed as arguments rather than input to xargs. We will use additional commands after xargs for which output will be passed as argument
Find files and delete

24. Standard Input, Output and Error
Standard Output – 1
Standard Error – 2

25. Standard Directories
/etc
/var/log
/opt
/usr
And more

26. Shell Scripting It is nothing but running series of commands
One need to have mastery over commands to excel in shell scripting

27. Basic constructs if condition
Reading input and processing using while loop
for loop
Checking if file exists

28. Shell Scripting and piped commands Examples
Get count by status and save output to a file
Run find command on root file system and
Save errors to find_errors.log
Valid output to finding_files.out
Find all the files which are greater than 10 MB in /data
Find all the files that are modified in past 24 hours
Save errors to finding-modified-files.err
Save output to finding-modified-files.out
Top 5 directories consuming space in /tmp
Top 5 users consuming space in /tmp
Number of sessions that are active per user
Top 5 users by number of sessions that are currently active
Get number of records from list of tables using sqoop eval (sqoop eval template will be provided)

29. Scheduling Jobs
Using cron