ITEC400 Week Six Professor Robert D’Andrea

Agenda Administrative Issues This Week’s Expected Outcomes Review for midterm exam Moving around in UNIX Break-Out Problems Upcoming Deadlines Hands-on Information Lab Assistance, Questions, and Answers

Administrative Midterm exam this week February 5 through 10, 2018. No script logic for Lab Assignments 8-1 and 9-1. Script logic will be emailed out for Lab Assignments 10-1 and 10-2.

This Week’s Expected Outcomes Upon successful completion of this module, the student will be able to: Be current on one of the latest memory technologies, SSD. Review package manager and manual commands. NAND memory characteristics Regular expressions Linux package management and it’s characteristics Tar command functionality

Unix Operating System http://www.kernel.org/ The Linux Kernel Archives web site http://www.kernelnewbies.org/ This web site is a community of people that want to improve or update their kernels. This site is referenced by experienced developers willing to share their knowledge with newcomers. The Kernel is the central component of most operating systems. The kernel is responsible for managing the system’s resources and the communications between hardware and software.

Unix Operating System Define: Kernel space is space allocated specifically for the kernel. Users are not able to access this area. This area is generally larger than user space. The kernel is never swapped out to disk. Define: User space is space in memory where all user mode applications execute. Applications executing in this space are subject to being swapped out to disk. Define: monolithic kernel is where all code is executed in kernel addresses space to increase the performance of the system. Define: microkernel uses system calls to implement minimal OS services such as memory management, multitasking, and inter process communication.

Unix Operating System http://vista.intersystems.com/csp/docbook/ DocBook.UI.Page.cls?KEY=GCI_unixparms #GCI_unixparms_notes_hpux Calculating System Parameters for UNIX and Linux Configure Unix Kernel Turnable Parameters Swap space Number of global/routine buffers Number of users Number of inodes Maximum database size Number of semaphores

UNIX Operating System Solid State Drive (SSD) or Electronic Disk The SSD actually does not have an actual disk or motors to drive the disks. There are no moving parts inside this device. It is a data storage device that uses integrated circuit assemblies as memory to store data persistently. The SSD technology uses electronic interfaces compatible with traditional block input and output HDD, which permits simple replacement in common applications. A new input and output interface like SATA Express was created to keep pace with the speed advancements in the SSD technology.

UNIX Operating System SSD features When compared to electromechanical disks, SSD are more resistant to physical shock, operates silently, has a lower access time, and has less latency. Since 2012, the price of SSDs has declined. SSDs are approximately 7 to 8 times more expensive per unit of storage than HDDs.

UNIX Operating System As of 2010, the majority of SSDs used NAND-based flash memory, which retains the stored data without power. For applications requiring fast access, but not persistent memory after a power loss or crash, SSDs can be constructed from RAM. Such devices could have separate power supplies or batteries, to maintain data after power loss.

UNIX Operating System

UNIX Operating System

UNIX Operating System NAND memory NAND is a flash based memory introduced by Toshiba in 1989. SSDs behave similar to HDDs that we are familiar with, like accessing data on a block device. Each block consists of a number of pages. Page sizes range from 512 or 2,048 or 4096 bytes.

Standardization of NAND UNIX Operating System Standardization of NAND The group called the Open NAND Flash Interface Working Group (ONFI) developed a standardized low-level interface for NAND flash chips. This allows for interoperability between conforming NAND devices from different vendors. The ONFI released their version 1 .0 on ONFI is supported by major NAND flash manufactures December 2006. Toshiba is one of those manufactures, but has chosen to use an interface of their own design known as Toggle Mode ( and now Toggle V2.0). This interface is not directly, pin and pin, which is compatible with the ONFI specification.

UNIX Operating System NAND characteristics NAND flash memory is a non-volatile memory technology that does not require power to retain data. A non-volatile storage is one where stored data can be retrieved even when not powered on. NAND flash is an electronically addressed system, rather than mechanical. Electrically addressed systems are more expensive, faster, where mechanical addressed systems are the inverse.

UNIX Operating System NAND characteristics One important goal of NAND flash development has been to reduce the cost per bit and increase chip capacity so that flash memory can compete with magnetic storage devices like hard drives.

UNIX Operating System NAND market NAND flash technology has a market in devices where large files are uploaded frequently and replaced. The digital cameras, USB drives, and MP-3 players all use NAND flash memory.

UNIX Operating System Down-side of NAND flash memory The NAND flash has a finite number of write cycles. Another term for write cycle is called program/erase (P/E) cycles. Overtime, NAND fails gradually as individual cells fail and performance becomes noticeably slower. To compensate for this shortcoming, some vendors include more memory in their devices than actually claimed.

UNIX Operating System Regular expression Regular expression is a sequence of characters that form a search pattern. This is similar to the principle of the wildcards that can be used to specify multiple file names. The grep command searches for files that contain a specific string and returns the name of the file and line of context for that string.

grep command /export/home/dandrear/Fall_2017_Solutions>grep if printnum.sh if [ $# -ne 1 ] elif [ $NUMBER -le 0 ] # Test value of argument is less than or equal to zero if [ $NUMBER -gt 1 ]

UNIX Operating System Regular Expression Special Characters Open square bracket [ Backslash \ Caret ^ Dollar sign $ Dot . Pipe symbol | Question mark ? Asterisk * Plus sign + Opening and closing round brackets ( )

UNIX Operating System Regular Expression Special Characters ∅ denotes the empty set ∅ ε denotes the set containing only the "empty" string, which has no characters at all.

UNIX Operating System Regular Expression Examples: a | b * denotes {ε, "a", "b", "bb", "bbb", ...} (a | b) * denotes the set of all strings with no symbols other than "a" and "b", including the empty string: {ε, "a", "b", "aa", "ab", "ba", "bb", "aaa", ...}

UNIX Operating System Regular Expression Examples: .at matches any three character string ending with “at”, including “hat”, “cat”, and “bat”. [hc]at matches “hat” and “cat” [^b]at matches all strings matched by .at except “bat’. ^[hc]at matches “hat” and “cat”, but only at the beginning of the string or line.

Review week six lab assignment Regular expression Sed is a stream editor used for filtering and transforming text. The sed command modifies the contents of files, sending the changed file to standard output.

UNIX Operating System Sed commands = Display current line number a\text Append text to the file i\text Insert text into the file r\text Append text from filename into the file c\text Replace the selected range of lines with the provided text. s/original info/replacement info/ Replace text that matches the regular expression (original info) with replace (replacement info).

Review week six lab assignment Sed commands w file name Write the current pattern space to the specified file q Immediately quit the script, but print the current pattern space Q Immediately quit the script.

Review week six lab assignment Sed commands Examples: sed s/day/night/ <original_file >new_file echo Sunday | sed ‘s/day/night/’

UNIX Operating Systems Linux package management Envision any computer’s software as a house of cards. One program may rely on other programs or libraries, each of which relies on more and more software. The basic foundation for which house of cards relies on the Linux kernel. Anyone one of these packages can be replaced with an equivalent program. In turn, replacing programs can lead to much needed performance improvement or cause problems. Extracting cards from the stack can cause the house to tumble.

UNIX Operating Systems Linux package management Packages are basic information that package systems maintain about the software package, such as, the collection of files that are installed on the computer. Packages are distributed as one file, similar to a tarball. Most packages consist of many files, and the package system keeps track of them all.

UNIX Operating Systems Linux package components The database package systems maintain a database of installed files. The database contains a list of all files installed, provided by the package system, and other information. Dependencies is the package system that maintains dependency information. This is the requirements information of packages for one another.

UNIX Operating Systems Linux package components Checksum – The package system maintains checksums and other ancillary information on all files. This information can be used to verify the validity of the installed software. This feature is intended to help you to be aware of disk errors. It does have a limited use for detecting intrusion, because an intruder could use the package system to install altered system software.

UNIX Operating Systems Linux package management Upgrades and uninstallation – By tracking files and dependencies, a package system permits upgrades and uninstallations. Telling the package system to upgrade or remove a package, and it will do just that to files in the package.

UNIX Operating Systems Linux package management Binary package creation – RPM and Debian package systems provide tools to help create binary packages from source code. This feature is useful when executing Linux on a particular CPU.

UNIX Operating Systems Linux package management You can download source code and create a binary package that has advantages over compiling software from source in more conventional ways, because you can use the package management system to track dependencies, and attend to individual files.

Review week six lab assignment Software Installation Installing through a software package Package manager The simple definition for Ant might state that Ant is a Java-based build tool. There are a number of steps required to transform the source into a employable and useable software solution.

Review week six lab assignment Software Installation Installing through a software package Package manager RPM is a powerful and mature command- line driven package management system capable of installing, uninstalling, verifying, querying, and updating UNIX software packages.

Review week six lab assignment Software Installation Installing through a software package Package manager Each software package consists of an archive of files along with information about the package like its version, a description, and the like. There is also an API library, permitting advanced developers to manage such transactions from programming languages such as C, Perl or Python.

Review week six lab assignment Software Installation Installing through system commands and files Manual make command makefile file configure file

Review week six lab assignment Software Installation Installing software under UNIX is not always straightforward and easy. System administrators must be familiar with the layout of the file system. During a software installation, the possibility exists that the system could panic and/or have a hard disk failure.

Review week six lab assignment Package Install Instructions   A package is used by the installer to correctly place all the binary and data files on a system.  The installer records version numbers, file locations and check sums for verification, uninstalls and does version updates.  By using the installer all other packages that a particular application depend on will be installed at the same time. 

Unix Operating System Package Manager Commands - Doing an Installation To install a package use the pkg_update utility.  For example, if you wish to install bash you will enter the following command: pkg_update -L bash This command will automatically download and install the most current bash package.  If the package needs to have another package installed with it, that package will be downloaded and installed too.  Pkg_update also takes care of which machine architecture you are using and what is the most current version of the software.

Package Manager Commands What's Installed, Removing an Installation To display information about the current software packages installed on a system. Synopsis: pkg_info [options] [pkg-name] To remove a software package from a system . You do not need to know the version to delete a package. Synopsis: [pkg_delete [options] [pkg- name] To install a software package on a system. Synopsis: pkg_add [options] [pkg- name]

Package Manager Commands Package Licensing The licensing terms for each package can be easily viewed from the Warehouse web page before you decide to download by selecting the src directory.  It is important for you to understand that there are many different licenses used. Our preference is to use Full Open License software such as BSD and MIT.  There are many packages that are GPL or LGPL that may place restrictions on what you may do with a package's utilities and/or libraries. 

Package Manager Commands Installation Recommendations It is recommended that software packages be installed using the following command: pkg_update -L {pkg-name} This command provides to you the most trouble- free installation since so many packages are dependent on other packages. This command ensures that you get the most current release of a package and avoiding any unnecessary downloads. Regularly checking for updates is very important for bug and security fixes. Several times people have reported that they are having a problem with a package that they downloaded 30 to 40 days ago. During that time the problem has been fixed.

Package Manager Commands Checking for updates for all installed packages is easily done using the command: pkg_update -La These installations and updates are done over the Internet and packages are not stored in directory locations: /var/db/pkg/<pkg_name>

Package Manager Commands There are two ways to do an installation. You can choose the type of installation you want based on the type of installation you are going to perform.  For the multiple versions of software and multiple hardware platforms. It is strongly recommend to use the pkg_update for single or multiple packages.  Administrators will want to keep users using pkg_update. It is always recommended to install a package as: pkg_update -L {name} The software version number isn’t needed for this type of installation.  The current version number will be discovered. The most recent/current version number will be used.

Package Manager Commands Multiple Software Versions When a group of packages requires installation, you want to verify that all currently installed packages are up-to- date.  To do an automated check for new packages then pkg_update is the tool to use.  You may always refer to the pkg_update manual page for more information.

Package Manager Commands Multiple Software Versions New users usually want to install a group of packages to match their profile.  Currently three profiles exist: user, developer and administrator.  their profiles may be added in the future based on /Tools member feedback . By using a profile, a large number of packages can be selected and installed at one time. You can do the installations easily as: user: pkg_update -A developer: pkg_update -D administrator: pkg_update -G

Package Manager Commands pkg_add Install Caveat: The pkg_add utility is the "older" tool that should be used with care.  The pkg_add command should be used when a specific package is to be installed (such as an older archived version for comparison).  You may always refer to the pkg_add manual page for more information.

Package Manager Commands Multiple Hardware Platforms Administrators of several machines want to control what software can be installed or updated. Administrators usually like to test and validate software (for a multitude of reasons that we won't cover here) before making this software available to the user community.  

Package Manager Commands Multiple Hardware Platforms This control can be done setting these conditions: 1.  Cache packages have to be validated on a local network drive. 2.  Create your own "list" file in the cache directory with 'tags'. 3.  Set the PKG_LOCATION and PKG_PATH environment variables to this network drive on each system for members of the administrators group. 4.  Have members of the administrators group only use pkg_update for installations and updates.

Package Manager Commands Problems, Bugs, Your Suggestions If you encounter a problem installing a package please check that you are installing the most recent version of the package.  The installer itself receives regular updates too.  So keeping the installer up-to- date will help too.  The installer can update itself. It is often recommended that when a new version is announced you update to that version.  You can safely update the installer by doing: pkg_update -L pkg If you already have the current version the installer will notice and leave things as they are.

UNIX Operating System Tar was originally designed for tape backups. Tar creates one large file called a heap that contains a collection of many files and directories. A tar file occupies a large amount of space for the files and directories, and a bit more. A tar file is not compressed. The extension for a tar file is .tar. Create a tar file tar –cf newpack.tar /export/home/dandrear Extract the contents from a tar file tar –xvf origpack.tar List the table of contents tar –tvf origpack.tar

UNIX Operating System tar -xvf foo.tar verbosely extract foo.tar tar -xzf foo.tar.gz extract gzipped foo.tar.gz

UNIX Operating System -c, --create create a new archive -t, --list list the contents of an archive -u, --update only append files that are newer than the existing in archive -x, --extract, --get extract files from an archive --delete delete from the archive (not for use on mag tapes!)

UNIX Operating System Define: gzip is a utility that compresses a tar file. The compressed file extension becomes tar.gz. Compress a tar file gzip filename.tar Decompress a compressed tar file gzip –d filename.tar.gz gunzip filename.tar.gz Define: bzip2 / bunzip2 Used to compress and decompress tar files. File extension .bz2. bzip2 filename.tar Decompress a compressed bzip2 file bunzip2 –d filename.tar.bz2

UNIX Operating System Demonstrate the execution of listdir.pl script # Case #1: ./listdir.pl # Explanation: List current directory files in ascending order. # Case #2: ./listdir.pl –l # Explanation: List current directory files with format File name, Size, User, and Group column headings. # Case #3: ./listdir.pl -l -d /bin # Explanation: List /bin directory files with formatted column headings (File name, Size, User, and Group). # Case #4: ./listdir.pl -d /bin # Explanation: List /bin directory files in ascending order.

UNIX Operating System Points of interest gcc Compile a program written in C language g++ Compile a program written in C++ language ssh Open a secure connection to a remote host sftp Upload or download files with a secure connection.

Moving Around in UNIX ps –o pid,ppid,comm mpstat more password gzip ping du df

Break-out problems inode System panic Semaphore Tunable UNIX Parameters Kernel space Call by value Call by reference Data structure Definition of a kernel User space

After class assistance Questions Comments Concerns I am available after this Franklin Live session to discuss any problems and/or concerns regarding the lab assignments

Lab Assistance available by phone and/or email

Lab Assistance available by phone and/or email