1. Chapter 7 UNIX and LINUX

2. 2 Outline Overview Processes in UNIX Memory management in UNIX I/O in UNIX UNIX file system

3. 3 Overview Why some programmers like UNIX better than Windows? –Although GUI may be easy for beginners, they provide little flexibility and no insight into how the system works! –UNIX has GUI, too! History of UNIX –UNICS in Bell Labs –PDP-11 UNIX –Portable UNIX, portable C compiler –Berkeley UNIX (BSD) –UNIX standards: POSIX from IEEE –MINIX and Linux

4. 4 UNIX Goals (Philosophy) Designed by programmers, for programmers Simple, elegant and consistent Power and flexibility –A small number of basic elements –One program should do just one thing –An infinite combination to suit the applications No useless redundancy –Simple interface

5. 5 Interfaces to UNIX Hardware (CPU, memory, disks, terminals, etc) UNIX operating system (process management, memory management, the file system, I/O, etc) Standard library (e.g. open, close, read, write, fork, etc) Standards utility programs (shell, editors, compilers, etc) Users User interface Library interface System call interface User mode Kernel mode

6. 6 The UNIX Shell: A Command Line Interface Wait for a command line Extract the first word from the command line, use it as program name, run the program Pass arguments Flags: argument controlling the operation or specify an optional value Wild cards and magic characters Standard input/output Filter and pipe symbol –grep ter *.t | sort | head –20 | tail –5 > foo Background execution and shell scripts

7. 7 UNIX Utility Programs File and directory manipulation commands: cp, mv Filters: grep, head, sort, tail Program development tools: cc Text processing: vi, edit System administration: ps Miscellaneous: time

8. 8 Structure of UNIX Kernel System callsInterrupts and traps Terminal handlingSockets File naming Mapping Page faults Signal handling Process creation and termination Raw tty Cooked tyy Network protocols File systems Virtual memory Line disciplines Routing Buffer cache Page cacheProcess scheduling Character devices Network device drivers Disk device driversProcess dispatching Hardware

9. 9 Outline Overview Processes in UNIX Memory management in UNIX I/O in UNIX UNIX file system

10. 10 Basic Concepts UNIX is a multiprogramming system –User processes –Daemons Create new processes by system call fork() –Parent process and child process –PID, parent process gets child PID from fork() Pipes between two processes –Shell pipelines are implemented using pipes Signals: soft interrupt

11. 11 Process Management System Calls in UNIX pid=fork()Create a child process identical to the parent pid=waitpid(pid, &statloc, opts)Wait for a child to terminate s=execve(name, argv, envp)Replace a process’ core image exit(status)Terminate process execution and return status s=sigaction(sig, &act, &oldact)Define action to take on signals s=sigreturn(&context)Return from a signal s=sigprocmask(how, &set, &old)Examine or change the signal mask s=sigpending(set)Get the set of blocked signals s=sigsuspend(sigmask)Replace the signal mask and suspend the process s=kil(pid, sig)Send a signal to a process residual=alarm(seconds)Sent the alarm clock s=pause()Suspend the caller until the next signal

12. 12 Thread Management Not every UNIX has threads package –Threads package becomes popular Threads can be implemented in either user space or kernel –POSIX does not specify

13. 13 Thread Management System Calls Thread CallDescription pthread_createCreate a new thread in the caller’s address space pthread_exitTerminate the calling thread pthread_joinWait for a thread to terminate pthread_mutex_initCreate a new mutex pthread_mutex_destroyDestroy a mutex pthread_mutex_lockLock a mutex pthread_mutex_unlockUnlock a mutex pthread_cond_initCreate a condition variable pthread_cond_destroyDestroy a condition variable pthread_cond_waitWait on a condition variable pthread_cond_signalRelease one thread waiting on a condition variable

14. 14 Implementation of Processes Two key data structures for processes Process table: always in main memory –Scheduling parameters, memory image, signals, miscellaneous –A process has one entry in the table User structure: in memory only when the process is in memory –Machine registers, system call state, file descriptor table, accounting info, kernel stack –A block per process, adjacent to the stack segment

15. 15 Executing “ls” Typed to the Shell sh PID=501PID=748 Fork codeExec code 1. Fork call 2. New sh created 3. Exec call 4. Sh overlaid with ls New processSame process 1.Allocate child’s process table entry 2.Fill child’s entry from parent 3.Allocate child’s stack and user area 4.Fill child’s user area from parent 5.Allocate PID for child 6.Set up child to share parent’s text 7.Copy page tables for data and stack 8.Set up sharing of open files 9.Copy parent’s registers to child 1.Find the executable program 2.Verify the execute permission 3.Read and verify the header 4.Copy arguments, environ to kernel 5.Free the old address space 6.Allocate new address space 7.Copy arguments, environ to stack 8.Reset signals 9.Initialize registers

16. 16 Threads in UNIX Threads in user space: a user space library Threads in kernel: may lead to many problems –How to maintain the correct traditional UNIX semantics? –File I/O –Signal handling –All solutions to these problems cause something to break somewhere

17. 17 Threads in Linux System call clone – pid=clone(func, stack_ptr, sharing_flgs, arg) Sharing_flgs determine whether the thread is in current process or in a new process –In the same process: changes visible to other threads The new thread executes func The new thread has its own stack

18. 18 Scheduling in UNIX A two level algorithm –Low-level algorithm: pick the process to run –High-level algorithm: move processes between memory and disk Low-level algorithm uses multiple queues – Priority=CPU_usage+nice+base

19. 19 Multilevel Queue Structure Highest… Process waiting in kernel mode -4Waiting for disk I/O -3Waiting for disk buffer -2Waiting for terminal input Waiting for terminal output 0Waiting for child to exist 0User priority 0Process waiting in user mode 1User priority 1 2User priority 2 3User priority 3 Lowest…

20. 20 Scheduling in Linux Scheduling is based on threads Three classes of threads –Real-item FIFO, non-preemptive, highest priority –Real-time round robin, preemptive, high priority –Timesharing Scheduling based on priority and quantum

21. 21 Booting UNIX Read & run first sector of the boot disk –MBR has a small program (up to 512 bytes) –Load program boot Boot reads the root directory of boot device Boot reads OS kernel, boot ends Kernel starts, set parameters & environment Initialization, build kernel data structure System auto-configuration, detect devices Load device drivers Start process 0

22. 22 Process 0 Continue initialization Program the real-time clock Mount the root file system Create init (process 1) and page daemon (process 2)

23. 23 Process 1 Check flags of single-user/multi-user mode Single user mode –Fork off a process running shell –Wait for that process to exit Multi-user mode –Fork off a process Run system initialization shell script /etc/rc –Read /etc/ttys, fork off processes for terminals Run gtty

24. 24 Sequence of Processes in Booting Process 0 Page daemo n Process 2 ini t Process 1 gett y login sh cp Login: Terminal 0 Password: Terminal 1 Terminal 2 % cp f1 f2

25. 25 Loading Drivers Dynamically loading –One binary code can be used everywhere –Drivers are loaded dynamically, even through a network Non-dynamically loading –Only the system administrator can make kernel binary code –No one can insert any component into the kernel

26. 26 Outline Overview Processes in UNIX Memory management in UNIX I/O in UNIX UNIX file system

27. 27 Three Segments in UNIX Processes Text segment: program’s executable code –Read-only, size fixed after process is created Data segment: program’s variables, strings, and other data –Initialized data and un-initialized data (BSS) –Un-initialized global variables are in BSS, save space –Size can change Stack segment

28. 28 Sharing Between Processes Text segment & mapped file can be shared

29. 29 Memory Management System Calls POSIX does not specify Common system calls System calldescription s=brk(addr)Change data segment size a=mmap(addr, len, prot, flags, fd, offset)Map a file in s=unmap(addr, len)Unmap a file

30. 30 Swapping High level scheduling: swapper What cause swapping? –Fork() needs memory for a child process –Brk() needs to expand a data segment –Stack needs to be expanded Which process will be swapped out? –The ones blocked/waiting for I/O –The ones with high (priority + residence time) The ones consuming much CPU/staying long in mem

31. 31 Swapping Back Processes Check processes on disk every few seconds –Find processes are ready –Select the one staying on disk longest If have enough free memory (easy swap), bring that process in Otherwise, swap out some processes in main memory No process is swapped out until it stays in memory for 2 seconds

32. 32 Paging in UNIX Do not use the working set model Done by kernel & page daemon (process 2) Never page out kernel and core map –Core map records info about contents of the page frames

33. 33 Page Replacement Algorithm Run every 250 msec by page daemon If insufficient free page frames, page daemon transfers pages to disk A modified version of the clock algorithm

34. 34 Memory Management in Linux A three-level paging scheme –Directory, middle, page + offset Buddy algorithm 64 32 16 32 16 8 8 32 16 8 8 32 8 8 8 8 8 4 4 8 8 8 4 4 8 8 8 4 4 16

35. 35 Outline Overview Processes in UNIX Memory management in UNIX I/O in UNIX UNIX file system

36. 36 Input/Output in UNIX Integrate devices into file system as special files –Each I/O device is assigned a path name Block special files and character special files

37. 37 Networking and Sockets Socket: interface to network Types of networking supported by sockets –Reliable connection-oriented byte stream –Reliable connection-oriented packet stream –Unreliable packet transmission Sending process Socket Receiving process User space kernel space

38. 38 I/O System Calls in UNIX POSIX specifies function calls for terminal Function call ioctl is used in many UNIX Function callDescription s=cfsetospeed(&termios, speed)Set the output speed s=cfsetispeed(&termios, speed)Set the input speed s=cfgetospeed(&termios, speed)Get the output speed s=cfgetispeed(&termios, speed)Get the input speed s=tcsetattr(fd, opt, &termios)Set the attributes s=tcgetattr(fd, &termios)Get the attributes

39. 39 Outline Overview Processes in UNIX Memory management in UNIX I/O in UNIX UNIX file system

40. 40 UNIX File A sequence of bytes –No distinction between ASCII, binary, or any other kinds of files Directories are stored as files Important directories in UNIX systems DirectoryContents binBinary (executable) programs devSpecial files for I/O devices etcMiscellaneous system files libLibraries usrUser directories

41. 41 Mounting and Locking Only one file system (tree) –One disk is mounted in another disk’s file tree UNIX allows user to lock files –From single byte to an entire file –Shared locks and exclusive locks –A process specifies whether it wants to block if the lock cannot be placed –Shared locked regions may overlap

42. 42 System Calls About Files System callDescription fd=create(name, mode)One way to create a new file fd=open(file, how, …) Open a file for reading, writing or both s=close(fd)Close an open file n=read(fd, buffer, nbytes)Read data from a file into a buffer n=write(fd, buffer, nbytes)Write data from a buffer into a file position=lseek(fd, offset, whence)Move the file pointer s=stat(name, &buf)Get a file’s status information s=fstat(fd, &buf)Get a file’s status information s=pipe(&fd[0])Create a pipe s=fcntl(fd, cmd, …)File locking and other operations

43. 43 System Calls About Directories System callDescription s=mkdir(path, mode)Create a new directory s=rmdir(path)Remove a directory s=link(oldpath, newpath)Create a link to an existing file s=unlink(path)Unlink a file s=chdir(path)Change the working directory dir=opendir(path)Open a directory for reading s=closedir(dir)Close a directory dirent=readdir(dir)Read one directory entry Rewinddir(dir) Rewind a directory so it can be reread

44. 44 Disk Layout of Classical UNIX Boot block: contain code to boot computer Superblock: contain critical info about the layout of the file system –# of i-nodes, # of disk blocks, the start of the list of free disk blocks, … I-nodes Data blocks Boot blockSuper blkI-nodesData blocks

45. 45 Reading A File System call n=read(fd, buffer, nbytes) Start with file descriptor fd, find i-node –Put pointer to the i-node in file descriptor? Only one current position for one file, recorded in i- node Two processes open one file with different current position –Put current position info in file descriptor? One file can have multiple current positions P1 and P2 open f at the same time, P2 want to write after P1 finishes P2 cannot get the correct current position

46. 46 Open File description Table

47. 47 Berkeley Fast File System Allow file names up to 255 characters Divide the disk up into cylinder groups –Each with its own superblock, i-nodes and data blocks –Whenever possible, blocks are allocated in the cylinder group containing the i-node Two block sizes –Large files use large block size

48. 48 Summary Three interfaces to UNIX: –Shell, C library, and system calls Key concepts in UNIX –Process, memory model, I/O, and file system Process management in UNIX –Process table and user structure Memory model: text, data and stack I/O and file system