1. General overview of the Unix System Kernel Sub In-charge: Mr. V. V. Bhujade Dept. of CSE DMIETR, Wardha

2. Syllabus Architecture of Unix OS, Kernel data structures, system administration. The buffer cache; advantages & disadvantages.

3. Learning Objective Learning the basics of Unix operating system. Different types of Unix operating system. What are the features of UNIX operating system. The architecture of Unix operating system and the system kernel. The data structures used in the system. Use of buffer and its functions.

4. Brief UNIX History In 1969, Ken Thompson at AT&T Bell Labs began developing Unix – First done in assembly language – Ran on DEC PDP-7 with 8K words of memory In 1974, Unix was rewritten in C – By writing C compilers, Unix can be ported to other computers

5. BSD UNIX – BSD: Berkeley Software Distribution – UCB put the Internet Protocols (IP) into UNIX – 4.4BSD (Berkeley): 1994 System V – AT&T System V (1984) – SVR4: System V Release 4 (current version)

6. Overview of UNIX Supports various architectures Structure varies Supports preemptive multitasking Multiuser environment - generally secure Supports multithreaded applications Protection/Security is high on modern versions Supports symmetric multiprocessing Highly scalable/portable to various systems Many types/flavours of UNIX exist

7. Features of UNIX OS Multitasking Multiuser Portable Security Communication

8. Unix programs and files File: collection of data – UNIX treats everything as a file including peripherals Program : collection of bytes representing code and data stored in file Process : a program in execution(currently running, loaded from disk into RAM) – Services provided for process.

9. UNIX OS UNIX is very popular. It is a unique, simple and easy to understand structure. Unix has a number of features. – Unix is a multi-user system.(More than one user) – It is a Multi-process or Multi-tasking.(More than one job at a time) – Time Sharing.(More than one user can use the system at anytime.) – Portability – Unix is highly portable.

10. Layers of UNIX OS User interface

11. Structure of Unix system Fig :Structure of UNIX operating system

12. Fig : Architecture of Unix operating system

13. Structure of Unix System The Unix system is supported by two agencies, 1. Kernel 2. Shell The Kernel interact with the H/W. The shell interact with user.(Relation between Kernel & shell).

14. Kernel The Kernel is the heart of the system. It is a collection of program. Written in C. It communicates with the H/W directly. There is only one Kernel for any system. The kernel is loaded into the memory when the system is booted. It manages the system resources. It allocates time between users and processes. It decides process priorities and perform all other tasks which we need not take care off. Kernel program are independent of the H/W. The kernel is often called the OS.

15. Shell The shell is the Sleeping Beauty. This is an Unix Command. It is the interpreter of user. It takes a command from the user, translate it, exchanges information and communicates with the kernel and executes the command. It is an I/F between the user and the kernel. The Shell insulates the user from the knowledge of kernel functions. The shell takes care of redirection features (> and |).

16. Types of shells Bourne shell Korn shell($) C shell(%) Bourne again shell(BASH)($) Each shell has its own programming language Functionality is same with specialized properties

17. User Perspective The UNIX system such as – The File system – The Processing Environment – Building Block Primitives ( Ex. Pipes)

18. File system A hierarchical structure Consistent treatment of file data. The ability to create and delete files. Dynamic growth of files. Protection of file data. Treatment of Peripheral devices as files.

19. Fig : Sample file system tree

20. Different types of Files Ordinary files Directories Special files Pipes

21. Ordinary Files or Regular Files A Steam of data which is resident on some permanent magnetic media. This includes all data, source program object & executable code, all Unix program as well as any file created by the user. The ordinary files are also referred to as a regular file. All the text files also belong to this type.

22. Directories Directory is file containing list of files and subdirectories It has fixed size records of 16 bytes each which contains – - a filename(14 bytes) – - an inode number (2 bytes) which acts as a pointer to where the system can find info about the file.

23. Special Files Special files are contained in the directory /dev. They are used to represent a real physical device such as a printer, tape device etc. Ex: Special device - /dev/null (unwanted output can be redirected).

24. Pipes UNIX allows us to link commands together using a pipe. The pipe acts as a temporary file which only exists to hold data from one command until its read by another. Ex: command1 | command2 | command3....

25. Building Block Primitives: The Unix system is to provide OS primitives. – redirect I/O. – Pipe. redirect I/O : Processes conventionally have access to three files. They read from their standard input file. Write to their standard output files. And write error messages to their standard error file. Example : ls : Lists all the files in the current directory on the standard output. ls > output – where > redirects the standard output to the file called:output “ in the current directory, using the create system call.

26. Building Block Primitives(contd.) Pipes: – A mechanism that allows a stream of data to be passed between reader and writer processes. – Processes can redirect their standard output to a pipe to be read by other processes that have redirect their standard input to come from the pipe. – The programmer uses the pipe primitive and redirection of I/O to integrate the piece parts.

27. Assumptions about H/W The execution of user processes on UNIX systems is divided into two levels: – User Mode – Processes in user mode can access their instructions and data but not kernel instructions and data. Some machine instructions are privileged and result in an error when executed in user mode. – Kernel Mode.- Can access kernel and user addresses. When a process executes s system call, the execution of the process changes from user mode to kernel mode.

28. Interrupts and exceptions Interrupts to happen between the execution of two instructions. The hardware usually prioritizes devices according to the order that interrupts should be handled. When the kernel services an interrupt, it blocks out lower priority interrupts but services higher priority interrupts. Exceptions : Exceptions happen “ in the middle “ of the execution of an instruction.

29. Architecture of system kernel

30. Architecture of system kernel(contd.) Block diagram of the system kernel. Two major components of the kernel – File subsystem and – process control subsystem. Three levels – User – Kernel – hardware

31. File subsystem File subsystem manages files allocates file space. – 1. Administers free space. – 2. Controls access to files and retrieves data to users. – 3. File subsystem uses buffering mechanism for block I/O – 4.Device drivers and interacts directly with raw I/O device drivers.

32. Process Control Subsystem The file subsystem and the process control subsystem interact when loading a file into memory for execution The process subsystem reads executable files into memory before executing them. Some of the system calls: – fork – Create a new process. – exec- Overly the image of a program onto the running process. – exit – Finish executing a process. – wait – synchronize process execution with the exit of a previously forked process. – brk – control the size of memory allocated to a process. – signal – control process response to extraordinary events.

33. Kernel data structures What is data structure? Are of fixed size Code is simple (advantage). No dynamic allocation of DS. Algorithms use simple loops to free table entry.

35. System Administration What is administrative process?? What is user process?? No conceptual difference between administrative process and user process. Same set of system calls used Administrative process have more rights and privileges. Super user.

36. The Buffer Cache Buffer Cache Buffer Pool Structure Retrieval of Buffer Release Buffer Reading and Writing Disk Blocks

37. 37 Buffer Cache When a process wants to access data from a file, the kernel brings the data into main memory, alters it and then request to save in the file system To increase the response time and throughput, the kernel minimizes the frequency of disk access by keeping a pool of internal data buffer called buffer cache.

38. 38 Buffer Cache Buffer cache contains the data in recently used disk blocks When reading data from disk, the kernel attempts to read from buffer cache. If data is already in the buffer cache, the kernel does not need to read from disk If data is not in the buffer cache, the kernel reads the data from disk and cache it

39. 39 Buffer Headers A buffer consists of two parts – a memory array – buffer header Buffer Header device num block num status ptr to next buf on hash queue ptr to previous buf on hash queue ptr to next buf on free list ptr to previous buf on free list ptr to data area

40. 40 Buffer Headers device num – logical file system number block num – block number of the data on disk status – The buffer is currently locked. – The buffer contains valid data. – delayed-write – The kernel is currently reading or writing the contents of the disk. – A process is currently waiting for the buffer to become free. kernel identifies the buffer content by examing device num and block num.

41. 41 Structures of the buffer pool Buffer pool according to LRU The kernel maintains a free list of buffer – doubly linked list – take a buffer from the head of the free list. – When returning a buffer, attaches the buffer to the tail. free list head buf 1 buf 2 buf n Forward ptrs Back ptrs Free list of Buffers

42. 42 Structures of the buffer pool When the kernel accesses a disk block – separate queue (doubly linked circular list) – hashed as a function of the device and block num – Every disk block exists on one and only on hash queue and only once on the queue 4 517 105098 9935 3 28 64 97 blkno0 mod 4 blkno1 mod 4 blkno2 mod 4 blkno3 mod 4 Hash queue headers Buffers on the Hash Queues

43. 43 Scenarios for retrieval of a buffer Determine the logical device num and block num The algorithms for reading and writing disk blocks use the algorithm getblk – The kernel finds the block on its hash queue The buffer is free. The buffer is currently busy. – The kernel cannot find the block on the hash queue The kernel allocates a buffer from the free list. In attempting to allocate a buffer from the free list, finds a buffer on the free list that has been marked “delayed write”. The free list of buffers is empty.

44. 44 Retrieval of a Buffer:1 st Scenario (a) The kernel finds the block on the hash queue and its buffer is free 4 517 105098 9935 3 28 64 97 blkno0 mod 4 blkno1 mod 4 blkno2 mod 4 blkno3 mod 4 Hash queue headers freelist header Search for block 4

45. 45 Retrieval of a Buffer:1 st Scenario (b) 4 517 105098 9935 3 28 64 97 blkno0 mod 4 blkno1 mod 4 blkno2 mod 4 blkno3 mod 4 freelist header Remove block 4 from free list

46. 46 Retrieval of a Buffer: 2 nd Scenario (a) The kernel cannot find the block on the hash queue, so it allocates a buffer from free list 4 517 105098 9935 3 28 64 97 blkno0 mod 4 blkno1 mod 4 blkno2 mod 4 blkno3 mod 4 Hash queue headers freelist header Search for block 18: Not in cache

47. 47 Retrieval of a Buffer: 2 nd Scenario (b) Hash queue headers 18 4 517 105098 9935 28 64 97 blkno0 mod 4 blkno1 mod 4 blkno2 mod 4 blkno3 mod 4 freelist header Remove 1 st block from free list: Assign to 18

48. 48 Retrieval of a Buffer: 3 rd Scenario (a) The kernel cannot find the block on the hash queue, and finds delayed write buffers on hash queue 4 517 105098 9935 3 28 64 97 blkno0 mod 4 blkno1 mod 4 blkno2 mod 4 blkno3 mod 4 Hash queue headers freelist header Search for block 18, Delayed write blocks on free list delay

49. 49 Retrieval of a Buffer: 3 rd Scenario (b) 517 105098 9935 3 28 64 97 blkno0 mod 4 blkno1 mod 4 blkno2 mod 4 blkno3 mod 4 Hash queue headers freelist header 18 writing (b) Writing Blocks 3, 5, Reassign 4 to 18

50. 50 Retrieval of a Buffer: 4th Scenario The kernel cannot find the buffer on the hash queue, and the free list is empty 28 blkno0 mod 4 blkno1 mod 4 blkno2 mod 4 blkno3 mod 4 Hash queue headers freelist header 4 64 5 97 17 10 50 98 99 35 3 Search for block 18, free list empty

51. 51 Retrieval of a Buffer: 5th Scenario Kernel finds the buffer on hash queue, but it is currently busy 4 517 105098 9935 3 28 64 97 blkno0 mod 4 blkno1 mod 4 blkno2 mod 4 blkno3 mod 4 Hash queue headers freelist header Search for block 99, block busy busy

52. 52 Algorithm: GetBlock GetBlock (file_system_no,block_no) – while (buffer not found) if (buffer in hash queue) – if (buffer busy) /* 5 */ » sleep (event buffer becomes free) » continue – mark buffer busy /* 1 */ – remove buffer from free list – return buffer else – if (there is no buffer on free list) /* 4 */ » sleep (event any buffer becomes free) » continue – remove buffer from free list – if (buffer marked as delayed write) /* 3 */ » asyschronous white buffer to disk » Continue – remove buffer from hash queue /* 2 */ – put buffer onto hash queue – return buffer

53. 53

54. 54 Algorithm: ReleaseBlock ReleaseBlock (locked buffer) – wakeup all process event, waiting for any buffer to become free – wakeup all process event, waiting for this buffer to become free – raise processor execution level to block interrupt – if (buffer content valid and buffer not old) enqueue buffer at the end of free list – else enqueue buffer at the beginning of free list – lower processor execution level to allow interrupt – unlock (buffer)

55. 55 Reading and Writing disk blocks To read block ahead – The kernel checks if the block is in the cache or not. – If the block in not in the cache, it invokes the disk driver to read the block. – The the process goes to sleep awaiting the event that the I/O is complete. – The disk controller interrupts the processor when the I/O is complete – The disk interrupt handler awakens the sleeping processes – The content of disk blocks are now in the buffer – When the process no longer need the buffer, it releases it so that other processes can access it

56. 56 Fig Algorithm for reading disk block

57. 57 Reading and Writing disk blocks To write a disk block – The kernel informs the disk driver that it has a buffer whose contents should be output. – The disk driver schedules the block for I/O. – If the write is synchronous, the calling process goes to sleep awaiting I/O completion and releases the buffer when awakens. – If the write is asynchronous, the kernel starts the disk write. The kernel release the buffer when the I/O completes.

58. 58 Fig. Algorithm for writing to disk block

59. Advantages of buffer The use of buffers allows uniform disk access. The system places no data alignment restrictions on user processes doing I/O, because the kernel aligns data internally. Use of the buffer cache can reduce the amount of disk traffic. The buffer algorithms help insure file system integrity.

60. Disadvantages of buffer The kernel does not immediately write data to the disk for a delayed write, the system is vulnerable to crashes that leave disk data in an incorrect state. Use of the buffer cache requires an extra data copy when reading and writing to and from user processes.

61. Summary The data structures and architecture of the Unix operating system is studied properly. The reading and allocation of disk blocks and buffer is seen in detail. The advantages and disadvantages are covered.

62. References “The Design Of Unix Operating System”- M. J. Bach “Unix Concepts and Applications”- Sumitabha Das.