1. File System Implementations CS-502 (EMC) Fall 20091 File System Implementations CS-502, Operating Systems Fall 2009 (EMC) (Slides include materials from Modern Operating Systems, 3 rd ed., by Andrew Tanenbaum and from Operating System Concepts, 7 th ed., by Silbershatz, Galvin, & Gagne)

2. File System Implementations CS-502 (EMC) Fall 20092 Review – the File abstraction A (potentially) large amount of information or data that lives a (potentially) very long time Often much larger than the memory of the computer Often much longer than any computation Sometimes longer than life of machine itself (Usually) organized as a linear array of bytes or blocks Internal structure is imposed by application (Occasionally) blocks may be variable length (Often) requiring concurrent access by multiple processes Even by processes on different machines!

3. File System Implementations CS-502 (EMC) Fall 20093 File Systems and Disks User view –File is a named, persistent collection of data OS & file system view –File is collection of disk blocks — i.e., a container –File System maps file names and offsets to disk blocks

4. File System Implementations CS-502 (EMC) Fall 20094 This topic – Implementation of Files Create file abstraction using physical disk devices and disk blocks –Efficient in time, space, use of disk resources –Fast enough for application requirements Must be scalable to a wide variety of file sizes –Many small files (< 1 page) –Huge files (100’s of gigabytes, terabytes, spanning disks) –Everything in between

5. File System Implementations CS-502 (EMC) Fall 20095 Reading Assignment Tanenbaum §4.3

6. File System Implementations CS-502 (EMC) Fall 20096 Overview File Allocation methods Sequential Linked and FAT Indexed Free blocks & bad blocks Scalability Mounting & Virtual File Systems Directory implementation notes

7. File System Implementations CS-502 (EMC) Fall 20097 Definition — Volume The fundamental unit of a file system Physical volume may be a physical disk storage device physical partition of a single disk (aka minidisk) Logical Volume is A physical volume A combination of other volumes –Usually similar in size and characteristics Volume is also used (loosely) to denote the part of the file system implemented on a physical or logical volume

8. File System Implementations CS-502 (EMC) Fall 20098 File Allocation Schemes Contiguous –Blocks of file stored in consecutive disk sectors –Directory points to first entry & specifies length Linked –Blocks of file scattered across disk, as linked list –Directory points to first entry Indexed –Blocks of file scattered across disk –Separate index block contains pointers to file blocks –Directory points to index block

9. File System Implementations CS-502 (EMC) Fall 20099 File Allocation Schemes (continued) The allocation scheme is an attribute of a file system, not of individual files within a system. All files within a file system follow same allocation model

10. File System Implementations CS-502 (EMC) Fall 200910 File Allocation Schemes Contiguous –Blocks of file stored in consecutive disk sectors –Directory points to first entry Linked –Blocks of file scattered across disk, as linked list –Directory points to first entry Indexed –Separate index block contains pointers to file blocks –Directory points to index block

11. File System Implementations CS-502 (EMC) Fall 200911 Contiguous Allocation Ideal for large, static files –Databases, fixed system structures, OS code –Multi-media video and audio –CD-ROM, DVD Simple address calculation –Directory entry points to first sector –File block i  disk sector address Fast multi-block reads and writes –Minimize seeks between blocks

12. File System Implementations CS-502 (EMC) Fall 200912 Contiguously Allocated Files

13. File System Implementations CS-502 (EMC) Fall 200913 Block-to-sector Calculation To find disk sector containing block i of file f –Starting_block(f) + i Starting block of each file is named in –Directory, or –File metadata

14. File System Implementations CS-502 (EMC) Fall 200914 File Creation (Contiguous File System) Search for an empty sequence of blocks –First-fit –Best-fit Prone to fragmentation when … Files come and go Files change size Similar to physical memory allocation in base-limit type of virtual memory

15. File System Implementations CS-502 (EMC) Fall 200915 Contiguous Allocation – Extents Extent: a contiguously allocated subset of a file Directory entry points to –(For file with one extent) the extent itself –(For file with multiple extents) pointer to an extent block describing multiple extents Advantages –Speed, ease of address calculation of contiguous file –Avoids (some of) the fragmentation issues –Can be adapted to support files across multiple disks …

16. File System Implementations CS-502 (EMC) Fall 200916 Contiguous Allocation – Extents … Disadvantages –Too many extents  degenerates to indexed allocation As in Unix-like systems, but not so well Popular in 1960s & 70s –OS/360, other systems for commercial data processing Currently used for large files in NTFS Rarely mentioned in textbooks Silbershatz, 7 th ed., §11.4.1 & 22.5.1

17. File System Implementations CS-502 (EMC) Fall 200917 Digression: Bad Block Management Bad blocks on disks are inevitable Part of manufacturing process (less than 1%) Most are detected during formatting Occasionally, blocks become bad during operation Manufacturers typically add extra tracks to disks Physical capacity = (1 + x) * rated_capacity Who handles bad blocks? Disk controller: Bad block list maintained internally –Automatically substitutes good blocks Formatter: Re-organize track to avoid bad blocks OS: Bad block list maintained by OS, bad blocks never used

18. File System Implementations CS-502 (EMC) Fall 200918 Bad Block Management in Contiguous Allocation File Systems Bad blocks must be concealed Foul up the block-to-sector calculation Methods Look-aside list of bad sectors –Check each sector request against hash table –If present, substitute a replacement sector behind the scenes Spare sectors in each track, remapped by formatting Handling Disk controller, invisible to OS Lower levels of OS; concealed from higher layers of file system and from application

19. File System Implementations CS-502 (EMC) Fall 200919 Questions?

20. File System Implementations CS-502 (EMC) Fall 200920 File Allocation Schemes Contiguous –Blocks of file stored in consecutive disk sectors –Directory points to first entry Linked –Blocks of file scattered across disk, as linked list –Directory points to first entry Indexed –Separate index block contains pointers to file blocks –Directory points to index block

21. File System Implementations CS-502 (EMC) Fall 200921 Linked Allocation Blocks scattered across disk Each block contains pointer to next block Directory points to first and last blocks Sector header: –Pointer to next block –ID and block number of file 10 16 01 25

22. File System Implementations CS-502 (EMC) Fall 200922 Linked Allocation Advantages –No external fragmentation of file space! –Easy to create, extend files –Ideal for lots of small files Disadvantages –Lots of disk arm movement –Space taken up by links –Sequential access only! Random access simulated by caching links Used in Xerox Alto file system

23. File System Implementations CS-502 (EMC) Fall 200923 Bad Block Management – Linked File Systems In OS:– format all sectors of disk Don’t reserve any spare sectors Allocate bad blocks to a hidden file for the purpose If a block becomes bad, “append” to the hidden file Advantages Very simple No look-aside or sector remapping needed Totally transparent without any hidden mechanism

24. File System Implementations CS-502 (EMC) Fall 200924 Linked File System – Limitation To access the i th block, it is necessary to physically read the first (i  1) blocks from disk! Serious problem for large files!

25. File System Implementations CS-502 (EMC) Fall 200925 Solution – File Allocation Table (FAT) Instead of link on each block, put all links in one table — the FAT Each entry corresponds to physical block in disk i th entry contains link for block i –Each entry points to next block (or EOF) Directory points to first & last blocks of file

26. File System Implementations CS-502 (EMC) Fall 200926 FAT File Systems Advantages –Advantages of Linked File System –FAT can be cached in memory –Searchable at CPU speeds  pseudo-random access Disadvantages –Limited size, not suitable for very large disks –FAT cache describes entire disk, not just open files! –Not fast enough for large databases Used in MS-DOS, early Windows systems –Also USB Flash drives, floppy disks, etc.

27. File System Implementations CS-502 (EMC) Fall 200927 Bad Block Management – FAT File Systems Same as Linked File Systems I.e., format all sectors of disk Don’t reserve any spare sectors Allocate bad blocks to a hidden file for the purpose If a block becomes bad, append to the hidden file Same advantages and disadvantages

28. File System Implementations CS-502 (EMC) Fall 200928 Disk Defragmentation Re-organize blocks in disk so that file is (mostly) contiguous Link or FAT organization preserved Purpose: –To reduce disk arm movement during sequential accesses –To permit contiguous reads or writes in one disk operation Does not change the linked structure of the file system!

29. File System Implementations CS-502 (EMC) Fall 200929 Exam Question (last spring) You have a humongous database stored in a file on a 4 GB flash drive with a FAT file system. What must the file system do to locate block n of the database? Assume that database has not been defragmented, so that its blocks are likely to be scattered randomly across the flash drive. Given that the file system has found the location of block n, what must it do to find the location of block n+1? block n-1?

30. File System Implementations CS-502 (EMC) Fall 200930 Questions? Linked and FAT File Systems

31. File System Implementations CS-502 (EMC) Fall 200931 File Allocation Schemes Contiguous –Blocks of file stored in consecutive disk sectors –Directory points to first entry Linked –Blocks of file scattered across disk, as linked list –Directory points to first entry Indexed –Separate index block contains pointers to file blocks –Directory points to index block

32. File System Implementations CS-502 (EMC) Fall 200932 Problems and Issues Contiguous file systems not suitable for files that come and go rapidly & frequently Linked file systems not suitable for very large disks or a lot of random access Can we do better?

33. File System Implementations CS-502 (EMC) Fall 200933 Indexed Allocation i-node: –Contains file metadata –Lists sector address of each block of file Advantages –True random access –Only i-nodes of open files need to be cached –Supports small and large files

34. File System Implementations CS-502 (EMC) Fall 200934 Unix/Linux i-nodes Direct blocks: –Pointers to first n sectors Single indirect table: –Extra block containing pointers to blocks n+1.. n+m Double indirect table: –Extra block containing single indirect blocks …

35. File System Implementations CS-502 (EMC) Fall 200935 Indexed Allocation Access to every block of file is via i-node Bad block management –Similar to Linked/FAT systems Disadvantage –Not as fast as contiguous allocation for large databases Requires reference to i-node for every access vs. Simple calculation of block to sector address

36. File System Implementations CS-502 (EMC) Fall 200936 Indexed Allocation (continued) Widely used in Unix, Linux, Windows NTFS Robust Has withstood the test of time Many variations

37. File System Implementations CS-502 (EMC) Fall 200937 Questions?

38. File System Implementations CS-502 (EMC) Fall 200938 Free Block Management in File Systems Bitmap –Very compact on disk –Expensive to search –Supports contiguous allocation Free list –Linked list of free blocks Each block contains pointer to next free block –Only head of list needs to be cached in memory –Very fast to search and allocate –Contiguous allocation very difficult

39. File System Implementations CS-502 (EMC) Fall 200939 Free Block Management Bit Vector … 012n-1 bit[i] =  0  block[i] free 1  block[i] occupied Free block number calculation (number of bits per word) * (number of 0-value words) + offset of first 1 bit

40. File System Implementations CS-502 (EMC) Fall 200940 Free Block Management Bit Vector (continued) Bit map –Must be kept both in memory and on disk –Copy in memory and disk may differ –Cannot allow for block[i] to have a situation where bit[i] = 1 in memory and bit[i] = 0 on disk

41. File System Implementations CS-502 (EMC) Fall 200941 Free Block Management Bit Vector (continued) Solution: –Set bit[i] = 1 in disk –Allocate block[i] –Set bit[i] = 1 in memory –Similarly for set of contiguous blocks Potential for lost blocks in event of crash! –Discussion:– How do we solve this problem?

42. File System Implementations CS-502 (EMC) Fall 200942 Free Block Management Linked List Linked list of free blocks –Not necessarily in order! Cache first few free blocks in memory Head of list must be stored both –On disk –In memory Each block must be written to disk when freed Potential for losing blocks?

43. File System Implementations CS-502 (EMC) Fall 200943 Free Block Management – Linked List (continued) Can also be implemented in FAT Can also be implemented in Indexed File system

44. File System Implementations CS-502 (EMC) Fall 200944 Reading Assignment Tanenbaum §4.3

45. File System Implementations CS-502 (EMC) Fall 200945 Scalability of File Systems Question: How large can a file be? Answer: limited by –Number of bits in length field in metadata –Size & number of block entries in FAT or i-node Question: How large can file system be? Answer: limited by –Size & number of block entries in FAT or i-node

46. File System Implementations CS-502 (EMC) Fall 200946 MS-DOS & Windows FAT-12 (primarily on floppy disks): 4096 512-byte blocks Only 4086 blocks usable! FAT-16 (early hard drives): 64 K blocks; block sizes up to 32 K bytes 2 GBytes max per partition, 4 partitions per disk FAT-32 (Windows 95) 2 28 blocks; up to 2 TBytes per disk Max size FAT requires 2 32 bytes in RAM!

47. File System Implementations CS-502 (EMC) Fall 200947 MS-DOS File System (continued) Maximum partition for different block sizes The empty boxes represent forbidden combinations

48. File System Implementations CS-502 (EMC) Fall 200948 Classical Unix Maximum number of i-nodes = 64K! How many files in a modern PC? I-node structure allows very large files, but … Limited by size of internal fields Limited by # of entries in triple-indirect block

49. File System Implementations CS-502 (EMC) Fall 200949 Modern Operating Systems Need much larger, more flexible file systems Many terabytes per system Multi-terabyte files Suitable for both large and small Cache only open files in RAM

50. File System Implementations CS-502 (EMC) Fall 200950 Examples of Modern File Systems Windows NTFS Tanenbaum §11.8 Linux ext2 and ext3 Tanenbaum §10.6.3 Other file systems … Consult your favorite Linux system documentation

51. File System Implementations CS-502 (EMC) Fall 200951 New Topic

52. File System Implementations CS-502 (EMC) Fall 200952 Mounting mount –t type device pathname Attach device (which contains a file system of type type) to the directory at pathname File system implementation for type gets loaded and connected to the device Anything previously below pathname becomes hidden until the device is un-mounted again The root of the file system on device is now accessed as pathname E.g., mount –t iso9660 /dev/cdrom /myCD

53. File System Implementations CS-502 (EMC) Fall 200953 Mounting (continued) OS automatically mounts devices in mount table at initialization time /etc/fstab in Linux Users or applications may mount devices at run time, explicitly or implicitly — e.g., Insert a floppy disk Plug in a USB flash drive Type may be implicit in device Windows equivalent Map drive

54. File System Implementations CS-502 (EMC) Fall 200954 Virtual File Systems Virtual File Systems (VFS) provide object- oriented way of implementing file systems. VFS allows same system call interface to be used for different types of file systems. The API is to the VFS interface, rather than any specific type of file system.

55. File System Implementations CS-502 (EMC) Fall 200955 Schematic View of Virtual File System

56. File System Implementations CS-502 (EMC) Fall 200956 Virtual File System (continued) Mounting: formal mechanism for attaching a file system to the Virtual File interface

57. File System Implementations CS-502 (EMC) Fall 200957 Linux Virtual File System (VFS) A generic file system interface provided by the kernel Common object framework –superblock: a specific, mounted file system –i-node object: a specific file in storage –d-entry object: a directory entry –file object: an open file associated with a process

58. File System Implementations CS-502 (EMC) Fall 200958 Linux Virtual File System (continued) VFS operations –super_operations: read_inode, sync_fs, etc. –inode_operations: create, link, etc. –d_entry_operations: d_compare, d_delete, etc. –file_operations: read, write, seek, etc.

59. File System Implementations CS-502 (EMC) Fall 200959 Linux Virtual File System (continued) Individual file system implementations conform to this architecture. May be linked to kernel or loaded as modules Linux kernel 2.6 supports over 50 file systems in official version E.g., minix, ext, ext2, ext3, iso9660, msdos, nfs, smb, …

60. File System Implementations CS-502 (EMC) Fall 200960 Reading references Tanenbaum §4.3, 10.6.3, 11.8 Also:– Robert Love, Linux Kernel Development, 2 nd edition, Chapter 12

61. File System Implementations CS-502 (EMC) Fall 200961 Questions?

62. File System Implementations CS-502 (EMC) Fall 200962 Implementation of Directories A list of [name, information] pairs Must be scalable from very few entries to very many Name: User-friendly, variable length Any language Fast access by name Information: File metadata (itself) Pointer to file metadata block (or i-node) on disk Pointer to first & last blocks of file Pointer to extent block(s) …

63. File System Implementations CS-502 (EMC) Fall 200963 Very Simple Directory Short, fixed length names Attribute & disk addresses contained in directory MS-DOS, etc. name1attributesname2attributesname3attributesname4attributes …

64. File System Implementations CS-502 (EMC) Fall 200964 Simple Directory Short, fixed length names Attributes in separate blocks (e.g., i-nodes) Attribute pointers are disk addresses (or i-node numbers) Older Unix versions, MS-DOS, etc. name1name2name3name4… i-node Data structures containing attributes

65. File System Implementations CS-502 (EMC) Fall 200965 More Interesting Directory Variable length file names –Stored in heap at end Modern Unix, Windows Linear or logarithmic search for name Compaction needed after –Deletion, Rename attributes … name1 longer_na me3 very_long_n ame4 name2 …

66. File System Implementations CS-502 (EMC) Fall 200966 Very Large Directories Hash-table implementation Each hash chain like a small directory with variable-length names Must be sorted for listing

67. File System Implementations CS-502 (EMC) Fall 200967 Questions?

68. File System Implementations CS-502 (EMC) Fall 200968 Raw Disk Layout Track format – n sectors –200 < n < 2000 in modern disks –Some disks have fewer sectors on inner tracks Inter-sector gap –Enables each sector to be read or written independently Sector format –Sector address: Cylinder, Track, Sector –Optional header –Data –Each field separated by small gap and with its own CRC Sector length –Almost all operating systems specify uniform sector length –512 – 4096 bytes

69. File System Implementations CS-502 (EMC) Fall 200969 Bad Sector Handling – within track a)A disk track with a bad sector b)Substituting a spare for the bad sector c)Shifting all the sectors to bypass the bad one