1. File System Implementations CS-502 Fall 20071 File System Implementations CS-502, Operating Systems Fall 2007 (Slides include materials from Operating System Concepts, 7 th ed., by Silbershatz, Galvin, & Gagne and from Modern Operating Systems, 2 nd ed., by Tanenbaum)

2. File System Implementations CS-502 Fall 20072 Implementation of Files Create file abstraction using physical disk devices and disk blocks –Efficient in time, space, use of disk resources –Fast enough for application requirements Must be scalable to a wide variety of file sizes –Many small files (< 1 page) –Huge files (100’s of gigabytes, terabytes, spanning disks) –Everything in between

3. File System Implementations CS-502 Fall 20073 File Allocation Schemes Contiguous –Blocks of file stored in consecutive disk sectors –Directory points to first entry Linked –Blocks of file scattered across disk, as linked list –Directory points to first entry Indexed –Separate index block contains pointers to file blocks –Directory points to index block

4. File System Implementations CS-502 Fall 20074 File Allocation Schemes (continued) The allocation scheme is an attribute of a file system, not of individual files within a system. All files within a file system follow same allocation model

5. File System Implementations CS-502 Fall 20075 Volume The fundamental unit of a file system Physical volume may be a physical disk storage device physical partition of a single disk (aka minidisk) Logical Volume A physical volume A combination of other volumes –Usually similar in size and characteristics

6. File System Implementations CS-502 Fall 20076 File Allocation Schemes Contiguous –Blocks of file stored in consecutive disk sectors –Directory points to first entry Linked –Blocks of file scattered across disk, as linked list –Directory points to first entry Indexed –Separate index block contains pointers to file blocks –Directory points to index block

7. File System Implementations CS-502 Fall 20077 Contiguous Allocation Ideal for large, static files –Databases, fixed system structures, OS code –Multi-media video and audio –CD-ROM, DVD Simple address calculation –Directory entry points to first sector –File block i  disk sector address Fast multi-block reads and writes –Minimize seeks between blocks

8. File System Implementations CS-502 Fall 20078 Contiguously Allocated Files

9. File System Implementations CS-502 Fall 20079 Block-to-sector Calculation To find disk sector containing block i of file f –Starting_block(f) + i Starting block of each file is named in –Directory, or –File metadata

10. File System Implementations CS-502 Fall 200710 File Creation (Contiguous File System) Search for an empty sequence of blocks –First-fit –Best-fit Prone to fragmentation when … Files come and go Files change size Similar to physical memory allocation in base-limit type of virtual memory

11. File System Implementations CS-502 Fall 200711 Digression: Bad Block Management Bad blocks on disks are inevitable Part of manufacturing process (less than 1%) Most are detected during formatting Occasionally, blocks become bad during operation Manufacturers typically add extra tracks to disks Physical capacity = (1 + x) * rated_capacity Who handles bad blocks? Disk controller: Bad block list maintained internally –Automatically substitutes good blocks Formatter: Re-organize track to avoid bad blocks OS: Bad block list maintained by OS, bad blocks never used

12. File System Implementations CS-502 Fall 200712 Bad Block Management in Contiguous Allocation File Systems Bad blocks must be concealed Foul up the block-to-sector calculation Methods Look-aside list of bad sectors –Check each sector request against hash table –If present, substitute a replacement sector behind the scenes Spare sectors in each track, remapped by formatting Handling Disk controller, invisible to OS Lower levels of OS; concealed from higher layers of file system and from application

13. File System Implementations CS-502 Fall 200713 Contiguous Allocation – Extents Extent: a contiguously allocated subset of a file Directory entry points to –(For file with one extent) the extent itself –(For file with multiple extents) pointer to an extent block describing multiple extents Advantages –Speed, ease of address calculation of contiguous file –Avoids (some of) the fragmentation issues –Can be adapted to support files across multiple disks …

14. File System Implementations CS-502 Fall 200714 Contiguous Allocation – Extents … Disadvantages –Too many extents  degenerates to indexed allocation As in Unix-like systems, but not so well Popular in 1960s & 70s –OS/360, other systems for commercial data processing Currently used for large files in NTFS Rarely mentioned in textbooks Silbershatz, §11.4.1 & 22.5.1

15. File System Implementations CS-502 Fall 200715 Questions?

16. File System Implementations CS-502 Fall 200716 File Allocation Schemes Contiguous –Blocks of file stored in consecutive disk sectors –Directory points to first entry Linked –Blocks of file scattered across disk, as linked list –Directory points to first entry Indexed –Separate index block contains pointers to file blocks –Directory points to index block

17. File System Implementations CS-502 Fall 200717 Linked Allocation Blocks scattered across disk Each block contains pointer to next block Directory points to first and last blocks Sector header: –Pointer to next block –ID and block number of file 10 16 01 25

18. File System Implementations CS-502 Fall 200718 Linked Allocation (Note) This is Silbershatz figure 11.5 Links in the book are incorrect 10 16 01 25

19. File System Implementations CS-502 Fall 200719 Linked Allocation Advantages –No space fragmentation! –Easy to create, extend files –Ideal for lots of small files Disadvantages –Lots of disk arm movement –Space taken up by links –Sequential access only! Random access simulated by caching links Used in Xerox Alto file system

20. File System Implementations CS-502 Fall 200720 Bad Block Management – Linked File Systems In OS:– format all sectors of disk Don’t reserve any spare sectors Allocate bad blocks to a hidden file for the purpose If a block becomes bad, append to the hidden file Advantages Very simple No look-aside or sector remapping needed Totally transparent without any hidden mechanism

21. File System Implementations CS-502 Fall 200721 Variation on Linked Allocation – File Allocation Table (FAT) Instead of link on each block, put all links in one table –the File Allocation Table — i.e., FAT Each entry corresponds to physical block in disk –Directory points to first & last blocks of file –Each block points to next block (or EOF)

22. File System Implementations CS-502 Fall 200722 FAT File Systems Advantages –Advantages of Linked File System –FAT can be cached in memory –Searchable at CPU speeds, pseudo-random access Disadvantages –Limited size, not suitable for very large disks –FAT cache describes entire disk, not just open files! –Not fast enough for large databases Used in MS-DOS, early Windows systems –Also USB Flash drives, floppy disks, etc.

23. File System Implementations CS-502 Fall 200723 Bad Block Management – FAT File Systems Same as Linked File Systems I.e., format all sectors of disk Don’t reserve any spare sectors Allocate bad blocks to a hidden file for the purpose If a block becomes bad, append to the hidden file Same advantages and disadvantages

24. File System Implementations CS-502 Fall 200724 Disk Defragmentation Re-organize blocks in disk so that file is (mostly) contiguous Link or FAT organization preserved Purpose: –To reduce disk arm movement during sequential accesses –Does not change the linked structure of the file system!

25. File System Implementations CS-502 Fall 200725 Exam Question (last spring) You have a humongous database stored in a file on a 4 GB flash drive with a FAT file system. What must the file system do to locate block n of the database? Assume that database has not been defragmented, so that its blocks are likely to be scattered randomly across the flash drive. Given that the file system has found the location of block n, what must it do to find the location of block n+1? block n-1?

26. File System Implementations CS-502 Fall 200726 Questions? Linked and FAT File Systems

27. File System Implementations CS-502 Fall 200727 File Allocation Schemes Contiguous –Blocks of file stored in consecutive disk sectors –Directory points to first entry Linked –Blocks of file scattered across disk, as linked list –Directory points to first entry Indexed –Separate index block contains pointers to file blocks –Directory points to index block

28. File System Implementations CS-502 Fall 200728 Indexed Allocation i-node: –Part of file metadata –Data structure lists the sector address of each block of file Advantages –True random access –Only i-nodes of open files need to be cached –Supports small and large files

29. File System Implementations CS-502 Fall 200729 Unix/Linux i-nodes Direct blocks: –Pointers to first n sectors Single indirect table: –Extra block containing pointers to blocks n+1.. n+m Double indirect table: –Extra block containing single indirect blocks …

30. File System Implementations CS-502 Fall 200730 Indexed Allocation Access to every block of file is via i-node Bad block management –Similar to Linked/FAT systems Disadvantage –Not as fast as contiguous allocation for large databases Requires reference to i-node for every access vs. Simple calculation of block to sector address

31. File System Implementations CS-502 Fall 200731 Indexed Allocation (continued) Widely used in Unix, Linux, Windows NTFS Robust Has withstood the test of time Many variations

32. File System Implementations CS-502 Fall 200732 Questions?

33. File System Implementations CS-502 Fall 200733 Free Block Management in File Systems Bitmap –Very compact on disk –Expensive to search –Supports contiguous allocation Free list –Linked list of free blocks Each block contains pointer to next free block –Only head of list needs to be cached in memory –Very fast to search and allocate –Contiguous allocation very difficult

34. File System Implementations CS-502 Fall 200734 Free Block Management Bit Vector … 012n-1 bit[i] =  0  block[i] free 1  block[i] occupied Free block number calculation (number of bits per word) * (number of 0-value words) + offset of first 1 bit

35. File System Implementations CS-502 Fall 200735 Free Block Management Bit Vector (continued) Bit map –Must be kept both in memory and on disk –Copy in memory and disk may differ –Cannot allow for block[i] to have a situation where bit[i] = 1 in memory and bit[i] = 0 on disk

36. File System Implementations CS-502 Fall 200736 Free Block Management Bit Vector (continued) Solution: –Set bit[i] = 1 in disk –Allocate block[i] –Set bit[i] = 1 in memory –Similarly for set of contiguous blocks Potential for lost blocks in event of crash! –Discussion:– How do we solve this problem?

37. File System Implementations CS-502 Fall 200737 Free Block Management Linked List Linked list of free blocks –Not necessarily in order! Cache first few free blocks in memory Head of list must be stored both –On disk –In memory Each block must be written to disk when freed Potential for losing blocks?

38. File System Implementations CS-502 Fall 200738 Reading Assignment Silbershatz, Chapter 11 Ignore §11.8 – 11.10 Tanenbaum (Modern Operating Systems), Chapter 6

39. File System Implementations CS-502 Fall 200739 Scalability of File Systems Question: How large can a file be? Answer: limited by –Number of bits in length field in metadata –Size & number of block entries in FAT or i-node Question: How large can file system be? Answer: limited by –Size & number of block entries in FAT or i-node

40. File System Implementations CS-502 Fall 200740 MS-DOS & Windows FAT-12 (primarily on floppy disks): 4096 512-byte blocks Only 4086 blocks usable! FAT-16 (early hard drives): 64 K blocks; block sizes up to 32 K bytes 2 GBytes max per partition, 4 partitions per disk FAT-32 (Windows 95) 2 28 blocks; up to 2 TBytes per disk Max size FAT requires 2 32 bytes in RAM!

41. File System Implementations CS-502 Fall 200741 MS-DOS File System (continued) Maximum partition for different block sizes The empty boxes represent forbidden combinations

42. File System Implementations CS-502 Fall 200742 Classical Unix Maximum number of i-nodes = 64K! How many files in a modern PC? I-node structure allows very large files, but … Limited by size of internal fields

43. File System Implementations CS-502 Fall 200743 Modern Operating Systems Need much larger, more flexible file systems Many terabytes per system Multi-terabyte files Suitable for both large and small Cache only open files in RAM

44. File System Implementations CS-502 Fall 200744 Examples of Modern File Systems Windows NTFS Silbershatz §22.5 Tanenbaum §11.7 Linux ext2fs Silbershatz §21.7.2 Other file systems … Consult your favorite Linux system documentation

45. File System Implementations CS-502 Fall 200745 New Topic

46. File System Implementations CS-502 Fall 200746 Mounting mount –t type device pathname Attach device (which contains a file system of type type) to the directory at pathname File system implementation for type gets loaded and connected to the device Anything previously below pathname becomes hidden until the device is un-mounted again The root of the file system on device is now accessed as pathname E.g., mount –t iso9660 /dev/cdrom /myCD

47. File System Implementations CS-502 Fall 200747 Mounting (continued) OS automatically mounts devices in mount table at initialization time /etc/fstab in Linux Users or applications may mount devices at run time, explicitly or implicitly — e.g., Insert a floppy disk Plug in a USB flash drive Type may be implicit in device Windows equivalent Map drive

48. File System Implementations CS-502 Fall 200748 Virtual File Systems Virtual File Systems (VFS) provide object- oriented way of implementing file systems. VFS allows same system call interface to be used for different types of file systems. The API is to the VFS interface, rather than any specific type of file system.

49. File System Implementations CS-502 Fall 200749 Schematic View of Virtual File System

50. File System Implementations CS-502 Fall 200750 Virtual File System (continued) Mounting: formal mechanism for attaching a file system to the Virtual File interface

51. File System Implementations CS-502 Fall 200751 Linux Virtual File System (VFS) A generic file system interface provided by the kernel Common object framework –superblock: a specific, mounted file system –i-node object: a specific file in storage –d-entry object: a directory entry –file object: an open file associated with a process

52. File System Implementations CS-502 Fall 200752 Linux Virtual File System (continued) VFS operations –super_operations: read_inode, sync_fs, etc. –inode_operations: create, link, etc. –d_entry_operations: d_compare, d_delete, etc. –file_operations: read, write, seek, etc.

53. File System Implementations CS-502 Fall 200753 Linux Virtual File System (continued) Individual file system implementations conform to this architecture. May be linked to kernel or loaded as modules Linux kernel 2.6 supports over 50 file systems in official version E.g., minix, ext, ext2, ext3, iso9660, msdos, nfs, smb, …

54. File System Implementations CS-502 Fall 200754 Reading references Silbershatz, §11.2.3 Robert Love, Chapter 12

55. File System Implementations CS-502 Fall 200755 Questions?

56. File System Implementations CS-502 Fall 200756 Implementation of Directories A list of [name, information] pairs Must be scalable from very few entries to very many Name: User-friendly, variable length Any language Fast access by name Information: File metadata (itself) Pointer to file metadata block (or i-node) on disk Pointer to first & last blocks of file Pointer to extent block(s) …

57. File System Implementations CS-502 Fall 200757 Very Simple Directory Short, fixed length names Attribute & disk addresses contained in directory MS-DOS, etc. name1attributesname2attributesname3attributesname4attributes …

58. File System Implementations CS-502 Fall 200758 Simple Directory Short, fixed length names Attributes in separate blocks (e.g., i-nodes) Attribute pointers are disk addresses (or i-node numbers) Older Unix versions, MS-DOS, etc. name1name2name3name4… i-node Data structures containing attributes

59. File System Implementations CS-502 Fall 200759 More Interesting Directory Variable length file names –Stored in heap at end Modern Unix, Windows Linear or logarithmic search for name Compaction needed after –Deletion, Rename attributes … name1 longer_na me3 very_long_n ame4 name2 …

60. File System Implementations CS-502 Fall 200760 Very Large Directories Hash-table implementation Each hash chain like a small directory with variable-length names Must be sorted for listing

61. File System Implementations CS-502 Fall 200761 Questions?

62. File System Implementations CS-502 Fall 200762 Raw Disk Layout Track format – n sectors –200 < n < 2000 in modern disks –Some disks have fewer sectors on inner tracks Inter-sector gap –Enables each sector to be read or written independently Sector format –Sector address: Cylinder, Track, Sector –Optional header –Data –Each field separated by small gap and with its own CRC Sector length –Almost all operating systems specify uniform sector length –512 – 4096 bytes

63. File System Implementations CS-502 Fall 200763 Bad Sector Handling – within track a)A disk track with a bad sector b)Substituting a spare for the bad sector c)Shifting all the sectors to bypass the bad one