File System Implementations CS-502 (EMC) Fall File System Implementations CS-502, Operating Systems Fall 2009 (EMC) (Slides include materials from Modern Operating Systems, 3 rd ed., by Andrew Tanenbaum and from Operating System Concepts, 7 th ed., by Silbershatz, Galvin, & Gagne)

File System Implementations CS-502 (EMC) Fall Review – the File abstraction A (potentially) large amount of information or data that lives a (potentially) very long time Often much larger than the memory of the computer Often much longer than any computation Sometimes longer than life of machine itself (Usually) organized as a linear array of bytes or blocks Internal structure is imposed by application (Occasionally) blocks may be variable length (Often) requiring concurrent access by multiple processes Even by processes on different machines!

File System Implementations CS-502 (EMC) Fall File Systems and Disks User view –File is a named, persistent collection of data OS & file system view –File is collection of disk blocks — i.e., a container –File System maps file names and offsets to disk blocks

File System Implementations CS-502 (EMC) Fall This topic – Implementation of Files Create file abstraction using physical disk devices and disk blocks –Efficient in time, space, use of disk resources –Fast enough for application requirements Must be scalable to a wide variety of file sizes –Many small files (< 1 page) –Huge files (100’s of gigabytes, terabytes, spanning disks) –Everything in between

File System Implementations CS-502 (EMC) Fall Reading Assignment Tanenbaum §4.3

File System Implementations CS-502 (EMC) Fall Overview File Allocation methods Sequential Linked and FAT Indexed Free blocks & bad blocks Scalability Mounting & Virtual File Systems Directory implementation notes

File System Implementations CS-502 (EMC) Fall Definition — Volume The fundamental unit of a file system Physical volume may be a physical disk storage device physical partition of a single disk (aka minidisk) Logical Volume is A physical volume A combination of other volumes –Usually similar in size and characteristics Volume is also used (loosely) to denote the part of the file system implemented on a physical or logical volume

File System Implementations CS-502 (EMC) Fall File Allocation Schemes Contiguous –Blocks of file stored in consecutive disk sectors –Directory points to first entry & specifies length Linked –Blocks of file scattered across disk, as linked list –Directory points to first entry Indexed –Blocks of file scattered across disk –Separate index block contains pointers to file blocks –Directory points to index block

File System Implementations CS-502 (EMC) Fall File Allocation Schemes (continued) The allocation scheme is an attribute of a file system, not of individual files within a system. All files within a file system follow same allocation model

File System Implementations CS-502 (EMC) Fall File Allocation Schemes Contiguous –Blocks of file stored in consecutive disk sectors –Directory points to first entry Linked –Blocks of file scattered across disk, as linked list –Directory points to first entry Indexed –Separate index block contains pointers to file blocks –Directory points to index block

File System Implementations CS-502 (EMC) Fall Contiguous Allocation Ideal for large, static files –Databases, fixed system structures, OS code –Multi-media video and audio –CD-ROM, DVD Simple address calculation –Directory entry points to first sector –File block i  disk sector address Fast multi-block reads and writes –Minimize seeks between blocks

File System Implementations CS-502 (EMC) Fall Contiguously Allocated Files

File System Implementations CS-502 (EMC) Fall Block-to-sector Calculation To find disk sector containing block i of file f –Starting_block(f) + i Starting block of each file is named in –Directory, or –File metadata

File System Implementations CS-502 (EMC) Fall File Creation (Contiguous File System) Search for an empty sequence of blocks –First-fit –Best-fit Prone to fragmentation when … Files come and go Files change size Similar to physical memory allocation in base-limit type of virtual memory

File System Implementations CS-502 (EMC) Fall Contiguous Allocation – Extents Extent: a contiguously allocated subset of a file Directory entry points to –(For file with one extent) the extent itself –(For file with multiple extents) pointer to an extent block describing multiple extents Advantages –Speed, ease of address calculation of contiguous file –Avoids (some of) the fragmentation issues –Can be adapted to support files across multiple disks …

File System Implementations CS-502 (EMC) Fall Contiguous Allocation – Extents … Disadvantages –Too many extents  degenerates to indexed allocation As in Unix-like systems, but not so well Popular in 1960s & 70s –OS/360, other systems for commercial data processing Currently used for large files in NTFS Rarely mentioned in textbooks Silbershatz, 7 th ed., § &

File System Implementations CS-502 (EMC) Fall Digression: Bad Block Management Bad blocks on disks are inevitable Part of manufacturing process (less than 1%) Most are detected during formatting Occasionally, blocks become bad during operation Manufacturers typically add extra tracks to disks Physical capacity = (1 + x) * rated_capacity Who handles bad blocks? Disk controller: Bad block list maintained internally –Automatically substitutes good blocks Formatter: Re-organize track to avoid bad blocks OS: Bad block list maintained by OS, bad blocks never used

File System Implementations CS-502 (EMC) Fall Bad Block Management in Contiguous Allocation File Systems Bad blocks must be concealed Foul up the block-to-sector calculation Methods Look-aside list of bad sectors –Check each sector request against hash table –If present, substitute a replacement sector behind the scenes Spare sectors in each track, remapped by formatting Handling Disk controller, invisible to OS Lower levels of OS; concealed from higher layers of file system and from application

File System Implementations CS-502 (EMC) Fall Questions?

File System Implementations CS-502 (EMC) Fall File Allocation Schemes Contiguous –Blocks of file stored in consecutive disk sectors –Directory points to first entry Linked –Blocks of file scattered across disk, as linked list –Directory points to first entry Indexed –Separate index block contains pointers to file blocks –Directory points to index block

File System Implementations CS-502 (EMC) Fall Linked Allocation Blocks scattered across disk Each block contains pointer to next block Directory points to first and last blocks Sector header: –Pointer to next block –ID and block number of file

File System Implementations CS-502 (EMC) Fall Linked Allocation Advantages –No external fragmentation of file space! –Easy to create, extend files –Ideal for lots of small files Disadvantages –Lots of disk arm movement –Space taken up by links –Sequential access only! Random access simulated by caching links Used in Xerox Alto file system

File System Implementations CS-502 (EMC) Fall Bad Block Management – Linked File Systems In OS:– format all sectors of disk Don’t reserve any spare sectors Allocate bad blocks to a hidden file for the purpose If a block becomes bad, “append” to the hidden file Advantages Very simple No look-aside or sector remapping needed Totally transparent without any hidden mechanism

File System Implementations CS-502 (EMC) Fall Linked File System – Limitation To access the i th block, it is necessary to physically read the first (i  1) blocks from disk! Serious problem for large files!

File System Implementations CS-502 (EMC) Fall Solution – File Allocation Table (FAT) Instead of link on each block, put all links in one table — the FAT Each entry corresponds to physical block in disk i th entry contains link for block i –Each entry points to next block (or EOF) Directory points to first & last blocks of file

File System Implementations CS-502 (EMC) Fall FAT File Systems Advantages –Advantages of Linked File System –FAT can be cached in memory –Searchable at CPU speeds  pseudo-random access Disadvantages –Limited size, not suitable for very large disks –FAT cache describes entire disk, not just open files! –Not fast enough for large databases Used in MS-DOS, early Windows systems –Also USB Flash drives, floppy disks, etc.

File System Implementations CS-502 (EMC) Fall Bad Block Management – FAT File Systems Same as Linked File Systems I.e., format all sectors of disk Don’t reserve any spare sectors Allocate bad blocks to a hidden file for the purpose If a block becomes bad, append to the hidden file Same advantages and disadvantages

File System Implementations CS-502 (EMC) Fall Disk Defragmentation Re-organize blocks in disk so that file is (mostly) contiguous Link or FAT organization preserved Purpose: –To reduce disk arm movement during sequential accesses –To permit contiguous reads or writes in one disk operation Does not change the linked structure of the file system!

File System Implementations CS-502 (EMC) Fall Exam Question (last spring) You have a humongous database stored in a file on a 4 GB flash drive with a FAT file system. What must the file system do to locate block n of the database? Assume that database has not been defragmented, so that its blocks are likely to be scattered randomly across the flash drive. Given that the file system has found the location of block n, what must it do to find the location of block n+1? block n-1?

File System Implementations CS-502 (EMC) Fall Questions? Linked and FAT File Systems

File System Implementations CS-502 (EMC) Fall File Allocation Schemes Contiguous –Blocks of file stored in consecutive disk sectors –Directory points to first entry Linked –Blocks of file scattered across disk, as linked list –Directory points to first entry Indexed –Separate index block contains pointers to file blocks –Directory points to index block

File System Implementations CS-502 (EMC) Fall Problems and Issues Contiguous file systems not suitable for files that come and go rapidly & frequently Linked file systems not suitable for very large disks or a lot of random access Can we do better?

File System Implementations CS-502 (EMC) Fall Indexed Allocation i-node: –Contains file metadata –Lists sector address of each block of file Advantages –True random access –Only i-nodes of open files need to be cached –Supports small and large files

File System Implementations CS-502 (EMC) Fall Unix/Linux i-nodes Direct blocks: –Pointers to first n sectors Single indirect table: –Extra block containing pointers to blocks n+1.. n+m Double indirect table: –Extra block containing single indirect blocks …

File System Implementations CS-502 (EMC) Fall Indexed Allocation Access to every block of file is via i-node Bad block management –Similar to Linked/FAT systems Disadvantage –Not as fast as contiguous allocation for large databases Requires reference to i-node for every access vs. Simple calculation of block to sector address

File System Implementations CS-502 (EMC) Fall Indexed Allocation (continued) Widely used in Unix, Linux, Windows NTFS Robust Has withstood the test of time Many variations

File System Implementations CS-502 (EMC) Fall Questions?

File System Implementations CS-502 (EMC) Fall Free Block Management in File Systems Bitmap –Very compact on disk –Expensive to search –Supports contiguous allocation Free list –Linked list of free blocks Each block contains pointer to next free block –Only head of list needs to be cached in memory –Very fast to search and allocate –Contiguous allocation very difficult

File System Implementations CS-502 (EMC) Fall Free Block Management Bit Vector … 012n-1 bit[i] =  0  block[i] free 1  block[i] occupied Free block number calculation (number of bits per word) * (number of 0-value words) + offset of first 1 bit

File System Implementations CS-502 (EMC) Fall Free Block Management Bit Vector (continued) Bit map –Must be kept both in memory and on disk –Copy in memory and disk may differ –Cannot allow for block[i] to have a situation where bit[i] = 1 in memory and bit[i] = 0 on disk

File System Implementations CS-502 (EMC) Fall Free Block Management Bit Vector (continued) Solution: –Set bit[i] = 1 in disk –Allocate block[i] –Set bit[i] = 1 in memory –Similarly for set of contiguous blocks Potential for lost blocks in event of crash! –Discussion:– How do we solve this problem?

File System Implementations CS-502 (EMC) Fall Free Block Management Linked List Linked list of free blocks –Not necessarily in order! Cache first few free blocks in memory Head of list must be stored both –On disk –In memory Each block must be written to disk when freed Potential for losing blocks?

File System Implementations CS-502 (EMC) Fall Free Block Management – Linked List (continued) Can also be implemented in FAT Can also be implemented in Indexed File system

File System Implementations CS-502 (EMC) Fall Reading Assignment Tanenbaum §4.3

File System Implementations CS-502 (EMC) Fall Scalability of File Systems Question: How large can a file be? Answer: limited by –Number of bits in length field in metadata –Size & number of block entries in FAT or i-node Question: How large can file system be? Answer: limited by –Size & number of block entries in FAT or i-node

File System Implementations CS-502 (EMC) Fall MS-DOS & Windows FAT-12 (primarily on floppy disks): byte blocks Only 4086 blocks usable! FAT-16 (early hard drives): 64 K blocks; block sizes up to 32 K bytes 2 GBytes max per partition, 4 partitions per disk FAT-32 (Windows 95) 2 28 blocks; up to 2 TBytes per disk Max size FAT requires 2 32 bytes in RAM!

File System Implementations CS-502 (EMC) Fall MS-DOS File System (continued) Maximum partition for different block sizes The empty boxes represent forbidden combinations

File System Implementations CS-502 (EMC) Fall Classical Unix Maximum number of i-nodes = 64K! How many files in a modern PC? I-node structure allows very large files, but … Limited by size of internal fields Limited by # of entries in triple-indirect block

File System Implementations CS-502 (EMC) Fall Modern Operating Systems Need much larger, more flexible file systems Many terabytes per system Multi-terabyte files Suitable for both large and small Cache only open files in RAM

File System Implementations CS-502 (EMC) Fall Examples of Modern File Systems Windows NTFS Tanenbaum §11.8 Linux ext2 and ext3 Tanenbaum § Other file systems … Consult your favorite Linux system documentation

File System Implementations CS-502 (EMC) Fall New Topic

File System Implementations CS-502 (EMC) Fall Mounting mount –t type device pathname Attach device (which contains a file system of type type) to the directory at pathname File system implementation for type gets loaded and connected to the device Anything previously below pathname becomes hidden until the device is un-mounted again The root of the file system on device is now accessed as pathname E.g., mount –t iso9660 /dev/cdrom /myCD

File System Implementations CS-502 (EMC) Fall Mounting (continued) OS automatically mounts devices in mount table at initialization time /etc/fstab in Linux Users or applications may mount devices at run time, explicitly or implicitly — e.g., Insert a floppy disk Plug in a USB flash drive Type may be implicit in device Windows equivalent Map drive

File System Implementations CS-502 (EMC) Fall Virtual File Systems Virtual File Systems (VFS) provide object- oriented way of implementing file systems. VFS allows same system call interface to be used for different types of file systems. The API is to the VFS interface, rather than any specific type of file system.

File System Implementations CS-502 (EMC) Fall Schematic View of Virtual File System

File System Implementations CS-502 (EMC) Fall Virtual File System (continued) Mounting: formal mechanism for attaching a file system to the Virtual File interface

File System Implementations CS-502 (EMC) Fall Linux Virtual File System (VFS) A generic file system interface provided by the kernel Common object framework –superblock: a specific, mounted file system –i-node object: a specific file in storage –d-entry object: a directory entry –file object: an open file associated with a process

File System Implementations CS-502 (EMC) Fall Linux Virtual File System (continued) VFS operations –super_operations: read_inode, sync_fs, etc. –inode_operations: create, link, etc. –d_entry_operations: d_compare, d_delete, etc. –file_operations: read, write, seek, etc.

File System Implementations CS-502 (EMC) Fall Linux Virtual File System (continued) Individual file system implementations conform to this architecture. May be linked to kernel or loaded as modules Linux kernel 2.6 supports over 50 file systems in official version E.g., minix, ext, ext2, ext3, iso9660, msdos, nfs, smb, …

File System Implementations CS-502 (EMC) Fall Reading references Tanenbaum §4.3, , 11.8 Also:– Robert Love, Linux Kernel Development, 2 nd edition, Chapter 12

File System Implementations CS-502 (EMC) Fall Questions?

File System Implementations CS-502 (EMC) Fall Implementation of Directories A list of [name, information] pairs Must be scalable from very few entries to very many Name: User-friendly, variable length Any language Fast access by name Information: File metadata (itself) Pointer to file metadata block (or i-node) on disk Pointer to first & last blocks of file Pointer to extent block(s) …

File System Implementations CS-502 (EMC) Fall Very Simple Directory Short, fixed length names Attribute & disk addresses contained in directory MS-DOS, etc. name1attributesname2attributesname3attributesname4attributes …

File System Implementations CS-502 (EMC) Fall Simple Directory Short, fixed length names Attributes in separate blocks (e.g., i-nodes) Attribute pointers are disk addresses (or i-node numbers) Older Unix versions, MS-DOS, etc. name1name2name3name4… i-node Data structures containing attributes

File System Implementations CS-502 (EMC) Fall More Interesting Directory Variable length file names –Stored in heap at end Modern Unix, Windows Linear or logarithmic search for name Compaction needed after –Deletion, Rename attributes … name1 longer_na me3 very_long_n ame4 name2 …

File System Implementations CS-502 (EMC) Fall Very Large Directories Hash-table implementation Each hash chain like a small directory with variable-length names Must be sorted for listing

File System Implementations CS-502 (EMC) Fall Questions?

File System Implementations CS-502 (EMC) Fall Raw Disk Layout Track format – n sectors –200 < n < 2000 in modern disks –Some disks have fewer sectors on inner tracks Inter-sector gap –Enables each sector to be read or written independently Sector format –Sector address: Cylinder, Track, Sector –Optional header –Data –Each field separated by small gap and with its own CRC Sector length –Almost all operating systems specify uniform sector length –512 – 4096 bytes

File System Implementations CS-502 (EMC) Fall Bad Sector Handling – within track a)A disk track with a bad sector b)Substituting a spare for the bad sector c)Shifting all the sectors to bypass the bad one