CHAPTER 12: FILE SYSTEM IMPLEMENTATION File System Structure File System Implementation Directory Implementation Allocation Methods Free-Space Management Efficiency and Performance Recovery Log-Structured File Systems NFS

FILE-SYSTEM STRUCTURE Disks provide the bulk of secondary storage on which a file system is maintained can be written in place Sequential access and direct access I/O in blocks not in bytes File system allows the data on disks to be stored, located, and retrieved easily To define how the FS should look to the user To create algorithms and data structures to map the logical FS onto the physical secondary-storage devices.

File-System Structure: Layered file system

File-System Structure: Layered file system I/O control consists of device drivers and interrupt handlers to transfer information between the main memory and the disk system. Input: retrieve drive 1, cylinder 73, track 2, sector 10 Output: low-level, hardware-specific instructions that are used by the hardware controller Basic file system issues generic commands to the appropriate device driver to read and write physical blocks on the disk Input: retrieve block 123 Output: retrieve drive 1, cylinder 73, track 2, sector 10

File-System Structure: Layered file system The file-organization module knows about the files and their logical blocks, as well as physical blocks. To translate a file’s logical block addresses to its physical block addresses. Each file’s logical block addresses are numbered from 0 (or 1) through N. Each file’s physical block addresses are different, are unique within a partition. Free-space manager: Tracks unallocated blocks And provides these blocks when requested.

File-System Structure: Layered file system The logical file system manages metadata information (metadata v.s. actual data) To manage the directory structure To manage the file structures via FCB (file control blocks) Contains information such as ownership, permissions, location of the file contents Responsible for protection and security.

File-System Structure: Example FSes Windows: FAT (File Allocation Table) (12, 16, 32) NTFS (Windows NT File System) UNIX: UFS (Unix File system)  ext2  ext3 more /proc/filesystems cd /usr/src/linux-2.4/fs # to explore for more.

FILE-SYSTEM IMPLEMENTATION Overview: Data structures for implementing a FS On-disk structures In-memory structures Partitions and mounting VFS

File-System Implementation: Overview On-Disk structures Boot control block (boot block, partition boot sector) Used to boot an OS from that partition. Partition control block (super block, Master File Table) Block numbers, block size, free-block count, free-block pointers, free FCB count and FCB pointers Directory structure used to organized the files Linear list / hash tables FCB (inode, vnode, Master File Table record) File permissions, ownership, size, location of the data blocks

File-System Implementation: Overview In-memory structures used for both file-system management and performance improvement via caching In-memory partition table containing information about each mounted partition. In-memory directory structures containing the directory information of recently accessed directories. System-wide open-file table containing a copy of the FCB of each open file as well as other information. Per-process open file-tables containing a pointer to the appropriate entry in the system-wide open-file table, as well as other information.

File-System Implementation: Overview To create a new file An application program calls the logical file system The logical file system To allocate a new FCB (see the next slide for FCB) To read in the appropriate directory UNIX treats a directory exactly as a file. Windows NT treats a directory as a record inside the MFT. To add a new entry To fill in the new entry with the filename and the new FCB To write it back to the disk

File-System Implementation: Overview: FCB

File-System Implementation: Overview To open a file To pass a file name to the logical file system To search the directory for the given file name To read in the file’s FCB To put the file’s FCB to the system-wide open-file table To add a new entry in the per-process open-file table, with a pointer to the entry in the system-wide open-table and some other fields To return a pointer to the appropriate entry in the per-process file system table. (file descriptor or file handle)

File-System Implementation: Overview To close a file To remove the entry in the per-process open-file table To decrement the system-wide open-file entry’s open count. If open count is 0, copy the updated file information to the disk-based directory structure and delete this entry.

File-System Implementation: Overview

File-System Implementation: Overview To use the file system as interface to other system aspects, such as networking. To use caches to speed up the file operations BSD UNIX To use the file system for other purpose such as networking interface.

File-System Implementation: partitions and mounting Partitions vs disks A disk can be sliced into multiple partitions A partition can span multiple disks Partitions can either be “raw” or “cooked” Raw partition: No file system swap space, database Cooked partition: Has file system Boot block can be used for selective booting. Super block

File-System Implementation: partitions and mounting To mount a partition before using its FS Manual mounting vs automatic mounting How to mount a partition To read in the super block via its device driver To verify its consistency To repair it if necessary (fsck) To add an entry in the in-memory mount table structure. How to mount a partition for Windows To mount at boot To mount manually

File-System Implementation: partitions and mounting How to mount a partition for UNIX To mount a partition at a directory To add a entry at the mount-table To let one field of the mount-table entry point to the super block of the FS on that device To set a flag in the in-memory copy of the inode for that directory, indicating this directory is a mount point To set a field in the in-memory copy of the inode for that directory point to an entry in the mount table, indicating which device is mounted there.

File-System Implementation: VFS How to support multiple FSes？ How to integrate many Fses into a directory structure? How to seamlessly move among various FSes?  To write directory and file routines for each types. To use VFS (VFS uses oo techniques to simplify, organize, and modularize the implementation) Contributed by SUN Microsystems.

File-System Implementation: VFS Top layer: file-system interface Open, read, write, and close and file descriptors The middle layer: VFS To separate FS generic operations from their implementation by defining a clean VFS interface The VFS is based on a file-representation, called a vnode, that contains a numerical designator for a network-wide unique file. (UNIX inodes are unique within only a single file system) The bottom layer Various FS implementation Ext3 NFS

File-System Implementation: VFS

DIRECTORY IMPLEMENTATION Linear list To use a linear list of file names with pointers to the data blocks. To find a file To require a linear search. To create a file To search the directory to make sure no existing file has the same file name. To add a new entry at the end of the directory. To delete a file To search the directory for the file. To remove the entry. To free the space allocated to this file.

Directory Implementation Linear list To reuse a directory entry To mark the entry as used. To attach it to a list of free directory entries. To decrease the length of the directory. Discussion: Simple to program, time-consuming to execute The searching is expensive Binary search To keep it sorted list To use B-tree

Directory Implementation Hash Table To use a linear list to store the directory entries and to use a hash table to quickly find out the directory entry given a file name. No collisions allowed (Each hash entry has a single value) Use a hash function to map file name to a hash value. Hash function should be dynamically changed. Fastest. Collisions allowed (Each hash entry has a list of multiple values) Use a hash function to map file name to a hash value and use this value to index the hash table and then search the list to find out the directory entry. Faster.

ALLOCATION METHODS An allocation method refers to how disk blocks are allocated for files: Contiguous allocation Linked allocation Indexed allocation

Allocation Methods: Contiguous Allocation The contiguous-allocation methods requires each file to occupy a set of contiguous blocks on the disk The directory for a file consists only its starting location (block #) and length (number of blocks) (See the next slide) Discussions: Supports both sequential access and direct access. Simple to implement. External fragmentation (dynamic storage-allocation problem). How to specify an initial size for a file Under estimating its size Over estimating its size

Allocation Methods: Contiguous Allocation

Allocation Methods: Contiguous Allocation Extent-based file system Extent-based file systems allocate disk blocks in extents. An extent is a contiguous block of disks. Extents are allocated for file allocation. A file consists of one or more extents. Many newer file systems (I.e. Veritas File System) use a modified contiguous allocation scheme. Contiguous allocation can be combined with other allocation methods. Contiguous allocation for small files Other allocations for large files.

Allocation Methods: Linked Allocation With linked allocation, Each file is a linked list of disk blocks; The disk blocks may be scattered anywhere on the disk. The directory contains a pointer to the first and last blocks of the file. (See the next slide) Disk block An example linked file (see the next slide) pointer block =

Allocation Methods: Linked Allocation

Allocation Methods: Linked Allocation Simple – need only starting address Free-space management system – no waste of space No random access Pointers waste space: to use clusters rather than sectors To improve usage To speed up Poor reliability Imagine the pointer is messed up.

Allocation Methods: Linked Allocation FAT FS FAT (File Allocation Table) duplicated Supports direct access Cached Poor disk utilization

Allocation Methods: Linked Allocation

Allocation Methods: Indexed Allocation Problems: External fragmentation and size-declaration for contiguous allocation Direct access for linked allocation Indexed allocation Bringing all the pointers together into one location: index block (See the next slide) Logical view index table

Allocation Methods: Indexed Allocation

Allocation Methods: Indexed Allocation Need index block Access methods: sequential access and direct access. Mapping from logical to physical in a file of maximum size of 256K words (or 1024KB) and block size of 512 words. We need only 1 block for index table. (512x2KB=1024KB) Index blocks waste space How large should the index block be? Linked scheme Multilevel index Combined scheme

Allocation Methods: Indexed Allocation Linked scheme For a small file, one index block For a large file, more index blocks can be linked together. Multilevel index 1-level index block: 1024*4KB 2-level index block: 1024x1024*4KB 3-level index block: 1024x1024*1024*4KB (similar to paging)

Allocation Methods: Indexed Allocation  outer-index index table file

Allocation Methods: Indexed Allocation Combined scheme 12 for direct pointers 1 single-indirect block 1 double-indirect block 1 triple-indirect block

Allocation Methods: Indexed Allocation

Allocation Methods: Performance Two criteria: Storage utilization efficiency Data block access time Contiguous allocation: Good for known-size file Linked allocation: Good for storage utilization Indexed allocation: Access time depends on index structure, file size, block position Conclusion: Combining contiguous allocation and linked allocation (Some OS) Combining contiguous allocation and index allocation (SUN)

FREE-SPACE MANAGEMENT Bit vector Linked Lists Grouping Counting

Free-Space Management: Bit vector The free-space list is implemented as a bit map or bit vector. Each block is represented by 1 bit. If the block is free, the bit is 1; if the block is allocated, the bit is 0. An example 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 0 0 1 1 1 1 0 0 1 1 1 1 1 1 0 0

Free-Space Management: Bit vector Discussion: Simple to implement Efficient to find the first free block Easy to get contiguous files Block number calculation: (number of bits per word) *(number of 0-value words) +offset of first 1 bit Bit map requires extra space. Example: block size = 212 bytes disk size = 230 bytes (1 gigabyte) n = 230/212 = 218 bits (or 32K bytes)

Free-Space Management: Linked Lists Linked list (free list) Cannot get contiguous space easily No waste of space

Free-Space Management: Others Grouping To linked blocks to store the addresses To group n-1 addresses into an address block To use the last address in an address block to point to next address block. Easier to find a large number of free blocks. Counting Every entry is a pair of (starting address, contiguous block number) rather than just a address The total list is smaller.

EFFICIENCY AND PERFORMANCE Preallocating the i-node on a partition (UNIX FS) Different cluster size (BSD UNIX) File size To fix the parameters or dynamically change the parameters.

Efficiency and Performance Page Cache I/O Without a Unified Buffer Cache I/O Using a Unified Buffer Cache Reads/Writes Synchronous reads (initially) /Aynchronous reads (later on) Aynchronous writes (normally) / Synchronous write/Aynchronous (sometimes) RAM Disks vs OS caching.

Efficiency and Performance I/O Without a Unified Buffer Cache

Efficiency and Performance I/O Using a Unified Buffer Cache

Efficiency and Performance: Various Disk-Caching Locations

Recovery Consistency checking Metadata are more important than actual data  UNIX caches directory entries for reads, Not any data write that results in space allocation, or other metadata changes. Backup and restore Backup scheme Day 1: complete backup Day 2,3,4,…n: incremental backup Save the backup in different places.

LOG STRUCTURED FILE SYSTEMS An file operation, such as file create, can involve many structural changes within the file system on the disk. Directory structures are modified FCBs are allocated Data blocks are allocated The free counts for all of these blocks are decreased, … An file operation can be interrupted, can cause inconsistency, difficult to recover Transaction (DBMS).

Log Structured File System Log structured (or journaling) file systems record each update to the file system as a transaction. All transactions are written to a log. A transaction is considered committed once it is written to the log. However, the file system may not yet be updated. The transactions in the log are asynchronously written to the file system. When the file system is modified, the transaction is removed from the log. If the file system crashes, all remaining transactions in the log must still be performed.

NFS An implementation and a specification of a software system for accessing remote files across LANs (or WANs). Exports mount

Homework 12.1 12.4 12.5 12.6