File Systems

Main Points File layout Directory layout

File Systems (1) Essential requirements for long-term information storage: It must be possible to store a very large amount of information. Information must survive termination of process using it. Multiple processes must be able to access information concurrently. Tanenbaum & Bo, Modern Operating Systems:4th ed., (c) 2013 Prentice-Hall, Inc. All rights reserved.

File Systems (2) Think of a disk as a linear sequence of fixed-size blocks and supporting two operations: Read block k. Write block k Tanenbaum & Bo, Modern Operating Systems:4th ed., (c) 2013 Prentice-Hall, Inc. All rights reserved.

File Operations Create Append Delete Seek Open Get attributes Close Read Write Append Seek Get attributes Set attributes Rename Tanenbaum & Bo, Modern Operating Systems:4th ed., (c) 2013 Prentice-Hall, Inc. All rights reserved.

The External View of the File Manager Application Program CreateFile() ReadFile() CloseHandle() SetFilePointer() WriteFile() mount() write() open() close() read() lseek() File Mgr Device Mgr Memory Mgr Process Mgr File Mgr Device Mgr Memory Mgr Process Mgr UNIX Windows Hardware Operating Systems: A Modern Perspective, Chapter 13

Figure 4-3. (a) An executable file. (b) An archive File Types Figure 4-3. (a) An executable file. (b) An archive Tanenbaum & Bo, Modern Operating Systems:4th ed., (c) 2013 Prentice-Hall, Inc. All rights reserved.

Figure 4-4. Some possible file attributes. Tanenbaum & Bo, Modern Operating Systems:4th ed., (c) 2013 Prentice-Hall, Inc. All rights reserved.

File Management File is a named, ordered collection of information The file manager administers the collection by: Storing the information on a device Mapping the block storage to a logical view Allocating/deallocating storage Providing file directories Operating Systems: A Modern Perspective, Chapter 13

Low-level File System Architecture Block 0 b0 b1 b2 b3 … … bn-1 . . . Sequential Device Randomly Accessed Device Operating Systems: A Modern Perspective, Chapter 13

File Structure Figure 4-2. Three kinds of files. (a) Byte sequence. (b) Record sequence. (c) Tree. Tanenbaum & Bo, Modern Operating Systems:4th ed., (c) 2013 Prentice-Hall, Inc. All rights reserved.

Information Structure Applications Records Byte Stream Files Stream-Block Translation Storage device Operating Systems: A Modern Perspective, Chapter 13

Byte Stream File Interface fileID = open(fileName) close(fileID) read(fileID, buffer, length) write(fileID, buffer, length) seek(fileID, filePosition) Operating Systems: A Modern Perspective, Chapter 13

Low Level Files b0 b1 b2 bi ... ... Stream-Block Translation fid = open(“fileName”,…); … read(fid, buf, buflen); close(fid); ... ... b0 b1 b2 bi int open(…) {…} int close(…) {…} int read(…) {…} int write(…) {…} int seek(…) {…} Stream-Block Translation Storage device response to commands Operating Systems: A Modern Perspective, Chapter 13

Structured Record Files Structured Files Records Structured Record Files Record-Block Translation Operating Systems: A Modern Perspective, Chapter 13

Record-Oriented Sequential Files Logical Record fileID = open(fileName) close(fileID) getRecord(fileID, record) putRecord(fileID, record) seek(fileID, position) Operating Systems: A Modern Perspective, Chapter 13

Record-Oriented Sequential Files Logical Record H byte header k byte logical record ... Operating Systems: A Modern Perspective, Chapter 13

Record-Oriented Sequential Files Logical Record H byte header k byte logical record ... ... Physical Storage Blocks Fragment Operating Systems: A Modern Perspective, Chapter 13

Indexed Sequential File Suppose we want to directly access records Add an index to the file fileID = open(fileName) close(fileID) getRecord(fileID, index) index = putRecord(fileID, record) deleteRecord(fileID, index) Operating Systems: A Modern Perspective, Chapter 13

Indexed Sequential File (cont) Application structure Account # 012345 123456 294376 ... 529366 965987 Index i k j index = i index = k index = j Operating Systems: A Modern Perspective, Chapter 13

File System Design File System is an organized collection of regular files and directories (mkfs) Data structures Directories: file name -> file metadata Store directories as files File metadata: how to find file data blocks Free map: list of free disk blocks

File System Design Constraints For small files: Small blocks for storage efficiency Files used together should be stored together For large files: Contiguous allocation for sequential access Efficient lookup for random access May not know at file creation Whether file will become small or large

Design Challenges Index structure Index granularity Free space How do we locate the blocks of a file? Index granularity What block size do we use? Free space How do we find unused blocks on disk? Locality How do we preserve spatial locality? Reliability What if machine crashes in middle of a file system op?

Block Management The job of selecting & assigning storage blocks to the file Three basic strategies: Contiguous allocation Linked lists Indexed allocation Operating Systems: A Modern Perspective, Chapter 13

Contiguous Allocation Maps the N blocks into N contiguous blocks on the secondary storage device Difficult to support dynamic file sizes File descriptor Head position 237 … First block 785 Number of blocks 25 Operating Systems: A Modern Perspective, Chapter 13

Implementing Files Contiguous Layout Contiguous allocation of disk space for seven files. (b) The state of the disk after files D and F have been removed. Tanenbaum & Bo, Modern Operating Systems:4th ed., (c) 2013 Prentice-Hall, Inc. All rights reserved.

Linked Lists Each block contains a header with Number of bytes in the block Pointer to next block Blocks need not be contiguous Files can expand and contract Seeks can be slow First block … Head: 417 ... Length Length Length Byte 0 Byte 0 Byte 0 ... ... ... Byte 4095 Byte 4095 Byte 4095 Block 0 Block 1 Block N-1 Operating Systems: A Modern Perspective, Chapter 13

Linked List Allocation

Indexed Allocation Extract headers and put them in an index Simplify seeks May link indices together (for large files) Byte 0 ... Index block … Head: 417 ... Byte 4095 Length Block 0 Length Byte 0 ... Byte 4095 Block 1 Byte 0 ... Length Byte 4095 Block N-1 Operating Systems: A Modern Perspective, Chapter 13

File Systems Traditional FFS file system (Linux) Microsoft’s FAT, FAT2 and NTFS file systems Journaling file systems, ext3 … others

File System Design Options FAT FFS NTFS Index structure Linked list Tree (fixed, assym) (dynamic) granularity block extent free space allocation FAT array Bitmap (fixed location) (file) Locality defragmentation Block groups + reserve space Extents Best fit defrag

Microsoft File Allocation Table (FAT) Linked list index structure Simple, easy to implement Still widely used (e.g., thumb drives) File table: Linear map of all blocks on disk Each file a linked list of blocks

FAT

FAT Pros: Cons: Easy to find free block Easy to append to a file Easy to delete a file Cons: Small file access is slow Random access is very slow Fragmentation File blocks for a given file may be scattered Files in the same directory may be scattered Problem becomes worse as disk fills

Berkeley UNIX FFS (Fast File System) inode table Analogous to FAT table inode Metadata File owner, access permissions, access times, … Set of 12 data pointers With 4KB blocks => max size of 48KB files

File System Structure Basic unit for allocating space on the disk is a block Disk partition partition partition File System Boot Block Super-block i-node table Data blocks

I-nodes Each file or directory in the file system has a unique entry in the i-node table. File type (regular, symbolic link, directory…) Owner Permissions Timestamps for last access; last modification, last status change Size …

i-node entry … 5 11 12 13 14 15 DB 0 DB 5 DB IPB IPB IPB DB

FFS inode Metadata Set of 12 data pointers Indirect block pointer File owner, access permissions, access times, … Set of 12 data pointers With 4KB blocks => max size of 48KB files Indirect block pointer pointer to disk block of data pointers Indirect block: 1K data blocks => 4MB (+48KB)

FFS inode Metadata Set of 12 data pointers Indirect block pointer File owner, access permissions, access times, … Set of 12 data pointers With 4KB blocks => max size of 48KB Indirect block pointer pointer to disk block of data pointers 4KB block size => 1K data blocks => 4MB Doubly indirect block pointer Doubly indirect block => 1K indirect blocks 4GB (+ 4MB + 48KB)

FFS inode Metadata Set of 12 data pointers Indirect block pointer File owner, access permissions, access times, … Set of 12 data pointers With 4KB blocks => max size of 48KB Indirect block pointer pointer to disk block of data pointers 4KB block size => 1K data blocks => 4MB Doubly indirect block pointer Doubly indirect block => 1K indirect blocks 4GB (+ 4MB + 48KB) Triply indirect block pointer Triply indirect block => 1K doubly indirect blocks 4TB (+ 4GB + 4MB + 48KB)

Disk Organization Boot Sector Volume Directory Track 0, Cylinder 0 Blk0 Blk1 … Blkk-1 Track 0, Cylinder 0 … Blkk Blkk+1 Blk2k-1 Track 0, Cylinder 1 … … Blk Blk Blk Track 1, Cylinder 0 … … Blk Blk Blk Track N-1, Cylinder 0 … … Blk Blk Blk Track N-1, Cylinder M-1 Operating Systems: A Modern Perspective, Chapter 13

FFS Locality Block group allocation inode table spread throughout disk Block group is a set of nearby cylinders Files in same directory located in same group Subdirectories located in different block groups inode table spread throughout disk inodes, bitmap near file blocks First fit allocation Small files fragmented, large files contiguous

FFS First Fit Block Allocation

FFS First Fit Block Allocation

FFS First Fit Block Allocation

FFS Pros Cons Efficient storage for both small and large files Locality for both small and large files Locality for metadata and data Cons Inefficient for tiny files (a 1 byte file requires both an inode and a data block) Inefficient encoding when file is mostly contiguous on disk (no equivalent to superpages) Need to reserve 10-20% of free space to prevent fragmentation

NTFS Master File Table Extents Journalling for reliability Flexible 1KB storage for metadata and data Extents Block pointers cover runs of blocks Similar approach in linux (ext4) File create can provide hint as to size of file Journalling for reliability Discussed next time

NTFS Small File

NTFS Medium File

NTFS Indirect Block

NTFS Multiple Indirect Blocks

13 File Management Operating Systems: A Modern Perspective, Chapter 13

An open() Operation Locate the on-device (external) file descriptor Extract info needed to read/write file Authenticate that process can access the file Create an internal file descriptor in primary memory Create an entry in a “per process” open file status table Allocate resources, e.g., buffers, to support file usage Operating Systems: A Modern Perspective, Chapter 13

File Manager Data Structures Keep the state of the process-file session 2 Copy info from external to the open file descriptor 1 Open File Descriptor Process-File Session 3 Return a reference to the data structure External File Descriptor Operating Systems: A Modern Perspective, Chapter 13

Opening a UNIX File On-Device File Descriptor Open File Table fid = open(“fileA”, flags); … read(fid, buffer, len); On-Device File Descriptor 0 stdin 1 stdout 2 stderr 3 ... File structure inode Open File Table Internal File Descriptor Operating Systems: A Modern Perspective, Chapter 13

File Descriptors External name Current state Sharable Owner User Locks Protection settings Length Time of creation Time of last modification Time of last access Reference count Storage device details Operating Systems: A Modern Perspective, Chapter 13

Marshalling the Byte Stream Must read at least one buffer ahead on input Must write at least one buffer behind on output Seek  flushing the current buffer and finding the correct one to load into memory Inserting/deleting bytes in the interior of the stream Operating Systems: A Modern Perspective, Chapter 13

Full Block Buffering Storage devices use block I/O Files place an explicit order on the bytes Therefore, it is possible to predict what is likely to be read after bytei When file is opened, manager reads as many blocks ahead as feasible After a block is logically written, it is queued for writing behind, whenever the disk is available Buffer pool – usually variably sized, depending on virtual memory needs Interaction with the device manager and memory manager Operating Systems: A Modern Perspective, Chapter 13

File-Descriptor Table 1 2 3 File descriptor ref count access mode file location inode pointer . . . User address space n–1 Kernel address space

Allocation of File Descriptors Whenever a process requests a new file descriptor, the lowest numbered file descriptor not already associated with an open file is selected; thus #include <fcntl.h> #include <unistd.h> close(0); fd = open("file", O_RDONLY); will always associate file with file descriptor 0 (assuming that the open succeeds) One can depend on always getting the lowest available file descriptor.

Redirecting Output … Twice if (fork() == 0) { /* set up file descriptors 1 and 2 in the child process */ close(1); close(2); if (open("/home/twd/Output", O_WRONLY) == -1) { exit(1); } execl("/home/twd/bin/program", "program", 0); /* parent continues here */

Redirected Output File-descriptor table inode pointer 1 inode pointer WRONLY inode pointer File descriptor 1 File descriptor 2 1 WRONLY inode pointer User address space Kernel address space

Redirected Output After Write File-descriptor table 1 WRONLY 100 inode pointer File descriptor 1 File descriptor 2 1 WRONLY inode pointer User address space Kernel address space

Sharing Context Information if (fork() == 0) { /* set up file descriptors 1 and 2 in the child process */ close(1); close(2); if (open("/home/twd/Output", O_WRONLY) == -1) { exit(1); } dup(1); /* set up file descriptor 2 as a duplicate of 1 */ execl("/home/twd/bin/program", "program", 0); /* parent continues here */

Redirected Output After Dup File-descriptor table File descriptor 1 2 WRONLY 100 inode pointer File descriptor 2 User address space Kernel address space

Fork and File Descriptors int logfile = open("log", O_WRONLY); if (fork() == 0) { /* child process computes something, then does: */ write(logfile, LogEntry, strlen(LogEntry)); … exit(0); } /* parent process computes something, then does: */

File Descriptors After Fork logfile Parent’s address space 2 WRONLY inode pointer logfile Child’s address space Kernel address space

Naming (almost) everything has a path name files directories devices (known as special files) keyboards displays disks etc. The notion that almost everything in Unix has a path name was a startlingly new concept when Unix was first developed; one that has proven to be important.

Uniformity int file = open("/home/twd/data", O_RDWR); // opening a normal file int device = open("/dev/tty", O_RDWR); // opening a device (one’s terminal // or window) int bytes = read(file, buffer, sizeof(buffer)); write(device, buffer, bytes); This notion that everything has a path name facilitates a uniformity of interface. Reading and writing a normal file involves a different set of internal operations than reading and writing a device, but they are named in the same style and the I/O system calls treat them in the same way. What we have is a form of polymorphism (though the term didn’t really exist when the original Unix developers came up with this way of doing things). Note that the open system call returns an integer called a file descriptor, used in subsequent system calls to refer to the file.