Lecture 12: File Management

10.2 Silberschatz, Galvin and Gagne ©2005 Operating System Concepts – 7 th Edition, Jan 1, 2005 File Management provides the file abstraction for data storage guarantees data validity (most of the time) performance: throughput and response time enforces protection

10.3 Silberschatz, Galvin and Gagne ©2005 Operating System Concepts – 7 th Edition, Jan 1, 2005 File-System Interface File Concept Access Methods Directory Structure File-System Mounting File Sharing Protection

10.4 Silberschatz, Galvin and Gagne ©2005 Operating System Concepts – 7 th Edition, Jan 1, 2005 File Concept Contiguous logical address space Persistent - stored on non-volatile memory (storage) Identified by (external) name pathname in UNIX

10.5 Silberschatz, Galvin and Gagne ©2005 Operating System Concepts – 7 th Edition, Jan 1, 2005 File Structure None - sequence of words, bytes Simple record structure Lines Fixed length Variable length Complex Structures Formatted document Relocatable load file Can simulate last two with first method by inserting appropriate control characters Who decides: Operating system Program

10.6 Silberschatz, Galvin and Gagne ©2005 Operating System Concepts – 7 th Edition, Jan 1, 2005 File Attributes Name – only information kept in human-readable form Identifier – unique tag (number) identifies file within file system Type – needed for systems that support different types Location – pointer to file location on device Size – current file size Protection – controls who can do reading, writing, executing Time, date, and user identification – data for protection, security, and usage monitoring Information about files are kept in the directory structure, which is maintained on the disk

10.7 Silberschatz, Galvin and Gagne ©2005 Operating System Concepts – 7 th Edition, Jan 1, 2005 File Operations Create Write Read Reposition within file Delete Truncate Open(F i ) –open read/write session for F i File pointer: pointer to last read/write location, Close (F i ) – close read/write session

10.8 Silberschatz, Galvin and Gagne ©2005 Operating System Concepts – 7 th Edition, Jan 1, 2005 Access Methods Sequential Access read next write next reset no read after last write (rewrite) Direct Access read n write n position to n read next write next rewrite n n = relative block number

10.9 Silberschatz, Galvin and Gagne ©2005 Operating System Concepts – 7 th Edition, Jan 1, 2005 Directories Set of files Used to organize files Translates external file names to internal file names (file labels, i- nodes, file control blocks) Treated as a file but with special operations

10.10 Silberschatz, Galvin and Gagne ©2005 Operating System Concepts – 7 th Edition, Jan 1, 2005 Operations Performed on Directory Search for a file Create a file Delete a file List a directory Rename a file Traverse the file system

10.11 Silberschatz, Galvin and Gagne ©2005 Operating System Concepts – 7 th Edition, Jan 1, 2005 Organize the Directory (Logically) to Obtain Efficiency – locating a file quickly Naming – convenient to users Two users can have same name for different files The same file can have several different names Grouping – logical grouping of files by properties, (e.g., all Java programs, all games, …)

10.12 Silberschatz, Galvin and Gagne ©2005 Operating System Concepts – 7 th Edition, Jan 1, 2005 Single-Level Directory A single directory for all users Naming problem Grouping problem

10.13 Silberschatz, Galvin and Gagne ©2005 Operating System Concepts – 7 th Edition, Jan 1, 2005 Two-Level Directory Separate directory for each user Path name Can have the same file name for different user Efficient searching No grouping capability

10.14 Silberschatz, Galvin and Gagne ©2005 Operating System Concepts – 7 th Edition, Jan 1, 2005 Tree-Structured Directories

10.15 Silberschatz, Galvin and Gagne ©2005 Operating System Concepts – 7 th Edition, Jan 1, 2005 Tree-Structured Directories (Cont) Efficient searching Grouping Capability Current directory (working directory) cd /spell/mail/prog type list

10.16 Silberschatz, Galvin and Gagne ©2005 Operating System Concepts – 7 th Edition, Jan 1, 2005 Tree-Structured Directories (Cont) Absolute or relative path name Creating a new file is done in current directory Delete a file rm Creating a new subdirectory is done in current directory mkdir Example: if in current directory /mail mkdir count mail progcopyprtexpcount Deleting “mail”  deleting the entire subtree rooted by “mail”

10.17 Silberschatz, Galvin and Gagne ©2005 Operating System Concepts – 7 th Edition, Jan 1, 2005 Acyclic-Graph Directories Have shared subdirectories and files

10.18 Silberschatz, Galvin and Gagne ©2005 Operating System Concepts – 7 th Edition, Jan 1, 2005 Acyclic-Graph Directories (Cont.) Two different names (aliasing) If dict deletes list  dangling pointer Solutions: Backpointers, so we can delete all pointers Variable size records a problem Backpointers using a daisy chain organization Entry-hold-count solution New directory entry type Link – another name (pointer) to an existing file Resolve the link – follow pointer to locate the file

10.19 Silberschatz, Galvin and Gagne ©2005 Operating System Concepts – 7 th Edition, Jan 1, 2005 File System Mounting A file system must be mounted before it can be accessed A file system is mounted at a mount point (a directory)

10.20 Silberschatz, Galvin and Gagne ©2005 Operating System Concepts – 7 th Edition, Jan 1, 2005 (a) Existing. (b) Unmounted Partition

10.21 Silberschatz, Galvin and Gagne ©2005 Operating System Concepts – 7 th Edition, Jan 1, 2005 Mount Point

10.22 Silberschatz, Galvin and Gagne ©2005 Operating System Concepts – 7 th Edition, Jan 1, 2005 File Sharing Sharing of files on multi-user systems is desirable Sharing may be done through a protection scheme On distributed systems, files may be shared across a network Network File System (NFS) is a common distributed file-sharing method

10.23 Silberschatz, Galvin and Gagne ©2005 Operating System Concepts – 7 th Edition, Jan 1, 2005 File Sharing – Multiple Users User IDs identify users, allowing permissions and protections to be per-user Group IDs allow users to be in groups, permitting group access rights

10.24 Silberschatz, Galvin and Gagne ©2005 Operating System Concepts – 7 th Edition, Jan 1, 2005 File Sharing – Remote File Systems Uses networking to allow file system access between systems Manually via programs like FTP Automatically, seamlessly using distributed file systems Semi automatically via the world wide web Client-server model allows clients to mount remote file systems from servers Server can serve multiple clients Client and user-on-client identification is insecure or complicated NFS is standard UNIX client-server file sharing protocol CIFS is standard Windows protocol Standard operating system file calls are translated into remote calls

10.25 Silberschatz, Galvin and Gagne ©2005 Operating System Concepts – 7 th Edition, Jan 1, 2005 File Sharing – Failure Modes Remote file systems add new failure modes, due to network failure, server failure Recovery from failure can involve state information about status of each remote request Stateless protocols such as NFS include all information in each request, allowing easy recovery but less security

10.26 Silberschatz, Galvin and Gagne ©2005 Operating System Concepts – 7 th Edition, Jan 1, 2005 File Sharing – Consistency Semantics Consistency semantics specify how multiple users are to access a shared file simultaneously Similar to Ch 7 process synchronization algorithms  Tend to be less complex due to disk I/O and network latency (for remote file systems Andrew File System (AFS) implemented complex remote file sharing semantics Unix file system (UFS) implements:  Writes to an open file visible immediately to other users of the same open file  Sharing file pointer to allow multiple users to read and write concurrently AFS has session semantics  Writes only visible to sessions starting after the file is closed

10.27 Silberschatz, Galvin and Gagne ©2005 Operating System Concepts – 7 th Edition, Jan 1, 2005 Protection File owner/creator should be able to control: what can be done by whom Types of access Read Write Execute Append Delete List

10.28 Silberschatz, Galvin and Gagne ©2005 Operating System Concepts – 7 th Edition, Jan 1, 2005 Access Lists and Groups Mode of access: read, write, execute Three classes of users RWX a) owner access 7  RWX b) group access 6  RWX c) public access1  Ask manager to create a group (unique name), say G, and add some users to the group. For a particular file (say game) or subdirectory, define an appropriate access. ownergrouppublic chmod761game Attach a group to a file chgrp G game

10.29 Silberschatz, Galvin and Gagne ©2005 Operating System Concepts – 7 th Edition, Jan 1, 2005 Protection Mechanisms files are OS objects: unique names and a finite set of operations that processes can perform on them protection domain is a set of {object,rights}, where right is the permission to perform one of the operations at every instant in time, each process runs in some protection domain in Unix, a protection domain is {uid, gid} protection domain in Unix is switched when running a program with SETUID/SETGID set or when the process enters the kernel mode by issuing a system call how to store all the protection domains ?

10.30 Silberschatz, Galvin and Gagne ©2005 Operating System Concepts – 7 th Edition, Jan 1, 2005 Protection Mechanisms (cont’d) Access Control List (ACL): associate with each object a list of all the protection domains that may access the object and how in Unix ACL is reduced to three protection domains: owner, group and others Capability List (C-list): associate with each process a list of objects that may be accessed along with the operations C-list implementation issues: where/how to store them (hardware, kernel, encrypted in user space) and how to revoke them

10.31 Silberschatz, Galvin and Gagne ©2005 Operating System Concepts – 7 th Edition, Jan 1, 2005 A Sample UNIX Directory Listing

10.32 Silberschatz, Galvin and Gagne ©2005 Operating System Concepts – 7 th Edition, Jan 1, 2005 Secondary Storage Management Space must be allocated to files Must keep track of the space available for allocation

10.33 Silberschatz, Galvin and Gagne ©2005 Operating System Concepts – 7 th Edition, Jan 1, 2005 Preallocation Need the maximum size for the file at the time of creation Difficult to reliably estimate the maximum potential size of the file Tend to overestimated file size to avoid running out of space

10.34 Silberschatz, Galvin and Gagne ©2005 Operating System Concepts – 7 th Edition, Jan 1, 2005 Methods of File Allocation (1) Contiguous allocation Single set of blocks is allocated to a file at the time of creation A single entry in the file allocation table  Starting block and length of the file External fragmentation will occur

10.35 Silberschatz, Galvin and Gagne ©2005 Operating System Concepts – 7 th Edition, Jan 1, 2005

10.36 Silberschatz, Galvin and Gagne ©2005 Operating System Concepts – 7 th Edition, Jan 1, 2005

10.37 Silberschatz, Galvin and Gagne ©2005 Operating System Concepts – 7 th Edition, Jan 1, 2005 Methods of File Allocation (2) Chained allocation Allocation on basis of individual block Each block contains a pointer to the next block in the chain A single entry in the file allocation table  Starting block and length of file No external fragmentation Best for sequential files No accommodation for the principle of locality

10.38 Silberschatz, Galvin and Gagne ©2005 Operating System Concepts – 7 th Edition, Jan 1, 2005

10.39 Silberschatz, Galvin and Gagne ©2005 Operating System Concepts – 7 th Edition, Jan 1, 2005

10.40 Silberschatz, Galvin and Gagne ©2005 Operating System Concepts – 7 th Edition, Jan 1, 2005 Methods of File Allocation (3) Indexed allocation File allocation table contains a separate one-level index for each file The index has one entry for each portion allocated to the file The file allocation table contains block number for the index

10.41 Silberschatz, Galvin and Gagne ©2005 Operating System Concepts – 7 th Edition, Jan 1, 2005

10.42 Silberschatz, Galvin and Gagne ©2005 Operating System Concepts – 7 th Edition, Jan 1, 2005

10.43 Silberschatz, Galvin and Gagne ©2005 Operating System Concepts – 7 th Edition, Jan 1, 2005 File Allocation contiguous: a contiguous set of blocks is allocated to a file at the time of file creation consolidation to improve locality chained allocation: each block contains a pointer to the next one in the chain good for sequential files file size must be known at the time of file creation external fragmentation indexed allocation: good both for sequential and direct access (UNIX)

10.44 Silberschatz, Galvin and Gagne ©2005 Operating System Concepts – 7 th Edition, Jan 1, 2005 Free Space Management bitmap: one bit for each block on the disk chained free portions: {pointer to the next one, length} index: treats free space as a file good to find a contiguous group of free blocks small enough to be kept in memory

10.45 Silberschatz, Galvin and Gagne ©2005 Operating System Concepts – 7 th Edition, Jan 1, 2005 UNIX File System Naming External/Internal names translation using directories Lookup File blocks  Disk blocks Protection Free Space Management

10.46 Silberschatz, Galvin and Gagne ©2005 Operating System Concepts – 7 th Edition, Jan 1, 2005 File Naming External names (used by the application) Pathname: /usr/users/file1 Internal names (used by the OS kernel) I-node: file number/index on disk superblock File system on disk I-node area ( one I-node per file) File-block area 0 1

10.47 Silberschatz, Galvin and Gagne ©2005 Operating System Concepts – 7 th Edition, Jan 1, 2005 Directories Files that store translation tables (external names to internal names) usr 23 Root directory (always I-node 2) users 41 usr file1 87 usr users /usr/users/file1 corresponds to I-node 87

10.48 Silberschatz, Galvin and Gagne ©2005 Operating System Concepts – 7 th Edition, Jan 1, 2005 File Content Lookup address table used to translate logical file blocks into disk blocks address table stored in the I-node File with i-node 87 File System disk address table

10.49 Silberschatz, Galvin and Gagne ©2005 Operating System Concepts – 7 th Edition, Jan 1, 2005

10.50 Silberschatz, Galvin and Gagne ©2005 Operating System Concepts – 7 th Edition, Jan 1, 2005 File Protection ACL with three protection domains (file owner, file owner group, others) Access rights: read/write/execute Stored in the i-node

10.51 Silberschatz, Galvin and Gagne ©2005 Operating System Concepts – 7 th Edition, Jan 1, 2005 Free Space Management Free I-nodes Marked as free on disk An array of 50 free i-nodes stored in the superblock Free file blocks Stored as a list of 50- free block arrays First array stored in the superblock

10.52 Silberschatz, Galvin and Gagne ©2005 Operating System Concepts – 7 th Edition, Jan 1, 2005 In-Kernel File System Data Structures 0 1 File system on disk OS Kernel Buffer cache I-node cache Per-OS Open File Table (offset in file, ptr to I-node) Per-process Open File Table PCBs Application fd=open(pathname,mode); /* fd = index in Per-Proc OFT */ for (..) read(fd,buf,size); close(fd);

10.53 Silberschatz, Galvin and Gagne ©2005 Operating System Concepts – 7 th Edition, Jan 1, 2005 File System Metadata Consistency Problem file system uses the buffer cache for performance reasons two copies of a disk block (buffer cache, disk) -> consistency problem if the system crashes before all the modified blocks are written back to disk the problem is critical especially for the blocks that contain control information (meta-data): directory blocks, i-node, free-list Solution: write through meta-data blocks (expensive) or order of write-backs is important ordinary file data blocks written back periodically (sync) utility programs for checking block and directory consistency after crash

10.54 Silberschatz, Galvin and Gagne ©2005 Operating System Concepts – 7 th Edition, Jan 1, 2005 File System Metadata Consistency Problem: Examples Example 1: create a new file Two updates: (1) allocate a free I-node; (2) create an entry in the directory (1) and (2) must be write-through (expensive) or (1) must be written- back before (2) If (2) is written back first and a crash occurs before (1) is written back the directory structure is inconsistent and cannot be recovered Example 2: write a new block to a file Two updates: (1) allocate a free block; (2) update the address table of the I-node (1) and (2) must be write-through or (1) must be written-back before (2) If (2) is written back first and a crash occurs before (1) is written back the I-node structure is inconsistent and cannot be recovered