Slide 13-1 Copyright © 2004 Pearson Education, Inc. Operating Systems: A Modern Perspective, Chapter 13

Slide 13-2 Copyright © 2004 Pearson Education, Inc. Operating Systems: A Modern Perspective, Chapter File Management

Slide 13-3 Copyright © 2004 Pearson Education, Inc. Operating Systems: A Modern Perspective, Chapter 13 Fig 13-2: The External View of the File Manager Hardware Application Program Application Program File MgrDevice MgrMemory Mgr Process Mgr UNIX File MgrDevice MgrMemory Mgr Process Mgr Windows open() read() close() write() lseek() CreateFile() ReadFile() CloseHandle() SetFilePointer() WriteFile() mount()

Slide 13-4 Copyright © 2004 Pearson Education, Inc. Operating Systems: A Modern Perspective, Chapter 13 Persistent storage Shared device Why Programmers Need Files HTML Editor HTML Editor … … Web Browser Web Browser Structured information Can be read by any applic Accessibility Protocol … … … … foo.html File Manager File Manager File Manager File Manager

Slide 13-5 Copyright © 2004 Pearson Education, Inc. Operating Systems: A Modern Perspective, Chapter 13 File Management File is a named, ordered collection of information The file manager administers the collection by: –Storing the information on a device –Mapping the block storage to a logical view –Allocating/deallocating storage –Providing file directories What abstraction should be presented to programmer?

Slide 13-6 Copyright © 2004 Pearson Education, Inc. Operating Systems: A Modern Perspective, Chapter 13 Information Structure Records Applications Structured Record Files Record-Stream Translation Stream-Block Translation Byte Stream Files Storage device

Slide 13-7 Copyright © 2004 Pearson Education, Inc. Operating Systems: A Modern Perspective, Chapter 13 Byte Stream File Interface fileID = open(fileName) close(fileID) read(fileID, buffer, length) write(fileID, buffer, length) seek(fileID, filePosition)

Slide 13-8 Copyright © 2004 Pearson Education, Inc. Operating Systems: A Modern Perspective, Chapter 13 Low Level Files Stream-Block Translation b0b0 b1b1 b2b2 bibi... fid = open(“fileName”,…); … read(fid, buf, buflen); … close(fid); int open(…) {…} int close(…) {…} int read(…) {…} int write(…) {…} int seek(…) {…} Storage device response to commands

Slide 13-9 Copyright © 2004 Pearson Education, Inc. Operating Systems: A Modern Perspective, Chapter 13 Structured Files Records Record-Block Translation

Slide Copyright © 2004 Pearson Education, Inc. Operating Systems: A Modern Perspective, Chapter 13 Record-Oriented Sequential Files Logical Record fileID = open(fileName) close(fileID) getRecord(fileID, record) putRecord(fileID, record) seek(fileID, position)

Slide Copyright © 2004 Pearson Education, Inc. Operating Systems: A Modern Perspective, Chapter 13 Record-Oriented Sequential Files... H byte headerk byte logical record Logical Record

Slide Copyright © 2004 Pearson Education, Inc. Operating Systems: A Modern Perspective, Chapter 13 Record-Oriented Sequential Files... H byte headerk byte logical record... Fragment Physical Storage Blocks Logical Record

Slide Copyright © 2004 Pearson Education, Inc. Operating Systems: A Modern Perspective, Chapter 13 Electronic Mail Example struct message { /* The mail message */ address to; address from; line subject; address cc; string body; }; struct message *getRecord(void) { struct message *msg; msg = allocate(sizeof(message)); msg->to = getAddress(...); msg->from = getAddress(...); msg->cc = getAddress(...); msg->subject = getLine(); msg->body = getString(); return(msg); } putRecord(struct message *msg) { putAddress(msg->to); putAddress(msg->from); putAddress(msg->cc); putLine(msg->subject); putString(msg->body); }

Slide Copyright © 2004 Pearson Education, Inc. Operating Systems: A Modern Perspective, Chapter 13 Indexed Sequential File Suppose we want to directly access records Add an index to the file fileID = open(fileName) close(fileID) getRecord(fileID, index) index = putRecord(fileID, record) deleteRecord(fileID, index)

Slide Copyright © 2004 Pearson Education, Inc. Operating Systems: A Modern Perspective, Chapter 13 Indexed Sequential File (cont) Account # Index i k j index = i index = k index = j Application structure

Slide Copyright © 2004 Pearson Education, Inc. Operating Systems: A Modern Perspective, Chapter 13 More Abstract Files Inverted files –System index for each datum in the file Databases –More elaborate indexing mechanism –DDL & DML Multimedia storage –Records contain radically different types –Access methods must be general

Slide Copyright © 2004 Pearson Education, Inc. Operating Systems: A Modern Perspective, Chapter 13 Implementing Low Level Files Secondary storage device contains: –Volume directory (sometimes a root directory for a file system) –External file descriptor for each file –The file contents Manages blocks –Assigns blocks to files (descriptor keeps track) –Keeps track of available blocks Maps to/from byte stream

Slide Copyright © 2004 Pearson Education, Inc. Operating Systems: A Modern Perspective, Chapter 13 Disk Organization Blk 0 Blk 1 Blk k-1 Blk k Blk k+1 Blk 2k-1 Track 0, Cylinder 0 Track 0, Cylinder 1 Blk Track 1, Cylinder 0 Blk Track N-1, Cylinder 0 Blk Track N-1, Cylinder M-1 … … … … … … … … Boot SectorVolume Directory

Slide Copyright © 2004 Pearson Education, Inc. Operating Systems: A Modern Perspective, Chapter 13 Low-level File System Architecture b 0 b 1 b 2 b 3 b n-1 …… Block 0... Sequential Device Randomly Accessed Device

Slide Copyright © 2004 Pearson Education, Inc. Operating Systems: A Modern Perspective, Chapter 13 File Descriptors External name Current state Sharable Owner User Locks Protection settings Length Time of creation Time of last modification Time of last access Reference count Storage device details

Slide Copyright © 2004 Pearson Education, Inc. Operating Systems: A Modern Perspective, Chapter 13 An open() Operation Locate the on-device (external) file descriptor Extract info needed to read/write file Authenticate that process can access the file Create an internal file descriptor in primary memory Create an entry in a “per process” open file status table Allocate resources, e.g., buffers, to support file usage

Slide Copyright © 2004 Pearson Education, Inc. Operating Systems: A Modern Perspective, Chapter 13 File Manager Data Structures External File Descriptor Open File Descriptor Copy info from external to the open file descriptor 1 Process-File Session Keep the state of the process- file session 2 Return a reference to the data structure 3

Slide Copyright © 2004 Pearson Education, Inc. Operating Systems: A Modern Perspective, Chapter 13 Opening a UNIX File fid = open(“fileA”, flags); … read(fid, buffer, len); 0 stdin 1 stdout 2 stderr 3... Open File Table File structure inode Internal File Descriptor On-Device File Descriptor

Slide Copyright © 2004 Pearson Education, Inc. Operating Systems: A Modern Perspective, Chapter 13 Block Management The job of selecting & assigning storage blocks to the file For a fixed sized file of k blocks –File of length m requires N =  m/k  blocks –Byte b i is stored in block  i/k  Three basic strategies: –Contiguous allocation –Linked lists –Indexed allocation

Slide Copyright © 2004 Pearson Education, Inc. Operating Systems: A Modern Perspective, Chapter 13 Contiguous Allocation Maps the N blocks into N contiguous blocks on the secondary storage device Difficult to support dynamic file sizes Head position237 … First block785 Number of blocks25 File descriptor

Slide Copyright © 2004 Pearson Education, Inc. Operating Systems: A Modern Perspective, Chapter 13 Linked Lists Each block contains a header with –Number of bytes in the block –Pointer to next block Blocks need not be contiguous Files can expand and contract Seeks can be slow First block … Head: Length Byte 0 Byte Length Byte 0 Byte Length Byte 0 Byte Block 0Block 1Block N-1

Slide Copyright © 2004 Pearson Education, Inc. Operating Systems: A Modern Perspective, Chapter 13 Indexed Allocation Extract headers and put them in an index Simplify seeks May link indices together (for large files) Index block … Head: Byte 0 Byte Byte 0 Byte Byte 0 Byte Block 0 Block 1 Block N-1 Length

Slide Copyright © 2004 Pearson Education, Inc. Operating Systems: A Modern Perspective, Chapter 13 DOS FAT Files Disk Block File Descriptor Disk Block Disk Block … File Access Table (FAT) Disk Block Disk Block Disk Block … File Descriptor

Slide Copyright © 2004 Pearson Education, Inc. Operating Systems: A Modern Perspective, Chapter 13 UNIX Files Data mode owner … Direct block 0 Direct block 1 … Direct block 11 Single indirect Double indirect Triple indirect inode Data Index Data Index Data Index Data

Slide Copyright © 2004 Pearson Education, Inc. Operating Systems: A Modern Perspective, Chapter 13 Unallocated Blocks How should unallocated blocks be managed? Need a data structure to keep track of them –Linked list Very large Hard to manage spatial locality –Block status map (“disk map”) Bit per block Easy to identify nearby free blocks Useful for disk recovery

Slide Copyright © 2004 Pearson Education, Inc. Operating Systems: A Modern Perspective, Chapter 13 Marshalling the Byte Stream Must read at least one buffer ahead on input Must write at least one buffer behind on output Seek  flushing the current buffer and finding the correct one to load into memory Inserting/deleting bytes in the interior of the stream

Slide Copyright © 2004 Pearson Education, Inc. Operating Systems: A Modern Perspective, Chapter 13 Full Block Buffering Storage devices use block I/O Files place an explicit order on the bytes Therefore, it is possible to predict what is likely to be read after byte i When file is opened, manager reads as many blocks ahead as feasible After a block is logically written, it is queued for writing behind, whenever the disk is available Buffer pool – usually variably sized, depending on virtual memory needs –Interaction with the device manager and memory manager

Slide Copyright © 2004 Pearson Education, Inc. Operating Systems: A Modern Perspective, Chapter 13 Directories A set of logically associated files and sub directories File manager provides set of controls: –enumerate –copy –rename –delete –traverse –etc.

Slide Copyright © 2004 Pearson Education, Inc. Operating Systems: A Modern Perspective, Chapter 13 Directory Structures How should files be organized within directory? –Flat name space All files appear in a single directory –Hierarchical name space Directory contains files and subdirectories Each file/directory appears as an entry in exactly one other directory -- a tree Popular variant: All directories form a tree, but a file can have multiple parents.

Slide Copyright © 2004 Pearson Education, Inc. Operating Systems: A Modern Perspective, Chapter 13 Directory Implementation Device Directory –A device can contain a collection of files –Easier to manage if there is a root for every file on the device -- the device root directory File Directory –Typical implementations have directories implemented as a file with a special format –Entries in a file directory are handles for other files (which can be files or subdirectories)

Slide Copyright © 2004 Pearson Education, Inc. Operating Systems: A Modern Perspective, Chapter 13 UNIX mount Command / binusretcfoo billnutt abc / blah cdexyz FS / binusretcfoo billnutt abc / blah cdexyz mount FS at foo FS

Slide Copyright © 2004 Pearson Education, Inc. Operating Systems: A Modern Perspective, Chapter 13 VFS-based File Manager File System Independent Part of File Manager File System Independent Part of File Manager Exports OS-specific API Virtual File System Switch MS-DOS Part of File Manager MS-DOS Part of File Manager ISO 9660 Part of File Manager ISO 9660 Part of File Manager ext2 Part of File Manager ext2 Part of File Manager …

Rensselaer Polytechnic Institute CSCI-4210 – Operating Systems David Goldschmidt, Ph.D.

very small very large very fast very slow volatile non-volatile managed as part of the filesystem

 A File Management System is a set of OS services that supports files and directories for user applications, OS programs, etc.  Data should be organized and available in a convenient and efficient manner  Files are the basic building blocks

 A file is an abstraction that represents user data, OS data, an executable, a device, etc.  A file is simply a sequence of bytes  Actual storage location (via network?) and format are transparent to users  Storage scheme on disk is also transparent ▪ Typically involves cylinders, tracks, sectors, etc.

 File attributes include:  Human-readable symbolic name  Type (e.g. executable, directory, text, PDF, etc.)  Logical location (i.e. containing directory or path)  Physical location on disk  Size (in bytes)  Protection or security (i.e. permissions)  Timestamps (created, last modified, last accessed)

 The pathname (or just path) of a file specifies the sequence of directories one must traverse to locate the file  An absolute path starts at the root node  A relative path starts anywhere

 A link provides a shortcut to a file and may circumvent the given directory hierarchy  A hard link in Unix is indistinguishable from the original file  A symbolic link in Unix is merely a shortcut  A Windows shortcut is just a symbolic link

 File creation requires space allocation  Opening a file returns a handle or file descriptor  Read and write operations use the handle and an offset (or file pointer)  The close operation simply deletes the handle and deallocates any memory

 Deleting a file deallocates all disk space marked as in use by the file  But likely does not erase file contents  Deleted files are recoverable until the disk space is used for (and overwritten by) other file(s)  The delete operation also removes the corresponding entry in the containing directory

 An access method describes the manner and mechanisms by which a process accesses the data in a file  Two common access methods:  Sequential access (open, read, write, close)  Random access (open, read, write, seek, close)

 In a contiguous disk space allocation scheme, files are allocated to contiguous blocks of disk space

 Four files allocated contiguously to disk:  File B outgrows its space and is reallocated:  File D outgrows its space and is reallocated:  Defragmentation combines free disk space: AAABBCCCDAAACCCDBBBBAAACCCBBBBDDAAACCCBBBBDD

 In a clustered disk space allocation scheme, files are allocated to clusters of disk space on an as needed basis

 Four files allocated contiguously to disk:  File B outgrows its space, so one or more new clusters are allocated:  File D outgrows its space, so one or more new clusters are allocated: AAABBCCCDAAABBCCCDBBAAABBCCCDBBD

 File accesses may cross cluster boundaries, causing slower response times

 Using a linked approach, each cluster has a pointer to the next cluster  Using an indexed approach, a single table maintains pointers to each individual cluster

 Managing free blocks (or free clusters) may follow the same scheme as for a single file  Linked scheme  Indexed approach  Bitmapped approach:  A bitmap is maintained in memory  Each bit corresponds to a block or cluster  A 0 indicates a free block  A 1 indicates the block is in use

 A file allocation table (FAT) maps logical files to their physical addresses on a disk  A pure FAT would have an entry for each sector (e.g. 512 bytes)  To improve performance, clusters of contiguous sectors are used  Clusters range from 4 to 64 sectors (e.g. 2,048 to 32,768 bytes)

 Microsoft FAT-32 filesystem maps to clusters  May require linked clusters to house entire FAT

 Linux uses indexed clusters

 In Unix, file information is stored in an index node (I-node), which contains:  Mode and type of file  Number of links to the file  File owner’s userid and groupid  Size of file (in bytes)  Last access and last modified times  Number of blocks allocated to the file  Pointers to the first twelve blocks  Pointers to three additional blocks of pointers

inode cluster … inode 1024 single indirect … inode 1024 double indirect inode 1024 inode 1024 … … … inode triple indirect …

 A virtual filesystem provides transparent access to different filesystem types on multiple device types and disk partitions

 A virtual machine is a layered approach that logically combines the kernel operating system and hardware  Creates the illusion of multiple processes, each executing on its own virtual processor with its own virtual memory

Non-virtual machine Virtual machine

 Java programs execute on a native Java Virtual Machine (JVM)

 Virtual machines provide complete protection of system resources  Each virtual machine is isolated from all other virtual machines ▪ which prohibits direct sharing of system resources  Virtual machines can be difficult to implement due to the effort required to provide an exact duplicate of each underlying machine