1. File Systems

2. Main Points
File layout
Directory layout

3. File Systems (1)
Essential requirements for long-term information storage:
It must be possible to store a very large amount of information.
Information must survive termination of process using it.
Multiple processes must be able to access information concurrently.
Tanenbaum & Bo, Modern Operating Systems:4th ed., (c) 2013 Prentice-Hall, Inc. All rights reserved.

4. File Systems (2)
Think of a disk as a linear sequence of fixed-size blocks and supporting two operations:
Read block k.
Write block k
Tanenbaum & Bo, Modern Operating Systems:4th ed., (c) 2013 Prentice-Hall, Inc. All rights reserved.

5. File Operations Create Append Delete Seek Open Get attributes Close
Read
Write
Append
Seek
Get attributes
Set attributes
Rename
Tanenbaum & Bo, Modern Operating Systems:4th ed., (c) 2013 Prentice-Hall, Inc. All rights reserved.

6. The External View of the File Manager
Application
Program
CreateFile()
ReadFile()
CloseHandle()
SetFilePointer()
WriteFile()
mount()
write()
open()
close()
read()
lseek()
File Mgr
Device Mgr
Memory Mgr
Process Mgr
File Mgr
Device Mgr
Memory Mgr
Process Mgr
UNIX
Windows
Hardware
Operating Systems: A Modern Perspective, Chapter 13

7. Figure 4-3. (a) An executable file. (b) An archive
File Types
Figure 4-3. (a) An executable file. (b) An archive
Tanenbaum & Bo, Modern Operating Systems:4th ed., (c) 2013 Prentice-Hall, Inc. All rights reserved.

8. Figure 4-4. Some possible file attributes.
Tanenbaum & Bo, Modern Operating Systems:4th ed., (c) 2013 Prentice-Hall, Inc. All rights reserved.

9. File Management File is a named, ordered collection of information
The file manager administers the collection by:
Storing the information on a device
Mapping the block storage to a logical view
Allocating/deallocating storage
Providing file directories
Operating Systems: A Modern Perspective, Chapter 13

10. Low-level File System Architecture
Block 0
b0 b1 b2 b3 … …
bn-1 . . .
Sequential Device
Randomly Accessed Device
Operating Systems: A Modern Perspective, Chapter 13

11. File Structure
Figure 4-2. Three kinds of files. (a) Byte sequence. (b) Record sequence. (c) Tree.
Tanenbaum & Bo, Modern Operating Systems:4th ed., (c) 2013 Prentice-Hall, Inc. All rights reserved.

12. Information Structure
Applications
Records
Byte Stream Files
Stream-Block Translation
Storage device
Operating Systems: A Modern Perspective, Chapter 13

13. Byte Stream File Interface
fileID = open(fileName)
close(fileID)
read(fileID, buffer, length)
write(fileID, buffer, length)
seek(fileID, filePosition)
Operating Systems: A Modern Perspective, Chapter 13

14. Low Level Files b0 b1 b2 bi ... ... Stream-Block Translation
fid = open(“fileName”,…); …
read(fid, buf, buflen);
close(fid);
...
... b0 b1 b2 bi
int open(…) {…}
int close(…) {…}
int read(…) {…}
int write(…) {…}
int seek(…) {…}
Stream-Block Translation
Storage device response to commands
Operating Systems: A Modern Perspective, Chapter 13

15. Structured Record Files
Structured Files
Records
Structured Record Files
Record-Block Translation
Operating Systems: A Modern Perspective, Chapter 13

16. Record-Oriented Sequential Files
Logical Record
fileID = open(fileName)
close(fileID)
getRecord(fileID, record)
putRecord(fileID, record)
seek(fileID, position)
Operating Systems: A Modern Perspective, Chapter 13

17. Record-Oriented Sequential Files
Logical Record
H byte header
k byte logical record
...
Operating Systems: A Modern Perspective, Chapter 13

18. Record-Oriented Sequential Files
Logical Record
H byte header
k byte logical record
...
...
Physical Storage Blocks
Fragment
Operating Systems: A Modern Perspective, Chapter 13

19. Indexed Sequential File
Suppose we want to directly access records
Add an index to the file
fileID = open(fileName)
close(fileID)
getRecord(fileID, index)
index = putRecord(fileID, record)
deleteRecord(fileID, index)
Operating Systems: A Modern Perspective, Chapter 13

20. Indexed Sequential File (cont)
Application structure
Account #
012345
123456
294376
...
529366
965987
Index i k j
index = i
index = k
index = j
Operating Systems: A Modern Perspective, Chapter 13

21. File System Design
File System is an organized collection of regular files and directories (mkfs)
Data structures
Directories: file name -> file metadata
Store directories as files
File metadata: how to find file data blocks
Free map: list of free disk blocks

22. File System Design Constraints
For small files:
Small blocks for storage efficiency
Files used together should be stored together
For large files:
Contiguous allocation for sequential access
Efficient lookup for random access
May not know at file creation
Whether file will become small or large

23. Design Challenges Index structure Index granularity Free space
How do we locate the blocks of a file?
Index granularity
What block size do we use?
Free space
How do we find unused blocks on disk?
Locality
How do we preserve spatial locality?
Reliability
What if machine crashes in middle of a file system op?

24. Block Management
The job of selecting & assigning storage blocks to the file
Three basic strategies:
Contiguous allocation
Linked lists
Indexed allocation
Operating Systems: A Modern Perspective, Chapter 13

25. Contiguous Allocation
Maps the N blocks into N contiguous blocks on the secondary storage device
Difficult to support dynamic file sizes
File descriptor
Head position 237 …
First block 785
Number of blocks 25
Operating Systems: A Modern Perspective, Chapter 13

26. Implementing Files Contiguous Layout
Contiguous allocation of disk space for seven files.
(b) The state of the disk after files D and F have been removed.
Tanenbaum & Bo, Modern Operating Systems:4th ed., (c) 2013 Prentice-Hall, Inc. All rights reserved.

27. Linked Lists Each block contains a header with
Number of bytes in the block
Pointer to next block
Blocks need not be contiguous
Files can expand and contract
Seeks can be slow
First block …
Head: 417
...
Length
Length
Length
Byte 0
Byte 0
Byte 0
...
...
...
Byte 4095
Byte 4095
Byte 4095
Block 0
Block 1
Block N-1
Operating Systems: A Modern Perspective, Chapter 13

28. Linked List Allocation

29. Indexed Allocation Extract headers and put them in an index
Simplify seeks
May link indices together (for large files)
Byte 0
...
Index block …
Head: 417
...
Byte 4095
Length
Block 0
Length
Byte 0
...
Byte 4095
Block 1
Byte 0
...
Length
Byte 4095
Block N-1
Operating Systems: A Modern Perspective, Chapter 13

30. File Systems Traditional FFS file system (Linux)
Microsoft’s FAT, FAT2 and NTFS file systems
Journaling file systems, ext3
… others

31. File System Design Options
FAT
FFS
NTFS
Index structure
Linked list
Tree
(fixed, assym)
(dynamic)
granularity
block
extent
free space
allocation
FAT array
Bitmap
(fixed location)
(file)
Locality
defragmentation
Block groups + reserve space
Extents
Best fit
defrag

32. Microsoft File Allocation Table (FAT)
Linked list index structure
Simple, easy to implement
Still widely used (e.g., thumb drives)
File table:
Linear map of all blocks on disk
Each file a linked list of blocks

33. FAT

34. FAT Pros: Cons: Easy to find free block Easy to append to a file
Easy to delete a file
Cons:
Small file access is slow
Random access is very slow
Fragmentation
File blocks for a given file may be scattered
Files in the same directory may be scattered
Problem becomes worse as disk fills

35. Berkeley UNIX FFS (Fast File System)
inode table
Analogous to FAT table
inode
Metadata
File owner, access permissions, access times, …
Set of 12 data pointers
With 4KB blocks => max size of 48KB files

36. File System Structure
Basic unit for allocating space on the disk is a block
Disk
partition
partition
partition
File System
Boot Block
Super-block
i-node table
Data blocks

37. I-nodes
Each file or directory in the file system has a unique entry in the i-node table.
File type (regular, symbolic link, directory…)
Owner
Permissions
Timestamps for last access; last modification, last status change
Size …

39. i-node entry … 5 11 12 13 14 15
DB 0
DB 5 DB
IPB
IPB
IPB DB

40. FFS inode Metadata Set of 12 data pointers Indirect block pointer
File owner, access permissions, access times, …
Set of 12 data pointers
With 4KB blocks => max size of 48KB files
Indirect block pointer
pointer to disk block of data pointers
Indirect block: 1K data blocks => 4MB (+48KB)

41. FFS inode Metadata Set of 12 data pointers Indirect block pointer
File owner, access permissions, access times, …
Set of 12 data pointers
With 4KB blocks => max size of 48KB
Indirect block pointer
pointer to disk block of data pointers
4KB block size => 1K data blocks => 4MB
Doubly indirect block pointer
Doubly indirect block => 1K indirect blocks
4GB (+ 4MB + 48KB)

42. FFS inode Metadata Set of 12 data pointers Indirect block pointer
File owner, access permissions, access times, …
Set of 12 data pointers
With 4KB blocks => max size of 48KB
Indirect block pointer
pointer to disk block of data pointers
4KB block size => 1K data blocks => 4MB
Doubly indirect block pointer
Doubly indirect block => 1K indirect blocks
4GB (+ 4MB + 48KB)
Triply indirect block pointer
Triply indirect block => 1K doubly indirect blocks
4TB (+ 4GB + 4MB + 48KB)

43. Disk Organization Boot Sector Volume Directory Track 0, Cylinder 0
Blk0
Blk1 …
Blkk-1
Track 0, Cylinder 0 …
Blkk
Blkk+1
Blk2k-1
Track 0, Cylinder 1 … …
Blk
Blk
Blk
Track 1, Cylinder 0 … …
Blk
Blk
Blk
Track N-1, Cylinder 0 … …
Blk
Blk
Blk
Track N-1, Cylinder M-1
Operating Systems: A Modern Perspective, Chapter 13

44. FFS Locality Block group allocation inode table spread throughout disk
Block group is a set of nearby cylinders
Files in same directory located in same group
Subdirectories located in different block groups
inode table spread throughout disk
inodes, bitmap near file blocks
First fit allocation
Small files fragmented, large files contiguous

45. FFS First Fit Block Allocation

46. FFS First Fit Block Allocation

47. FFS First Fit Block Allocation

48. FFS Pros Cons Efficient storage for both small and large files
Locality for both small and large files
Locality for metadata and data
Cons
Inefficient for tiny files (a 1 byte file requires both an inode and a data block)
Inefficient encoding when file is mostly contiguous on disk (no equivalent to superpages)
Need to reserve 10-20% of free space to prevent fragmentation

49. NTFS Master File Table Extents Journalling for reliability
Flexible 1KB storage for metadata and data
Extents
Block pointers cover runs of blocks
Similar approach in linux (ext4)
File create can provide hint as to size of file
Journalling for reliability
Discussed next time

50. NTFS Small File

51. NTFS Medium File

52. NTFS Indirect Block

53. NTFS Multiple Indirect Blocks

54. 13
File Management
Operating Systems: A Modern Perspective, Chapter 13

55. An open() Operation Locate the on-device (external) file descriptor
Extract info needed to read/write file
Authenticate that process can access the file
Create an internal file descriptor in primary memory
Create an entry in a “per process” open file status table
Allocate resources, e.g., buffers, to support file usage
Operating Systems: A Modern Perspective, Chapter 13

56. File Manager Data Structures
Keep the state of the process-file session 2
Copy info from external to the open file descriptor 1
Open File
Descriptor
Process-File
Session 3
Return a reference to the data structure
External File Descriptor
Operating Systems: A Modern Perspective, Chapter 13

57. Opening a UNIX File On-Device File Descriptor Open File Table
fid = open(“fileA”, flags); …
read(fid, buffer, len);
On-Device File Descriptor
0 stdin
1 stdout
2 stderr
3 ...
File structure
inode
Open File Table
Internal File Descriptor
Operating Systems: A Modern Perspective, Chapter 13

58. File Descriptors External name Current state Sharable Owner User Locks
Protection settings
Length
Time of creation
Time of last modification
Time of last access
Reference count
Storage device details
Operating Systems: A Modern Perspective, Chapter 13

59. Marshalling the Byte Stream
Must read at least one buffer ahead on input
Must write at least one buffer behind on output
Seek  flushing the current buffer and finding the correct one to load into memory
Inserting/deleting bytes in the interior of the stream
Operating Systems: A Modern Perspective, Chapter 13

60. Full Block Buffering Storage devices use block I/O
Files place an explicit order on the bytes
Therefore, it is possible to predict what is likely to be read after bytei
When file is opened, manager reads as many blocks ahead as feasible
After a block is logically written, it is queued for writing behind, whenever the disk is available
Buffer pool – usually variably sized, depending on virtual memory needs
Interaction with the device manager and memory manager
Operating Systems: A Modern Perspective, Chapter 13

61. File-Descriptor Table1 2 3
File descriptor
ref count
access mode
file location
inode pointer . . .
User
address space
n–1
Kernel address space

62. Allocation of File Descriptors
Whenever a process requests a new file descriptor, the lowest numbered file descriptor not already associated with an open file is selected; thus
#include <fcntl.h>
#include <unistd.h>
close(0);
fd = open("file", O_RDONLY);
will always associate file with file descriptor 0 (assuming that the open succeeds)
One can depend on always getting the lowest available file descriptor.

63. Redirecting Output … Twice
if (fork() == 0) { /* set up file descriptors 1 and 2 in the child process */ close(1); close(2); if (open("/home/twd/Output", O_WRONLY) == -1) { exit(1); } execl("/home/twd/bin/program", "program", 0); /* parent continues here */

64. Redirected Output File-descriptor table inode pointer 1 inode pointer
WRONLY
inode pointer
File descriptor 1
File descriptor 2 1
WRONLY
inode pointer
User
address space
Kernel address space

65. Redirected Output After Write
File-descriptor
table 1
WRONLY
100
inode pointer
File descriptor 1
File descriptor 2 1
WRONLY
inode pointer
User
address space
Kernel address space

66. Sharing Context Information
if (fork() == 0) { /* set up file descriptors 1 and 2 in the child process */ close(1); close(2); if (open("/home/twd/Output", O_WRONLY) == -1) { exit(1); } dup(1); /* set up file descriptor 2 as a duplicate of 1 */ execl("/home/twd/bin/program", "program", 0); /* parent continues here */

67. Redirected Output After Dup
File-descriptor
table
File descriptor 1 2
WRONLY
100
inode pointer
File descriptor 2
User
address space
Kernel address space

68. Fork and File Descriptors
int logfile = open("log", O_WRONLY); if (fork() == 0) { /* child process computes something, then does: */ write(logfile, LogEntry, strlen(LogEntry)); … exit(0); } /* parent process computes something, then does: */

69. File Descriptors After Fork
logfile
Parent’s
address space 2
WRONLY
inode pointer
logfile
Child’s
address space
Kernel address space

70. Naming (almost) everything has a path name files directories
devices (known as special files)
keyboards
displays
disks
etc.
The notion that almost everything in Unix has a path name was a startlingly new concept when Unix was first developed; one that has proven to be important.

71. Uniformity int file = open("/home/twd/data", O_RDWR);
// opening a normal file
int device = open("/dev/tty", O_RDWR);
// opening a device (one’s terminal
// or window)
int bytes = read(file, buffer, sizeof(buffer));
write(device, buffer, bytes);
This notion that everything has a path name facilitates a uniformity of interface. Reading and writing a normal file involves a different set of internal operations than reading and writing a device, but they are named in the same style and the I/O system calls treat them in the same way. What we have is a form of polymorphism (though the term didn’t really exist when the original Unix developers came up with this way of doing things).
Note that the open system call returns an integer called a file descriptor, used in subsequent system calls to refer to the file.