1. File Systems

2. Main Points File layout Directory layout

3. Operating Systems: A Modern Perspective, Chapter 13 The External View of the File Manager Hardware Application Program Application Program File MgrDevice MgrMemory Mgr Process Mgr UNIX File MgrDevice MgrMemory Mgr Process Mgr Windows open() read() close() write() lseek() CreateFile() ReadFile() CloseHandle() SetFilePointer() WriteFile() mount()

4. Operating Systems: A Modern Perspective, Chapter 13 Persistent storage Shared device Why Programmers Need Files HTML Editor HTML Editor … … Web Browser Web Browser Structured information Can be read by any applic Accessibility Protocol … … … … foo.html File Manager File Manager File Manager File Manager

5. Operating Systems: A Modern Perspective, Chapter 13 File Management File is a named, ordered collection of information The file manager administers the collection by: – Storing the information on a device – Mapping the block storage to a logical view – Allocating/deallocating storage – Providing file directories

6. Operating Systems: A Modern Perspective, Chapter 13 Information Structure Records Applications Structured Record Files Record-Stream Translation Stream-Block Translation Byte Stream Files Storage device

7. Operating Systems: A Modern Perspective, Chapter 13 Information Structure Records Applications Stream-Block Translation Byte Stream Files Storage device

8. Operating Systems: A Modern Perspective, Chapter 13 Byte Stream File Interface fileID = open(fileName) close(fileID) read(fileID, buffer, length) write(fileID, buffer, length) seek(fileID, filePosition)

9. Operating Systems: A Modern Perspective, Chapter 13 Low Level Files Stream-Block Translation b0b0 b1b1 b2b2 bibi... fid = open(“fileName”,…); … read(fid, buf, buflen); … close(fid); int open(…) {…} int close(…) {…} int read(…) {…} int write(…) {…} int seek(…) {…} Storage device response to commands

10. Operating Systems: A Modern Perspective, Chapter 13 Structured Files Records Record-Block Translation Structured Record Files

11. Operating Systems: A Modern Perspective, Chapter 13 Record-Oriented Sequential Files Logical Record fileID = open(fileName) close(fileID) getRecord(fileID, record) putRecord(fileID, record) seek(fileID, position)

12. Operating Systems: A Modern Perspective, Chapter 13 Record-Oriented Sequential Files... H byte headerk byte logical record Logical Record

13. Operating Systems: A Modern Perspective, Chapter 13 Record-Oriented Sequential Files... H byte headerk byte logical record... Fragment Physical Storage Blocks Logical Record

14. Operating Systems: A Modern Perspective, Chapter 13 Indexed Sequential File Suppose we want to directly access records Add an index to the file fileID = open(fileName) close(fileID) getRecord(fileID, index) index = putRecord(fileID, record) deleteRecord(fileID, index)

15. Operating Systems: A Modern Perspective, Chapter 13 Indexed Sequential File (cont) Account # 012345 123456 294376... 529366... 965987 Index i k j index = i index = k index = j Application structure

16. File System Design File System is an organized collection of regular files and directories (mkfs) Data structures – Directories: file name -> file metadata Store directories as files – File metadata: how to find file data blocks – Free map: list of free disk blocks

17. File System Design Constraints For small files: – Small blocks for storage efficiency – Files used together should be stored together For large files: – Contiguous allocation for sequential access – Efficient lookup for random access May not know at file creation – Whether file will become small or large

18. Design Challenges Index structure – How do we locate the blocks of a file? Index granularity – What block size do we use? Free space – How do we find unused blocks on disk? Locality – How do we preserve spatial locality? Reliability – What if machine crashes in middle of a file system op?

19. File Systems Traditional FFS file system (Linux) Microsoft’s FAT, FAT2 and NTFS file systems Journaling file systems, ext3 … others

20. File System Design Options FATFFSNTFS Index structure Linked listTree (fixed, assym) Tree (dynamic) granularityblock extent free space allocation FAT arrayBitmap (fixed location) Bitmap (file) LocalitydefragmentationBlock groups + reserve space Extents Best fit defrag

21. Named Data in a File System

22. Microsoft File Allocation Table (FAT) Linked list index structure – Simple, easy to implement – Still widely used (e.g., thumb drives) File table: – Linear map of all blocks on disk – Each file a linked list of blocks

23. FAT

24. Pros: – Easy to find free block – Easy to append to a file – Easy to delete a file Cons: – Small file access is slow – Random access is very slow – Fragmentation File blocks for a given file may be scattered Files in the same directory may be scattered Problem becomes worse as disk fills

25. Berkeley UNIX FFS (Fast File System) inode table – Analogous to FAT table inode – Metadata File owner, access permissions, access times, … – Set of 12 data pointers – With 4KB blocks => max size of 48KB files

26. File System Structure Basic unit for allocating space on the disk is a block partition Disk File System Boot Block Super-block i-node table Data blocks

27. I-nodes Each file or directory in the file system has a unique entry in the i-node table. File type (regular, symbolic link, directory…) Owner Permissions Timestamps for last access; last modification, last status change Size …

28. i-node entry 0 … 5 … 11 12 13 14 15 DB 0 DB 5 IPB DB

29. FFS inode Metadata – File owner, access permissions, access times, … Set of 12 data pointers – With 4KB blocks => max size of 48KB files Indirect block pointer – pointer to disk block of data pointers Indirect block: 1K data blocks => 4MB (+48KB)

30. FFS inode Metadata – File owner, access permissions, access times, … Set of 12 data pointers – With 4KB blocks => max size of 48KB Indirect block pointer – pointer to disk block of data pointers – 4KB block size => 1K data blocks => 4MB Doubly indirect block pointer – Doubly indirect block => 1K indirect blocks – 4GB (+ 4MB + 48KB)

31. FFS inode Metadata – File owner, access permissions, access times, … Set of 12 data pointers – With 4KB blocks => max size of 48KB Indirect block pointer – pointer to disk block of data pointers – 4KB block size => 1K data blocks => 4MB Doubly indirect block pointer – Doubly indirect block => 1K indirect blocks – 4GB (+ 4MB + 48KB) Triply indirect block pointer – Triply indirect block => 1K doubly indirect blocks – 4TB (+ 4GB + 4MB + 48KB)

33. FFS Asymmetric Tree Small files: shallow tree – Efficient storage for small files Large files: deep tree – Efficient lookup for random access in large files

34. Operating Systems: A Modern Perspective, Chapter 13 Disk Organization Blk 0 Blk 1 Blk k-1 Blk k Blk k+1 Blk 2k-1 Track 0, Cylinder 0 Track 0, Cylinder 1 Blk Track 1, Cylinder 0 Blk Track N-1, Cylinder 0 Blk Track N-1, Cylinder M-1 … … … … … … … … Boot SectorVolume Directory

35. Operating Systems: A Modern Perspective, Chapter 13 Low-level File System Architecture b 0 b 1 b 2 b 3 b n-1 …… Block 0... Sequential Device Randomly Accessed Device

36. Operating Systems: A Modern Perspective, Chapter 13 Block Management The job of selecting & assigning storage blocks to the file Three basic strategies: – Contiguous allocation – Linked lists – Indexed allocation

37. Operating Systems: A Modern Perspective, Chapter 13 Contiguous Allocation Maps the N blocks into N contiguous blocks on the secondary storage device Difficult to support dynamic file sizes Head position237 … First block785 Number of blocks25 File descriptor

38. Operating Systems: A Modern Perspective, Chapter 13 Linked Lists Each block contains a header with – Number of bytes in the block – Pointer to next block Blocks need not be contiguous Files can expand and contract Seeks can be slow First block … Head: 417... Length Byte 0 Byte 4095... Length Byte 0 Byte 4095... Length Byte 0 Byte 4095... Block 0Block 1Block N-1

39. Operating Systems: A Modern Perspective, Chapter 13 Indexed Allocation Extract headers and put them in an index Simplify seeks May link indices together (for large files) Index block … Head: 417... Byte 0 Byte 4095... Byte 0 Byte 4095... Byte 0 Byte 4095... Block 0 Block 1 Block N-1 Length

40. Operating Systems: A Modern Perspective, Chapter 13 DOS FAT Files Disk Block File Descriptor Disk Block Disk Block … 43 107 254 File Access Table (FAT) Disk Block Disk Block Disk Block … 43 107 43 254 File Descriptor

41. Operating Systems: A Modern Perspective, Chapter 13 UNIX Files Data mode owner … Direct block 0 Direct block 1 … Direct block 11 Single indirect Double indirect Triple indirect inode Data Index Data Index Data Index Data

42. Operating Systems: A Modern Perspective, Chapter 13 Unallocated Blocks How should unallocated blocks be managed? Need a data structure to keep track of them – Linked list Very large Hard to manage spatial locality – Block status map (“disk map”) Bit per block Easy to identify nearby free blocks Useful for disk recovery

43. FFS Locality Block group allocation – Block group is a set of nearby cylinders – Files in same directory located in same group – Subdirectories located in different block groups inode table spread throughout disk – inodes, bitmap near file blocks First fit allocation – Small files fragmented, large files contiguous

45. FFS First Fit Block Allocation

48. FFS Pros – Efficient storage for both small and large files – Locality for both small and large files – Locality for metadata and data Cons – Inefficient for tiny files (a 1 byte file requires both an inode and a data block) – Inefficient encoding when file is mostly contiguous on disk (no equivalent to superpages) – Need to reserve 10-20% of free space to prevent fragmentation

49. NTFS Master File Table – Flexible 1KB storage for metadata and data Extents – Block pointers cover runs of blocks – Similar approach in linux (ext4) – File create can provide hint as to size of file Journalling for reliability – Discussed next time

50. NTFS Small File

51. NTFS Medium File

52. NTFS Indirect Block

53. NTFS Multiple Indirect Blocks

55. Named Data in a File System

56. Operating Systems: A Modern Perspective, Chapter 13 Directories A set of logically associated files and sub directories File manager provides set of controls: – enumerate – copy – rename – delete – traverse – etc.

57. Operating Systems: A Modern Perspective, Chapter 13 Directory Implementation Device Directory – A device can contain a collection of files – Easier to manage if there is a root for every file on the device -- the device root directory File Directory – Typical implementations have directories implemented as a file with a special format – Entries in a file directory are handles for other files (which can be files or subdirectories)

58. Operating Systems: A Modern Perspective, Chapter 13 Directory Structures How should files be organized within directory? – Flat name space All files appear in a single directory – Hierarchical name space Directory contains files and subdirectories Each file/directory appears as an entry in exactly one other directory -- a tree Popular variant: All directories form a tree, but a file can have multiple parents.

59. Directories unixetchomeprodev twd passwd motdunix...slide1slide2

60. Directory Representation Component NameInode Number unix117 etc4 home18 pro36 dev93 directory entry.1..1

61. Directories

62. Directories can be files – Map file name to file number (MFT #, inode num) Table of file name -> file number – Small directories: linear search

63. Large Directories: B-Trees

64. Hard Links unixetchomeprodev twd imagemotdunix...slide1slide2 % ln /unix /etc/image # link system call

65. Directory Representation unix117 etc4 home18 pro36 dev93.4..1 image117 motd33.1..1

66. Create hard links $ echo –n “It is good to collect things, “ > abc $ ln abc xyz $ echo “ but it is better to go on walks” >> xyz $ cat abc It is good to collect things, but it is better to on walks

67. Hard links If one is removed, the other name and the file itself continue to exist.

68. Soft (symbolic) links Differing from a hard link, a soft link (or symbolic link) is a special kind of file containing the name of another file. Created with the ln –s

69. Soft Links unixetchomeprodev twd imagetwd unix... slide1slide2 % ln –s /unix /home/twd/mylink % ln –s /home/twd /etc/twd # symlink system call mylink /unix /home/twd

70. Working Directory Maintained in kernel for each process – paths not starting from “/” start with the working directory – changed by use of the chdir system call – displayed (via shell) using “pwd” how is this done?

71. Operating Systems: A Modern Perspective, Chapter 13 UNIX mount Command / binusretcfoo billnutt abc / blah cdexyz FS / binusretcfoo billnutt abc / blah cdexyz mount FS at foo FS

72. Operating Systems: A Modern Perspective, Chapter 13 VFS-based File Manager File System Independent Part of File Manager File System Independent Part of File Manager Exports OS-specific API Virtual File System Switch MS-DOS Part of File Manager MS-DOS Part of File Manager ISO 9660 Part of File Manager ISO 9660 Part of File Manager ext2 Part of File Manager ext2 Part of File Manager …

73. Operating Systems: A Modern Perspective, Chapter 13 13 File Management

74. Operating Systems: A Modern Perspective, Chapter 13 An open() Operation Locate the on-device (external) file descriptor Extract info needed to read/write file Authenticate that process can access the file Create an internal file descriptor in primary memory Create an entry in a “per process” open file status table Allocate resources, e.g., buffers, to support file usage

75. Operating Systems: A Modern Perspective, Chapter 13 File Manager Data Structures External File Descriptor Open File Descriptor Copy info from external to the open file descriptor 1 Process-File Session Keep the state of the process- file session 2 Return a reference to the data structure 3

76. Operating Systems: A Modern Perspective, Chapter 13 Opening a UNIX File fid = open(“fileA”, flags); … read(fid, buffer, len); 0 stdin 1 stdout 2 stderr 3... Open File Table File structure inode Internal File Descriptor On-Device File Descriptor

77. Operating Systems: A Modern Perspective, Chapter 13 File Descriptors External name Current state Sharable Owner User Locks Protection settings Length Time of creation Time of last modification Time of last access Reference count Storage device details

78. Operating Systems: A Modern Perspective, Chapter 13 Marshalling the Byte Stream Must read at least one buffer ahead on input Must write at least one buffer behind on output Seek  flushing the current buffer and finding the correct one to load into memory Inserting/deleting bytes in the interior of the stream

79. Operating Systems: A Modern Perspective, Chapter 13 Full Block Buffering Storage devices use block I/O Files place an explicit order on the bytes Therefore, it is possible to predict what is likely to be read after byte i When file is opened, manager reads as many blocks ahead as feasible After a block is logically written, it is queued for writing behind, whenever the disk is available Buffer pool – usually variably sized, depending on virtual memory needs – Interaction with the device manager and memory manager

80. File-Descriptor Table 0 1 2 3... n–1 File-descriptor table File descriptor User address space Kernel address space ref count access mode file location inode pointer

81. Allocation of File Descriptors Whenever a process requests a new file descriptor, the lowest numbered file descriptor not already associated with an open file is selected; thus #include close(0); fd = open("file", O_RDONLY); – will always associate file with file descriptor 0 (assuming that the open succeeds)

82. Redirecting Output … Twice if (fork() == 0) { /* set up file descriptors 1 and 2 in the child process */ close(1); close(2); if (open("/home/twd/Output", O_WRONLY) == -1) { exit(1); } if (open("/home/twd/Output", O_WRONLY) == -1) { exit(1); } execl("/home/twd/bin/program", "program", 0); exit(1); } /* parent continues here */

83. Redirected Output File-descriptor table File descriptor 1 User address space Kernel address space File descriptor 2 1 WRONLY 0 inode pointer 1 WRONLY 0 inode pointer

84. Redirected Output After Write File-descriptor table File descriptor 1 User address space Kernel address space File descriptor 2 1 WRONLY 100 inode pointer 1 WRONLY 0 inode pointer

85. Sharing Context Information if (fork() == 0) { /* set up file descriptors 1 and 2 in the child process */ close(1); close(2); if (open("/home/twd/Output", O_WRONLY) == -1) { exit(1); } dup(1); /* set up file descriptor 2 as a duplicate of 1 */ execl("/home/twd/bin/program", "program", 0); exit(1); } /* parent continues here */

86. Redirected Output After Dup File-descriptor table File descriptor 1 User address space Kernel address space File descriptor 2 2 WRONLY 100 inode pointer

87. Fork and File Descriptors int logfile = open("log", O_WRONLY); if (fork() == 0) { /* child process computes something, then does: */ write(logfile, LogEntry, strlen(LogEntry)); … exit(0); } /* parent process computes something, then does: */ write(logfile, LogEntry, strlen(LogEntry)); …

88. File Descriptors After Fork logfile Parent’s address space Kernel address space 2 WRONLY 0 inode pointer logfile Child’s address space

89. Naming (almost) everything has a path name – files – directories – devices (known as special files) keyboards displays disks etc.

90. Uniformity int file = open("/home/twd/data", O_RDWR); // opening a normal file int device = open("/dev/tty", O_RDWR); // opening a device (one’s terminal // or window) int bytes = read(file, buffer, sizeof(buffer)); write(device, buffer, bytes);