The need for File Systems Need to store data and programs in files Must be able to store lots of data Must be nonvolatile and survive crashes and power outages Must allow multiple processes concurrent access Store on disks OS manages files in a file system

Different views on file systems User view of file systems - file names, protections, operations File system designers views - how to keep track of free blocks - how disk blocks are grouped and managed to form files First look at files from a users perspective.

User views of file systems File naming - give users useful names for files - MS-DOS limitations, file extensions - NTFS - Unix -256 chars - extensions may have meaning to programs (e.g. cc) OS viewing files as a sequence of bytes gives most flexibility. - leave up to app what to do

File Types Regular files – ascii or binary Directories Character special files Block special files Symbolic links Sockets, Named pipe, Doors The file command, /etc/magic compressed files and archives Sequential files – on tape random access files

File attributes - metadata Returned with stat(2) family mostly - filename (not from stat(2)) - size - mtime, ctime, atime - mode (permissions and type) - nlinks - uid, gid

File operations Create, Delete Open, Close Read, Write Append, Seek Get attributes, Set attributes Rename File descriptors – open files memory-mapped files – map files into process address space

Directories Root directory Could have just root directory with all files in it - not useful on multiuser systems - may be useful for flash, etc. Have directories in the root directory Hierarchical directories – tree Path names - absolute and relative to cwd Directory ops: Create, Delete, Opendir, Closedir, Readdir, Rename, link, unlink

Directories and Files Contents of a directory Unix. and.. directories (dot and dot-dot) Disk partitions Slices Filesystems Disk labels How to store files and directories on disk? Want efficiency and reliability

Contiguous Allocation of files Advantages - simple to implement (address of first block + number of blocks) - very good read performance Disadvantages - Disk becomes fragmented after awhile (need to compact, keep track of holes) - Files change in size CD-ROMS are a good use for this - write once

Linked List allocation of files Keep a linked list of disk blocks no external fragmentation, little internal Need to store first block in directory Random access is slow Pointer takes some disk block space.

Linked List allocation using a table in memory Table has a pointer to each disk block FAT – File Allocation Table Entire block can be used for data Random access works well Problem is entire table needs to be in memory (proportionate to disk size) “Implementing pointers using arrays”

I-nodes Needs to be in memory only when file is open. Point to a bunch of disk blocks. Point to a block that points to more disk blocks. File Attributes 10 or so direct block ptrs More block ptrs

Directories Need to map filenames to disk blocks on disk (inode number) file attributes can be in dir too or elsewhere like in the inode typical filename max length is 255 linear search of a directory can be slow Use a hash table or a tree to speed up lookups and/or cache the searches - dnlc

File storage Storing files on disk have many of the same issues as memory allocation. - contiguous - noncontiguous – with fixed size blocks Block size - too small and too slow - seek time and rotational delay - too big wastes space (internal frag) ½ kB, 1kB, 4kB commonly used Need to keep track of free blocks - use a linked list or a bitmap

Other file issues Disk Quotas - hard and soft limit or just hard limit - time based or not - number of files or just space used Backups – Importance of data - equipment can be replaced - but losing data is unacceptable - Backups to tape or other media - full and incremental, restores - physical security of tapes - offsite copies, encryption

Filesystem consistency System crashes can leave filesystem in inconsistent state. - need for scandisk and fsck - check blocks and files - missing blocks, duplicate blocks - lost+found sync – write out modified blocks - done every 30 seconds fsck can take a long time on large filesystems with lots of files - can make booting up slow

Logging or Journaling A filesystem that logs changes to on disk data in a separate sequential rolling log. - maintain accurate picture of filesystem - speeds up booting greatly - more reliable in case of power failure Records each disk transaction Filesystem can be checked with the log instead of a full scan

Logging or Journaling - log update at start - modify filesystem - marked done When filesystem is checked if intent to change is marked, but not completed file structure for that block is checked. UFS logging, ext3, reisorfs Disksuite, Veritas separate log

Unix Inodes Structure contains metadata (stuff returned by stat(2)) - mode (permissions and type) - nlinks, size - uid, gid - atime, ctime, mtime - device file is on 10 or 12 direct disk block addresses single, double, triple indirect blocks (picture of inodes)

Unix Inodes Small files can be accessed quickly directly from inode. Larger files use the indirect indexing. Capacity of unix files using inodes. - example with 4 byte (32bit) addressing - block size of 1kB (1024bytes)

Unix Inodes 4 byte addresses and 1k block size Level# of blocks # of bytes Direct12 12kB Single Ind kB Double “256*256=65k 65MB Triple Ind256*65k=16M16GB Max size file is 16GB + 65MB +268kB

Unix Inodes newfs, mkfs superblock (found in block 1 and other backup copies) - info about filesystem layout - # of inodes - # of disk blocks - free list for disk blocks Directory entry needs filenames and inode number

Filesystems Berkeley Fast Filesystem, ufs - file names up to 255 chars - cylinder groups – keep data blocks of file close together Linux – ext2, ext3, ext4 XFS, Reisorfs VxFS WAFL (slide to come) ZFS (more on this later) NFS, CIFS Virtual filesystems, /proc

WAFL File System Write Anywhere File Layout from Network Appliance optimized for random writes Used on file servers from NetApp provide files using NFS, CIFS, ftp, http servers have NVRAM cache for writes Ease of use is a principle of WAFL Similar to Berkeley Fast File System, but with several changes. Uses inodes – 16 pointers to blocks (or indirect blocks)

WAFL Snapshots Each filesystem has a root inode A snapshot duplicates a root inode A snapshot is a read only version of the filesystem at the point of time it is taken. Existing blocks are not overwritten. Space used by snapshot is blocks modified since snapshot was taken. Need to keep track of how many snapshots reference a block. When gets to zero the block can be freed. See Figure on page 446