1. OPERATING SYSTEMS COURSE THE HEBREW UNIVERSITY SPRING 2014 1 FUSE File System

2. File system 2 Used to control how data is stored and retrieved What would happen without file system? The file system manages access to the content of the files and the metadata There are files systems for many different kings of storage devices: Hard Disk Flash memory Not locally via a network protocol. For example, the Network File System (NFS) uses Remote Procedure Call (RPC).

3. Virtual File System (VFS) - motivation 3 Standard use of a user requires several files systems  Linux standard file system for the local files  NFS file system for the files in your directory  FAT32 file system in the memory device However, we would like to use the same set of file system commands, e.g. “ls”, “cd”, “open”, “read”, etc.

4. VFS 4 The Virtual File System is an abstraction layer on top of a more concrete file system. A VFS specifies an interface between the kernel and a concrete file system. It allows the client application to access different types of concrete file system in a uniform way.

5. VFS Figure 5

6. Mount 6 Before a user can access a file on a Unix- like machine, the file system that contains it needs to be mounted with the mount command. Mount is used for SD card, USB storage, DVD and other removable storage devices The mount command instructs the operating system that a file system is ready to use, and associates it with a particular point in the overall file system hierarchy (its mount point)

7. Unmount 7 Its counterpart umount instructs the operating system that the file system should be disassociated from its mount point, making it no longer accessible and may be removed from the computer. It is important to umount a device before removing it since changes to files may have only partially written and are completed as part of the umount

8. Filesystem in USErspace (FUSE) 8 FUSE is a loadable kernel module for Unix-like computer operating systems that lets non-privileged users create their own file systems without editing kernel code This is achieved by running file system code in user space while the FUSE module provides only a "bridge" to the actual kernel interfaces FUSE is often used for writing virtual file system, which don’t actually store data, but only translate existing file system. Extremely efficient and therefore usefull.

9. How FUSE Works 9 The FUSE kernel module and the FUSE library communicate via a special file descriptor which is obtained by opening /dev/fuse This file can be opened multiple times, and the obtained file descriptor is passed to the mount system call, to match up the descriptor with the mounted file system

10. What Do People use FUSE for? 10 Archive file systems Accessing the content of archive (i.e. tar or zip) without decompressing them. Encrypted file systems Storing files in a more secure way Database file systems Storing files in a relational database (MySQL, BerkeleyDB etc.) Monitoring file systems Providing notification when a file changes

11. HELLO WORLD FILE SYSTEM 11 FUSE Example #1

12. Hello World File System 12 In the next slides, we’ll be reviewing the Hello_World example This file system consists of a single file, with a predefined message, which can only be read: static const char *hello_str = "Hello World!\n"; // the string static const char *hello_path = "/hello"; To be honest, the file is not really read, but that’s what the user is lead to believe You can find the original documentation of this example herehere

13. Programming 13 Include the following: #include

14. struct fuse_operations 14 Instead of using the system’s implementation of the Unix File System API (which we reviewed in a previous TA), FUSE replaces it with the user’ implementation This struct defines which of the functions you provide should be used to implement the file system API instead of the default system’s implementation In this example, we’ll be overriding only 4 functions The fuse.h file contains the struct, and so a list of all the functions that can be overridden

15. struct fuse_operations (2) 15 How we use struct fuse_operations ? 1. Create a struct. 2. Init it with pointers to our functions 3. Send it to the fuse in the fuse_main function. Two different designs are possible 1. “static design”, the struct and all the functions are static. We will see example in the following slides. 2. “non-static design”, both are not static. We used this design in your exercise.

16. struct fuse_operations Hello World example 16 static struct fuse_operations hello_oper = {.getattr = hello_getattr,.readdir = hello_readdir,.open = hello_open,.read = hello_read, }; /* In our case, you must init the rest of the functions to NULL. We have done that for you… /*

17. fuse_file_info struct 17 This struct is given as input in may fuse functions. In the EX, you will use only the flags & fh fields.

18. hello_getattr 18 This function returns metadata concerning a file specified by path in a special stat structure

19. hello_getattr 19 static int hello_getattr(const char *path, struct stat *stbuf){ int res = 0; memset(stbuf, 0, sizeof(struct stat)); //Reset vals to 0 if (strcmp(path, "/") == 0) { // Root directory stbuf->st_mode = S_IFDIR | 0755; //irrelevant for you stbuf->st_nlink = 2; // # of hard links } else if (strcmp(path, hello_path) == 0) { stbuf->st_mode = S_IFREG | 0444; //irrelevant for you stbuf->st_nlink = 1; stbuf->st_size = strlen(hello_str); //size } else res = -ENOENT; return res; }

20. hello_readdir 20 Used to read directory contents Because this implementation is very simple, we will be interested only in path, buf and filler arguments passed to it Path is the path to the directory from which we will have to read our content buf will hold them, filler is a fuse_fill_dir_t function which we will use to add contents to directory Offset and fi are not important now

21. hello_readdir - cont. 21 static int hello_readdir(const char *path, void *buf, fuse_fill_dir_t filler, off_t offset, struct fuse_file_info *fi){ (void) offset; (void) fi; if (strcmp(path, "/") != 0) return -ENOENT; filler(buf, ".", NULL, 0); //Add "." directory entry filler(buf, "..", NULL, 0); //Add “.." filler(buf, hello_path + 1, NULL, 0); // w.o. “/” return 0; }

22. hello_open 22 This function is used to open a file. Before opening, it must check whatever the user is permitted to open it (using the flags given in the fuse_file_info structure) In our case, there is no need to open the file, because the content is saved in the memory (it’s a string). static int hello_open(const char *path, struct fuse_file_info *fi){ if (strcmp(path, hello_path) != 0) return -ENOENT; if ((fi->flags & 3) != O_RDONLY) return -EACCES; return 0; // Can open the file }

23. hello_read 23 Used to feed the user with data from the file Path is the path to the file to read buf is a buffer to which we will copy information from the file, size is at the same time the size of the buffer and tells how much of the data we have to read offset is the place from which we have to start reading fi contains additional information about the type of access

24. hello_read - cont. 24 static int hello_read(const char *path, char *buf, size_t size, off_t offset, struct fuse_file_info *fi){ size_t len; (void) fi; if(strcmp(path, hello_path) != 0) return -ENOENT; // Can’t open a different file len = strlen(hello_str); if (offset < len) { if (offset + size > len) size = len - offset; memcpy(buf, hello_str + offset, size); } else size = 0; return size; }

25. The main function 25 Should always return the value received from calling fuse_main In the Hello World example: int main(int argc, char *argv[[]]) { return fuse_main(argc, argv, &hello_oper, NULL); }

26. Compiling and Running 26 gcc -Wall `pkg-config fuse --cflags -- libs` hello.c -o hello Run it with a single parameter - a mount point: $mkdir tmp $./hello tmp/ $

27. Compiling and Running – cont. 27 After running the program you should have a single file in the tmp/ directory named hello, its content should be Hello World!: $ ls tmp/ hello $ cat tmp/hello Hello World!

28. Unmount 28 To unmount, use: fusermount -u /tmp/fuse

29. BIG BROTHER FILE SYSTEM 29 FUSE Example #2

30. Big Brother File System 30 This FUSE program receives 2 arguments The root directory, which contains the actual directory data The mount directory All of the files that are really in rootdir appear to also be in mountdir. But, every time you perform any file operation in mountdir, the operation (and a whole bunch of both relevant and irrelevant stuff) gets logged to a new file in the current working directory called bbfs.log

31. Big Brother File System - How To Run 31 Download the tar file from the tutorial websitethe tutorial website Use the make file to compile the code in the /src folder Go to the /example directory, and execute:../src/bbfs rootdir mountdir Execute tail -F bbfslog in another terminal window to watch the operations get logged

32. 32 Ex4 – Caching File System Using FUSE

33. Ex4 - Assignment 33 In this exercise you will implement a simple single threaded caching file system and a simple algorithm to manage the file system's cache memory Caching file systems saves files or parts of them to a memory segment called cached memory, which is stored in the main memory That way, if the user wishes to access information that is saved in the cache, it can be retrieved from the cache instead of the disk, thus reducing the number of disk access required

34. Guidelines 34 Start with a clean slate – don’t start with a working system as a base We’ll be providing a clean slate file for you – use it! Going through the Big Brother FUSE file system comes in handy when trying to understand the different structs and how they work We strongly suggest you not to do to copy lines from there. Don’t use any line (e.g.) a function that you don’t fully understand!

35. Our Caching File System 35 The caching in this exercise (not in real life though) is for read only, so you don’t have to implement the write system call It does not mean you won’t have to implement other system calls relating to read You are expected to separate the logic of the cache from the FUSE functions

36. FUSE Tips (1) 36 Mount folder must be empty! You should use only directories within “\tmp”. Don’t try to use files within your user, because they are accessed via NFS. FUSE doesn’t support non-static global variables. If you want to have a global variable – make it static. FUSE can’t be runing on the System’s servers. We suggest you to work from HUJI, but you may also work from home using virtual machine

37. FUSE Tips (2) 37 Be sure to unmount, even if your program cashes or you abort it You won’t be able to mount again until you do FUSE disables the cout prints, so print out a file in order to debug You may also use the FUSE –f flag, which will make the program run in the foreground (and therefore the prints will be visible).

38. Running and debugging 38 A bug may lead your OS to collapse. Make sure that you follow all the Guidelines and Tips mentioned in the exercise. Try to implement and check each function separately. We suggest you to have two phases of testing  Basic tests using shell commands.  More sophisticated tests via scripts.