1. Nachos Instructional OS: Part 3 CS 170, Tao Yang, Fall 2011

2. 2/13/20162 Topics File system implementation. Assignment 3 Virtual memory File system extension

3. 2/13/20163 Nachos File System Two versions (check openfile.h and filesys.h): A ``stub'' version is simply a front-end to the Linux. A Nachos version implemented on top of a raw disk. Function layers (from top to down): FileSystem object Directory support. I-node support (FileHeader) Free block map OpenFile objects (many) Created by every call to open/access a file in a directory. SynchDisk object Created by synchDisk = new SynchDisk("DISK") in system.cc The Disk object. Created by disk = new Disk(name,..) in SynchDisk::SynchDisk()

4. File System Layers in Nachos Base Operating System (Linux for our class) FileSystem : Directory management file create/open/delete Disk: low-level interface that initiates asynchronous IO to physical disk sectors OpenFile: access individual file with read/write/seek. Operate on bytes. SynchDisk: synchronous access to a disk with concurrent threads. Operate on sectors. Nachos kernel thead Directory: Manage a directory of file names FreeMap Bitmap for 32x32 blocks FileHeader: i-node

5. 2/13/20165 Operations of File System Object FileSystem(bool format). called once at ``boot'' time. For the stub version, just use Unix file system Otherwise create a directory/control info. Sector 0 is fileheader (i-node) for FreeMap. Sector 1 is i-node for directory table. Create(char *name, int initialSize). Create a Nachos disk file. ``initialSize'' is ignored in the stub version; Unix doesn't require it. OpenFile *Open(char *name). Assume both read and write access. in the stub version, Open simply opens the Unix file. Remove(char *name) List()

6. 2/13/20166 Directory Table stored in DirectoryFile Sector 1 is the i-node (file header). File content is 10 table entries. DirectoryFile disk space is allocated in FileSystem() during initialization … Bool inUse --- is this entry used? Int sector --- location of file header (i-node) Filename -- 9 characters+ 1 for ‘\n’ …

7. 2/13/20167 directory.cc Limit FileNameMaxLen 9 // 9 characters at most for a file name NumDirEntries 10 // at most 10 files Directory Object attributes tableSize -- # of directory entries which is 10 now. table = in-memory directory entries, each of which is for one Nachos file. Content is copied from and copies back to a disk DirectoryFile. Operations Directory (int size) -- Create an in-memory directory of the desired size. FetchFrom (OpenFile *file) -- Fetch disk DirectoryFile that contains directory entries, and copy to in-memory directory table. WriteBack(OpenFile *file) -- Write back in-memory directory content to disk DirectoryFile called every time when a file is added or removed to directory. FindIndex (Char *name) --Search a file string name in a directory Add (Char *name, int sector) – Add file name into a directory with disk sector no that contains the file header (i-node). Remove(char *name) – remove a file.

8. 2/13/20168 File header (i-node) … int numBytes; // file size Data sector pointer 0 … int numSectors; //sectors used Data sector pointer 29 Data sector pointer 1 Data sector Data sector Data sector int dataSectors[NumDirect] One filer header per file, fitting one sector (128 bytes) Operations Allocate(BitMap *bitMap, int fileSize); Initialize a file header FetchFrom (int SectorNo) -- Fetch Fileheader sector from disk. WriteBack(int SectorNo) -- Write back in-memory file header content to disk ByteToSector(int offset) –Convert a file offset in bytes to a sector no

9. 2/13/20169 OpenFile Object OpenFile(int sector) sector – the location on disk of the file header. Create an OpenFile object which contains file control info, loaded from the disk file header. Seek(int position) Read/write in the current position Read(char *buffer, int numBytes) Write(char *buffer, int numBytes) Read/write in a specific position ReadAt(char *buffer, int numBytes, int FilePosition). Identify specific sectors in the file and fetch them, copy to buffer. WriteAt(char *buffer, int numBytes, int FilePosition) May read partially written sectors first, and then write out updated sectors. Length(). Return the file size.

10. 2/13/201610 SynchDisk Object: Synchronously access a disk SynchDisk () Called only once to create a physical disk object. Set the disk interrput hander as RequstDone(). Create lock/semaphore for synchronization. Notice that physical disk is an asynchronous device (disk requests return immediately, and an interrupt happens later on). ReadSector(int sectorNumber, char* buffer) Send a read request for a sector Wait IO complete (wait for interrupt using a semaphone) WriteSector(int sectorNumber, char* data) Send a sector write request. Wait for interrupt. RequestDone() Disk interrupt handler. Wake up any thread waiting for the disk request to finish.

11. 2/13/201611 Disk Object Simulates the behavior of a disk I/O device. The disk has only a single platter, with multiple tracks (32). Each track contains the same number of sectors (32). Each sector is 128 bytes Disk size = 32x32*128 =128K Asynchronous read/write with an option of disk cache Supported operations: Disk(char *name, VoidFunctionPtr callWhenDone, int callArg) Create a disk: just create a Unix file to simulate. Setup a handling function (RequestDone in SynchDisk.cc) when a read/write request is done.

12. 2/13/201612 Operations in Disk device ReadRequest(int sectorNumber, char *buffer) Read from the specific location of the “physical disk”. ticks = ComputeLatency(sectorNumber, FALSE); interrupt->Schedule(DiskDone, (int) this, ticks, DiskInt); WriteRequest(int sectorNumber, char *data) Write data to specific location of the “physical disk”. ticks = ComputeLatency(sectorNumber, TRUE); interrupt->Schedule(DiskDone, (int) this, ticks, DiskInt); ComputeLatency(int newSector, bool writing) Latency= seek time + rotation time

13. 2/13/201613 Assignment 3 Workload vm directory for part 1 is empty Need to come up a design and code Test C programs provided are sample code. Amount of work Part 1. ~400 lines of code in vm ( few changes in userprog) Part 2. ~250 or less in filesys (few in userprog) Part 1 and 2 can be done in parallel

14. 2/13/201614 Project 3 part 1: Virtual Memory Work on vm subdirectory mainly And addrspace.cc/.h and exception.cc in userprog Create/manage a backing store (a file called SWAP using the OpenFile class). Implement a page fault handler with dirty bit handling and a page replacement policy (LRU or second chance) Test under various conditions: One process with an address space larger than physical memory. Concurrent processes with combined address space larger than physical memory.

15. 2/13/201615 Project 3 part 2: File system Work on filesys subdirectoy mainly. And change system call handling code in userprog if needed (to use this extended file system). Extend the size of each file from 4K to 100K A few single indirect pointers in i-node. The size of a file can grow.

16. 2/13/201616 Report to be submitted HW3_WRITEUP 1. Summarize what is completed, what is not. 2. For VM, describe the design of VM. 3. For file system extension, Describe design options and their trade-offs (for i-node and for handling file size growth. List/explain main components and code modification in implementing your design it will be graded: Do you write down the design, list advantages/disadvantages of options? Explain the reason of your choice? 4. Summarize the test effort (what is passed, the purpose of each test)

17. 2/13/201617 Emphasis of Project 3 Play tradeoffs Understand the design options, especially for addressing required issue in file system extension. Deadline-driven approach Choose something simpler and easier to do first. Still not enough time? Complete with minimal efforts to pass the tests. Hidden tests are very similar to public test cases. No complex test cases. You are good as long as your code can run virtual memory>physical, can support growable file up to 100K. Interpret the requirement towards your advantage if vague. If not specified, do something easier.

18. HW 3 Implementation Notes

19. Part 1: Getting Started Read machine/translate.cc: In Machine:Translate() for virtual address translation, PageFaultException is detected when the desired page is not in memory. In Machine:ReadMem, Translate() is called for translating the desired virtual memory address and machine->RaiseException() is called with PageFaultException error.

20. What is next Read mipssim.cc Machine->ReadMem() is called in executing each instruction. If PageFaultException is detected, the exception handler should load the desired page. The hardware will try again. Need to expand exception.cc to handle PageFaultException. Once handled, return to user mode and restart the instruction caused the fault

21. User Instruction Execution Machine:Translate() ExceptionHandler() Deal with PageFaultException Cannot find this page? Raise PageFaultException Page writing? Set dirty bit Machine:Run () OneInstruction () ReadMem ()WriteMem () Re-execute if Exception is raised

22. Files to be modified for Part 1 New files in directory vm Virtual memory manager Swap space manager Directory userprog (extension to HW 2) exception.cc Extension to handle PageFaultException Addrspace.cc/.h Prepare code/data for SWAP backstore. Virtual address translation -> paging if needed

23. PageFaultException handling For a virtual address which is not in memory, Find free space to host the desired page in memory. Load the desired page from SWAP to memory. If no free space available, execute a replacement policy to find a victim page. Swap out the victim page. Swap in the desired page. Adjust the page table.

24. Suggested Class in vm Write a Swap Manager to facilitate free sector allocation in Swap File. Initialize SWAPSIZE (512) as the total free sector available. Allocate a sector. Free a sector.

25. Suggested Class in vm Virtual memory manager load a page containing the virtual address (copy from the SWAP file to an allocated memory page). Find a victim page Swap out a selected page to SWAP

26. Modify AddSpace.cc In translating a virtual user address for kernel, if it is not in memory, bring data from SWAP. When allocating a new address space, Allocate a proper number of sectors from SWAP for this process. Copy binary data to the allocated SWAP sectors. Initial page number for this process can be 0.

27. Synchronization issues Two system calls may be processed concurrently and the synchronization is needed. Any time a process sleeps in the kernel, another process can run Two processes try to initiate I/O on the same page at the same time May need a lock (or at least a ``busy'' flag) for each physical memory page, and possibly also for each virtual memory page.

28. Part 2: Steps for Nachos file system extension 1. Understand how the Nachos file system operates and how Nachos implements files. 2. Modify the file system calls from HW 2 to use Nachos file system 3. Modify Nachos file system so that you can create larger (100K) 4. Allow the Write system call to grow the size of a file.

29. Files to be modified in Part 2 Directory filesys filehdr.cc/.h Support single indirect pointers in i- node and can expand the file length with more sectors allocated. openfile.cc Expand file length/allocate sectors when writing data at the end or beyond. directory.cc Add more table entries when the current table is full and a new file is added. Directory userprog exception.cc Make sure that file system calls use extended Nachos file system.

30. Comparison: Unix file i-node (4K bytes per block)

31. Step 3: Extend maximum file size Modify the FileHeader i-node to support 100K size. Include 5 data sector pointers, and 25 sectors for single indirect mapping. At most map 5 + 25*32=805 blocks. Maximum file length = 805*128=103040bytes. Another option: every entry points to an i-node. Suggest to add a class called IndirectPointerBlock which takes 1 sector (32 entries) with following operations. FetchFrom (int sector). WriteBack(int sector). AddSector (int sector) – add a new sector. Deallocate(BitMap *freeMap) – deallocate all sectors. ByteToSector (int offset). – convert an offset in bytes to a sector.

32. Step 4: File size can grow Modify the current implementation so that a write beyond the end of the file extends the size of the file. Another option is: have a fixed size initially to allocate the i-node space initially. But donot allocate data blocks until they are needed. Gracefully handle the case of the disk being full or maximum size is reached - the user process should not crash. Instead an error should be returned.

33. Step 4 Modify WriteAt() method in openfile.cc : Case 1: Write request is within the bound of the file. Handle like the existing code Case 2: Write request overwrites some portion of existing file and goes beyond by N bytes. Case 3: No overwrite to the current data, but goes beyond the end of the file. Here. To extend a file dynamically, Extend i-node/data dynamically: include a routine in FileHeader to extend a file by N bytes. Or pre-allocate the i-nodes, allocate data sectors when needed.

34. Testing issues Clean out all the.c files in the code/test directory. There might be name conflicts from previous projects Get sample test programs in part1 and part2 of the ~cs170/nachos-projtest/HW3 directory. Test part 1. Test as you would for project 2. You can run the./nachos in the vm/ Test part 2. It can be convenient to create another test directory (e.g. test under test2 directory or fielsys/test) Notice file names cannot exceed 9 characters.

35. Testing issues in Part 2 May need to load user binaries/data into Nachos file system during execution Example: csil$ pwd cs170/nachos/code/filesys csil$./nachos -f -cp../test2/Prog1 Prog1 -x Prog1 Another example with two binary programs csil$./nachos -f -cp../test2/Prog2 Prog2 -cp../test2/Prog3/Prog3 -x Prog2