Protection and OS Structure Andrew Whitaker CSE451

Protection Challenge: OS must safely support multiple protection domains OS as “law enforcement” Goals Buggy application can’t crash the system Malicious application can’t take control User data is protected from untrusted users and programs Not always the case: e.g., DOS

The User/Kernel boundary Implemented in hardware Allows the OS to execute privileged instructions Applications enter kernel mode by executing a system call App App App user OS kernel

Examples of Privileged Instructions Manipulating I/O devices Why? Interrupt enable/disable flag Halt instruction

System Call Overview User program invokes helper procedure e.g., read, write, gettimeofday Helper passes control to the OS Indicates the system call number Packages user arguments into registers Issues a software interrupt (or trap) OS saves user state (registers) OS invokes appropriate system call handler OS returns control to the user application

A kernel crossing illustrated Firefox: read(int fileDescriptor, void *buffer,int numBytes); ) package arguments trap to kernel mode user mode kernel mode restore app state, return to user mode, resume trap handler save registers find sys_read( ) handler in vector table sys_read( ) kernel routine

Kernel Entry Points Interrupts Software interrupts (traps, exceptions) Disk, network, timer, etc. Software interrupts (traps, exceptions) System calls Protection violations e.g,. User executes a privileged instructions Page faults Error conditions e.g., divide by zero, illegal opcode

Memory Protection Problem #1: OS must protect applications from each other Solution: virtual memory -- each application has its own address space, which maps to private physical pages Problem #2: Kernel must protect its own code and data Solution: Split address space in half Kernel half requires privileged mode access

Simplified Linux Address Space Layout kernel space user-accessible Q: Why is the zero page unmapped? 0xc0000000

Other Forms of OS Protection Disk protection: Expressed in terms of file system access control permissions (UNIX: read, write, execute) drwxr-xr-x 4 gaetano www 4096 Mar 15 2005 sewpc drwxrwx--x 4 zahorjan www 4096 Mar 15 2005 software drwxrwxr-x 9 levy www 4096 Mar 16 2005 sosp16 -rw------- 1 lazowska www 2006 Oct 9 1998 staff drwxrwxr-x 3 beame ctheory 4096 Jun 1 2002 stoc96 CPU protection: Must guarantee each process a fraction of the CPU users files

Sample Test Question: Insecure System Call Consider a hypothetical system call, zeroFill, which fills a user buffer with zeroes: zeroFill(char* buffer, int bufferSize); The following kernel implementation of zeroFill contains a security vulnerability. What is the vulnerability, and how would you fix it? void sys_zeroFill(char* buffer, int bufferSize) { for (int i=0; i < bufferSize; i++) { buffer[i] = 0; }}

Solution The user buffer pointer is untrusted, and could point anywhere. In particular, it could point inside the kernel address space. This could lead to a system crash or security breakdown. Fix: verify the pointer is a valid user address

Follow-up Question Is it a security risk to execute the zeroFill function in user-mode? void zeroFill(char* buffer, int bufferSize) { for (int i=0; i < bufferSize; i++) { buffer[i] = 0; }}

Solution No. User-mode code does not have permission to access the kernel’s address space. If it tries, the hardware raises an exception, which is safely handled by the OS More generally, no user mode code should ever be a security vulnerability. Unless the OS has a bug…

Sample Test Question What bad thing could happen if a user application could overwrite the interrupt dispatch vector? How does the OS prevent this?

Solution An application could: 1) Prevent I/O operations from ever completing; 2) Prevent time from advancing, thus dominating the processor Applications cannot modify the interrupt vector because it lives in the kernel address space. Any attempt to modify the interrupt vector raises a kernel exception, which is safely handled.

Sample Test Question What prevents an application from directly reading from the disk, instead of passing through file system access control checks?

Solution Instructions for manipulating I/O devices are privileged. Any attempt to use them in user mode raises a protection exception, which the operating system gracefully handles.