1. Protection and OS Structure
Andrew Whitaker
CSE451

2. Protection
Challenge: OS must safely support multiple protection domains
OS as “law enforcement”
Goals
Buggy application can’t crash the system
Malicious application can’t take control
User data is protected from untrusted users and programs
Not always the case: e.g., DOS

3. The User/Kernel boundary
Implemented in hardware
Allows the OS to execute privileged instructions
Applications enter kernel mode by executing a system call
App
App
App
user OS
kernel

4. Examples of Privileged Instructions
Manipulating I/O devices
Why?
Interrupt enable/disable flag
Halt instruction

5. System Call Overview User program invokes helper procedure
e.g., read, write, gettimeofday
Helper passes control to the OS
Indicates the system call number
Packages user arguments into registers
Issues a software interrupt (or trap)
OS saves user state (registers)
OS invokes appropriate system call handler
OS returns control to the user application

6. A kernel crossing illustrated
Firefox: read(int fileDescriptor, void *buffer,int numBytes); )
package arguments
trap to kernel mode
user mode
kernel mode
restore app state, return to user mode, resume
trap handler
save registers
find sys_read( ) handler in vector table
sys_read( ) kernel routine

7. Kernel Entry Points Interrupts Software interrupts (traps, exceptions)
Disk, network, timer, etc.
Software interrupts (traps, exceptions)
System calls
Protection violations
e.g,. User executes a privileged instructions
Page faults
Error conditions
e.g., divide by zero, illegal opcode

8. Memory Protection
Problem #1: OS must protect applications from each other
Solution: virtual memory -- each application has its own address space, which maps to private physical pages
Problem #2: Kernel must protect its own code and data
Solution: Split address space in half
Kernel half requires privileged mode access

9. Simplified Linux Address Space Layout
kernel space
user-accessible
Q: Why is the zero page unmapped?
0xc

10. Other Forms of OS Protection
Disk protection:
Expressed in terms of file system access control permissions (UNIX: read, write, execute)
drwxr-xr-x 4 gaetano www Mar sewpc
drwxrwx--x 4 zahorjan www Mar software
drwxrwxr-x 9 levy www Mar sosp16
-rw lazowska www Oct staff
drwxrwxr-x 3 beame ctheory Jun stoc96
CPU protection:
Must guarantee each process a fraction of the CPU
users
files

11. Sample Test Question: Insecure System Call
Consider a hypothetical system call, zeroFill, which fills a user buffer with zeroes:
zeroFill(char* buffer, int bufferSize);
The following kernel implementation of zeroFill contains a security vulnerability. What is the vulnerability, and how would you fix it?
void sys_zeroFill(char* buffer, int bufferSize) {
for (int i=0; i < bufferSize; i++) {
buffer[i] = 0; }}

12. Solution
The user buffer pointer is untrusted, and could point anywhere. In particular, it could point inside the kernel address space. This could lead to a system crash or security breakdown.
Fix: verify the pointer is a valid user address

13. Follow-up Question
Is it a security risk to execute the zeroFill function in user-mode?
void zeroFill(char* buffer, int bufferSize) {
for (int i=0; i < bufferSize; i++) {
buffer[i] = 0; }}

14. Solution
No. User-mode code does not have permission to access the kernel’s address space. If it tries, the hardware raises an exception, which is safely handled by the OS
More generally, no user mode code should ever be a security vulnerability.
Unless the OS has a bug…

15. Sample Test Question
What bad thing could happen if a user application could overwrite the interrupt dispatch vector?
How does the OS prevent this?

16. Solution
An application could: 1) Prevent I/O operations from ever completing; 2) Prevent time from advancing, thus dominating the processor
Applications cannot modify the interrupt vector because it lives in the kernel address space. Any attempt to modify the interrupt vector raises a kernel exception, which is safely handled.

17. Sample Test Question
What prevents an application from directly reading from the disk, instead of passing through file system access control checks?

18. Solution
Instructions for manipulating I/O devices are privileged. Any attempt to use them in user mode raises a protection exception, which the operating system gracefully handles.