Secure Operating System

Mandatory Protection Systems Problem of discretionary access control: untrusted processes can modify protection states Mandatory protection system: – Subjects and objects represented by labels – Protection state: the operations that subject labels may perform on object labels – Labeling state: mapping objects to labels – Transition state: defines what relabeling is allowed

Example secretunclassifiedtrusteduntrusted secret unclassified trusted untrusted file1file2 Process 1 Process 2 R,W R R R W W R Labeling State R,W … Transistion State Protection State

Mandatory Access Control In a mandatory protection system – The set of labels are defined by trusted administrators – The set of labels are immutable – Protection state, labeling state, and transition state can only be modified by trusted administrators through trusted programs This is called Mandatory Access Control (MAC)

Reference Monitor An authorization system that determines whether a subject is allowed to perform an operation on an object – Takes as input a request – Returns a binary response indicating whether the request is authorized or not

Source: Operating system security, Jaeger’08, Morgan & Claypool

Secure Operating System A system with a reference monitor access enforcement mechanism that satisfies the requirements below when it enforces a mandatory protection system. – Complete Mediation: all security-sensitive ops – Tamperproof: untrusted processes cannot modify access enforcement system – Verifiable: small TCB

Examining Unix Complete mediation – Problem1: not all file access is mediated by RM, e.g., if a process possesses a file descriptor, it can perform any ad hoc command on the file using system calls ioctl or fcntl, as well as read and modify file metadata. – Problem 2: not all system resources are mediated

Examining Unix Tamperproof – Any user process can modify the protection state at its discretion. – User processes can access and modify kernels through special file systems (e.g., /proc, /kmem.) – Any root user process can modify any aspect of the protection system

Examining Unix Verifiable – Effectively unbounded TCB – Impossible to prove that security goals are met as long as TCB is OK