1. Operating System Security
Trent Jaeger
The Pennsylvania State University
SYNTHESIS LECTURES ON INFORMATION SECURITY, PRIVACY AND TRUST #1
Morgan &cLaypool publishers

2. Introduction
Operating systems provide the fundamental mechanisms for securing computer processing. Since the 1960s, operating systems designers have explored how to build “secure” operating systems —operating systems whose mechanisms protect the system against a motivated adversary. Recently, the importance of ensuring such security has become a mainstream issue for all operating systems.

4. Three major tasks
Operating systems must provide efficient resource mechanisms,
Second, it is the operating system’s responsibility to switch among the processes fairly
Third, access to resources should be controlled, such that one process cannot inadvertently or maliciously impact the execution of another.

5. This third task is the problem of ensuring the security of all processes
run on the system.
Ensuring the secure execution of all processes depends on the correct
implementation of resource and scheduling mechanisms.
Security becomes an issue because processes in modern computer
systems interact in a variety of ways, and the sharing of data among
users is a fundamental use of computer systems.
First, the output of one process may be used by other processes.
Second With the ubiquity of Internet-scale sharing mechanisms, such as , the web, and instant messaging, users may share anything with anyone in the world

6. The challenge in developing operating systems security is to design
security mechanisms that protect process execution and their generated
data in an environment with such complex interactions.
The current state of operating systems security takes two forms:
(1) constrained systems that can enforce security goals with a high degree of assurance and
(2) general-purpose systems that can enforce limited security goals
with a low to medium degree of assurance.

7. Security Goal
A secure operating system provides security mechanisms that ensure that
the system's security goals are enforced despite the threats faced by the
system.
Systems that provide a high degree of assurance in enforcement have
been called secure systems, or even more frequently “trusted”
Systems. However, it is also true that no system of modern complexity is
completely secure.
A security goal defines the operations that can be executed by a system
while still preventing unauthorized access.
Security goals describe how the system implements accesses to system
resources that satisfy the following:
1. secrecy,
2. integrity,
3. and availability.

8. An example of an functional security goal is the principle of least
privilege, which limits a process to only the set of operations necessary for
its execution.
To build any secure system requires that we consider how the
system achieves its security goals under a set of threats (i.e.,
a threat model) and given a set of software, including the security
mechanisms, that must be trusted(i.e., a trust model).
TRUSTMODEL
A system’s trust model defines the set of software and data upon which the
system depends for correct enforcement of system security goals. For
example, the operating system depends on a variety of programs to
authenticate the identity of users (e.g., login and SSH).

9. of data that the user depends on.
Threat Model
A threat model defines a set of operations that an attacker may use to
compromise a system. If an attacker is able to find a vulnerability in the system
that provides access to secret information (i.e., violate secrecy goals) or permits
the modification of information that subjects depend on (i.e.,violate integrity
goals), then the attacker is said to have compromised the system.
This threat model exposes a fundamental weakness in commercial operating systems
they assume that all software running on behalf of a subject is trusted by that
subject. This can result in the leakage of that user’s secrets and the modification
of data that the user depends on.

10. Access Control
An access enforcement mechanism authorizes requests from multiple
subjects (e.g. users, processes, etc.) to perform operations (e.g., read,
write, etc.) on objects (e.g., files, sockets, etc.).
An operating system provides an access enforcement mechanism.
Two fundamental concepts of access control:
a protection system that defines the access control specification and
a reference monitor that is the system’s access enforcement
mechanism that enforces this specification.

11. Protection system
A protection system consists of a protection state, which describes the
operations that system subjects can perform on system objects, and a
set of protection state operations, which enable modification of that
state.
A protection system enables the definition and management of a
protection state. A protection state consists of the specific system
subjects, the specific system objects, and the operations that those
subjects can perform on those objects.
The access matrix is used to define the protection domain of a process.

12. Mandatory protection system
Problems with access matrix
Untrusted processes can tamper with the protection system.
A protection system that permits untrusted processes to modify the protection state is called a discretionary access control (DAC) system.
Mandatory protection system
A mandatory protection system is a protection system that can only be
modified by trusted administrators via trusted software, consisting of the
following state representations:
A mandatory protection state is a protection state where subjects and objects are represented by labels where the state describes the operations that subject labels may take upon object labels;
A labelling state for mapping processes and system resource objects to labels;
A transition state that describes the legal ways that processes and system resource objects may be relabeled.

13. Mandatory access control
A label is simply an abstract identifier—the assignment of permissions to a label defines its security semantics. Labels are tamperproof .
Trusted administrators define the access matrix’s labels and set the operations that subjects of particular labels can perform on objects of particular labels. Such protection systems are mandatory access control (MAC) systems because the protection system is immutable to untrusted processes.

15. A reference monitor is the classical access enforcement mechanism
A reference monitor is the classical access enforcement mechanism. It takes a request as input, and returns a binary response Indicating whether the request is authorized by the reference monitor’s access control policy. We identify three distinct components of a reference monitor: (1) its interface -The interface defines where the authorization module needs to be invoked to perform an authorization query to the protection state, a labeling query to the labeling state, or a transition query to the transition state. (2) its authorization module-determines the exact queries that are to be made to the policy store. ; (3) Its policy store-The policy store responds to authorization, labeling, and transition queries based on the protection system that it maintains..

17. Concluding Remarks
A secure operating system is an operating system where its access
enforcement satisfies the reference monitor concept
The reference monitor concept defines the necessary and sufficient properties
of any system that securely enforces a mandatory protection system, consisting
of three guarantees:
1. Complete Mediation: The system ensures that its access enforcement mechanism mediates all security-sensitive operations.
2. Tamperproof: The system ensures that its access enforcement mechanism, including its protection system, cannot be modified by untrusted processes.
3. Verifiable:The access enforcement mechanism, including its protection system,“must be small enough to be subject to analysis and tests, the completeness of which can be assured” .
That is, we must be able to prove that the system enforces its security goal correctly.