Light-weight Contexts: An OS Abstraction for Safety and Performance Landon Cox March 1, 2017

What is a process? Informal Formal A program in execution Running code + things it can read/write Process ≠ program Formal ≥ 1 threads in their own address space (soon threads will share an address space)

Parts of a process Thread Address space Sequence of executing instructions Active: does things Address space Data the process uses as it runs Passive: acted upon by threads

Play analogy Process is like a play performance Program is like the play’s script What are the threads? Threads What is the address space? Address space

Dynamic address translation User process Translator (MMU) Physical memory Virtual address Physical address Base and bounds Segmentation Paging

What changes on a context switch? Two-level tree 1 … 1023 ? Level 1 NULL NULL Level 2 0: PhysPage, Res, Prot 0: PhysPage, Res, Prot 1: PhysPage, Res, Prot 1: PhysPage, Res, Prot … … ? 1023: PhysPage, Res, Prot 1023: PhysPage, Res, Prot ? What changes on a context switch?

All thread share the same page table Two-level tree 1 … 1023 ? Level 1 NULL NULL Level 2 0: PhysPage, Res, Prot 0: PhysPage, Res, Prot 1: PhysPage, Res, Prot 1: PhysPage, Res, Prot … … ? 1023: PhysPage, Res, Prot 1023: PhysPage, Res, Prot ? All thread share the same page table

Parts of a process (revised) Thread Sequence of executing instructions Active: does things Address space Data the process uses as it runs Passive: acted upon by threads File descriptors Communication channels (e.g., files, sockets, pipes) Passive: read-from/written-to by threads

Parts of a process (revised) Thread 1 Thread 2 Page table File descriptors Stack 1 Stack 2 Readable pages Writable pages TCP socket Open file Each thread has its own stack, but everything else is share: same pages, file descriptors

Parts of a process (revised) Threads are not well isolated from each other. Process Thread 1 Thread 2 Page table File descriptors Stack 1 Stack 2 Readable pages Writable pages TCP socket Open file Threads can run in parallel on different cores.

Parts of a process (revised) Thread 1 Thread 2 Page table File descriptors Stack 1 Stack 2 Readable pages Writable pages TCP socket Open file If one thread fails, entire process fails.

Parts of a process (revised) Thread 2 Page table File descriptors Stack 1 Stack 2 Readable pages Writable pages TCP socket Open file Compromising a thread, compromises all data.

Parts of a process (revised) Thread 2 Stack 1 Stack 2 Readable pages Writable pages TCP socket Open file Compromising a thread, compromises all data.

Parts of a process (revised) Thread 2 Stack 1 Stack 2 Readable pages Writable pages TCP socket Open file See: heartbleed, cloudbleed, etc.

Heartbleed

Heartbleed bug Nasty bug in OpenSSL implementation Made public by Google in April, 2014 “A missing bounds check in the handling of the TLS heartbeat extension.” What is a heartbeat message? A small message to check if machine is alive (like in Raft) In SSL, keeps peers from needlessly renegotiating keys

Heartbleed bug Sequence number + random num struct { HeartbeatMessageType type; uint16 payload_length; opaque payload[HeartbeatMessage.payload_length]; opaque padding[padding_length]; } HeartbeatMessage; … buffer = OPENSSL_malloc(1 + 2 + payload + padding); Created for response Payload length (up to 0xFFFF) Padding length

Heartbleed bug

Parts of a process (revised) Thread 1 Thread 2 Page table File descriptors Stack 1 Stack 2 Readable pages Writable pages TCP socket Open file How do we normally isolate tasks?

Parts of a process (revised) Thread Page table File descriptors … Stack Readable pages Writable pages TCP socket Open file … Separate processes!

Example: Chrome browser

Separate processes What has to happen to switch processes? Trap to the kernel (either by syscall or interrupt) Kernel chooses next process to run Point PTBR to page table of next process to run Flush the TLB Set mode to user Jump to next process Lightweight Contexts (lwCs) Want isolation provided by processes But with faster, easier switching than processes

Processes vs lwCs Processes Fork Exec lwCs Create Switch Create identical copy of vm Use CoW for isolation New process w/ new PID New process now schedulable Exec Launch new program Overwrite existing vm lwCs Create Create identical copy of vm New lwC shares PID w/ parent No execution until switch Simple way to snapshot state Switch Similar to thread yield Switch back resumes at call Restrict, overlay, syscall Define sharing between parent, child Trap syscalls from child in parent Key differences: Processes are scheduled by kernel lwCs are enabled by user code Threads execute w/i an lwC

Example 1 Snapshot and rollback Want to process requests from same state Rollback to eliminate residue from other requests Isolate influence of requests on each other Process starts in main

Server initializes its state Example 1 Snapshot and rollback Want to process requests from same state Rollback to eliminate residue from other requests Isolate influence of requests on each other Server initializes its state

Server creates a snapshot Example 1 Snapshot and rollback Want to process requests from same state Rollback to eliminate residue from other requests Isolate influence of requests on each other Server creates a snapshot

new stores copy (CoW) of current lwC Example 1 Snapshot and rollback Want to process requests from same state Rollback to eliminate residue from other requests Isolate influence of requests on each other new stores copy (CoW) of current lwC

caller is -1 for now (until new is switched to) Example 1 Snapshot and rollback Want to process requests from same state Rollback to eliminate residue from other requests Isolate influence of requests on each other caller is -1 for now (until new is switched to)

Example 1 Snapshot and rollback Want to process requests from same state Rollback to eliminate residue from other requests Isolate influence of requests on each other return new as snapshot

Keep ref to snap so that we can go back. Example 1 Snapshot and rollback Want to process requests from same state Rollback to eliminate residue from other requests Isolate influence of requests on each other Keep ref to snap so that we can go back.

Example 1 Snapshot and rollback Want to process requests from same state Rollback to eliminate residue from other requests Isolate influence of requests on each other Process request.

Now want to use snap to return to initial state. Example 1 Snapshot and rollback Want to process requests from same state Rollback to eliminate residue from other requests Isolate influence of requests on each other Now want to use snap to return to initial state.

Where will switching to snap will return thread to? Example 1 Snapshot and rollback Want to process requests from same state Rollback to eliminate residue from other requests Isolate influence of requests on each other Where will switching to snap will return thread to?

Switching to snap brings us back to create. Example 1 Snapshot and rollback Want to process requests from same state Rollback to eliminate residue from other requests Isolate influence of requests on each other Switching to snap brings us back to create.

Example 1 Snapshot and rollback Want to process requests from same state Rollback to eliminate residue from other requests Isolate influence of requests on each other What is the current lwC?

What are the values of caller and new? Example 1 Snapshot and rollback Want to process requests from same state Rollback to eliminate residue from other requests Isolate influence of requests on each other What are the values of caller and new?

state from processed request Example 1 Snapshot and rollback Want to process requests from same state Rollback to eliminate residue from other requests Isolate influence of requests on each other Close caller to delete state from processed request

Example 1 Snapshot and rollback Want to process requests from same state Rollback to eliminate residue from other requests Isolate influence of requests on each other What is the current lwC?

Why recursively call snapshot? Example 1 Snapshot and rollback Want to process requests from same state Rollback to eliminate residue from other requests Isolate influence of requests on each other Why recursively call snapshot?

new stores copy (CoW) of current lwC Example 1 Snapshot and rollback Want to process requests from same state Rollback to eliminate residue from other requests Isolate influence of requests on each other new stores copy (CoW) of current lwC

caller is -1 for now (until new is switched to) Example 1 Snapshot and rollback Want to process requests from same state Rollback to eliminate residue from other requests Isolate influence of requests on each other caller is -1 for now (until new is switched to)

Example 1 Snapshot and rollback Want to process requests from same state Rollback to eliminate residue from other requests Isolate influence of requests on each other return new as snapshot

Keep ref to snap so that we can go back … Example 1 Snapshot and rollback Want to process requests from same state Rollback to eliminate residue from other requests Isolate influence of requests on each other Keep ref to snap so that we can go back …

Example 2 Sensitive data isolation Only want specific lwC to access crypto key Other lwCs shouldn’t access key Process starts in main

Load key into variable privkey Example 2 Sensitive data isolation Only want specific lwC to access crypto key Other lwCs shouldn’t access key Load key into variable privkey

Create new snapshot lwC that can overlay any part of parent memory Example 2 Sensitive data isolation Only want specific lwC to access crypto key Other lwCs shouldn’t access key Create new snapshot lwC that can overlay any part of parent memory

caller is -1 (until child is switched to) Example 2 Sensitive data isolation Only want specific lwC to access crypto key Other lwCs shouldn’t access key caller is -1 (until child is switched to)

Destroy the key. Which lwC still has a copy? Example 2 Sensitive data isolation Only want specific lwC to access crypto key Other lwCs shouldn’t access key Destroy the key. Which lwC still has a copy?

Restrict current lwC from accessing child memory. Why? Example 2 Sensitive data isolation Only want specific lwC to access crypto key Other lwCs shouldn’t access key Restrict current lwC from accessing child memory. Why?

Loop to process requests. Example 2 Sensitive data isolation Only want specific lwC to access crypto key Other lwCs shouldn’t access key Loop to process requests.

To sign data switch to child lwC, passing in buffer from current lwC. Example 2 Sensitive data isolation Only want specific lwC to access crypto key Other lwCs shouldn’t access key To sign data switch to child lwC, passing in buffer from current lwC.

Why will child be able to access buffer? Example 2 Sensitive data isolation Only want specific lwC to access crypto key Other lwCs shouldn’t access key Why will child be able to access buffer?

Thread starts running in child lwC. Example 2 Sensitive data isolation Only want specific lwC to access crypto key Other lwCs shouldn’t access key Thread starts running in child lwC.

What is the value of caller? Example 2 Sensitive data isolation Only want specific lwC to access crypto key Other lwCs shouldn’t access key What is the value of caller?

Example 2 Sensitive data isolation Only want specific lwC to access crypto key Other lwCs shouldn’t access key Why a stub function?

Map content pointed to by arg in caller to arg in current lwC. Example 2 Sensitive data isolation Only want specific lwC to access crypto key Other lwCs shouldn’t access key Map content pointed to by arg in caller to arg in current lwC.

Example 2 Sensitive data isolation Only want specific lwC to access crypto key Other lwCs shouldn’t access key Why is this allowed?

Perform the actual signing using the key and mapped in arg. Example 2 Sensitive data isolation Only want specific lwC to access crypto key Other lwCs shouldn’t access key Perform the actual signing using the key and mapped in arg.

Why are writes to arg in current lwC visible in caller? Example 2 Sensitive data isolation Only want specific lwC to access crypto key Other lwCs shouldn’t access key Why are writes to arg in current lwC visible in caller?

Why unmap arg after signing? Example 2 Sensitive data isolation Only want specific lwC to access crypto key Other lwCs shouldn’t access key Why unmap arg after signing?

Where will thread resume? Example 2 Sensitive data isolation Only want specific lwC to access crypto key Other lwCs shouldn’t access key Switch back to caller. Where will thread resume?

Back in parent lwC (i.e., previous caller). Example 2 Sensitive data isolation Only want specific lwC to access crypto key Other lwCs shouldn’t access key Back in parent lwC (i.e., previous caller).

Parent lwC can see updates to arg performed in child. Example 2 Sensitive data isolation Only want specific lwC to access crypto key Other lwCs shouldn’t access key Parent lwC can see updates to arg performed in child.

With data signed, process request, loop back for more … Example 2 Sensitive data isolation Only want specific lwC to access crypto key Other lwCs shouldn’t access key With data signed, process request, loop back for more …

Evaluation Is switching between lwCs fast? Vs processes (different page tables) Vs kernel threads (thread queues in kernel) Vs user threads (thread queues in user space)

What is true for each approach (on FreeBSD)? Evaluation Is switching between lwCs fast? Vs processes (different page tables) Vs kernel threads (thread queues in kernel) Vs user threads (thread queues in user space) What is true for each approach (on FreeBSD)?

Note: swapcontext in FreeBSD is a syscall Evaluation Is switching between lwCs fast? Vs processes (different page tables) Vs kernel threads (thread queues in kernel) Vs user threads (thread queues in user space) Note: swapcontext in FreeBSD is a syscall

Note: swapcontext in Linux is not asyscall Evaluation Is switching between lwCs fast? Vs processes (different page tables) Vs kernel threads (thread queues in kernel) Vs user threads (thread queues in user space) Note: swapcontext in Linux is not asyscall

On Linux this number of 6% of lwC … Evaluation Is switching between lwCs fast? Vs processes (different page tables) Vs kernel threads (thread queues in kernel) Vs user threads (thread queues in user space) On Linux this number of 6% of lwC …

Why are processes and k-threads twice as slow as lwCs? Evaluation Is switching between lwCs fast? Vs processes (different page tables) Vs kernel threads (thread queues in kernel) Vs user threads (thread queues in user space) Why are processes and k-threads twice as slow as lwCs?

Evaluation Is switching between lwCs fast? Vs processes (different page tables) Vs kernel threads (thread queues in kernel) Vs user threads (thread queues in user space) lwCs avoid: * scheduling overhead (go right to next lwC)