Shreds: Fine-grained Execution Units with Private Memory Landon Cox February 21, 2018

Parts of a process Threads Address space File descriptors Sequences of executing instructions Active: do things Address space Data the process uses as it runs Passive: acted upon by threads File descriptors Communication channels (e.g., files, sockets, pipes) Passive: read-from/written-to by threads

Parts of a process Process Thread 1 Thread 2 Page table File descriptors Stack 1 Stack 2 Readable pages Writable pages TCP socket Open file Each thread has its own stack, but everything else is share: same pages, file descriptors

Parts of a process Threads are not well isolated from each other. Page table File descriptors Stack 1 Stack 2 Readable pages Writable pages TCP socket Open file Threads can run in parallel on different cores.

Parts of a process (revised) Thread 1 Thread 2 Page table File descriptors Stack 1 Stack 2 Readable pages Writable pages TCP socket Open file If one thread fails, entire process fails.

Parts of a process (revised) Thread 2 Page table File descriptors Stack 1 Stack 2 Readable pages Writable pages TCP socket Open file Compromising a thread, compromises all data.

Parts of a process (revised) Thread 2 Stack 1 Stack 2 Readable pages Writable pages TCP socket Open file Compromising a thread, compromises all data.

Parts of a process (revised) Thread 2 Stack 1 Stack 2 Readable pages Writable pages TCP socket Open file See: heartbleed, cloudbleed, etc.

Heartbleed

Threads vs shreds How do you define a thread’s work? Pass in a starting method The thread exits when the starting function exits What part of the address space can a thread access? A thread can access any read- or write-enabled page All threads are bound by the same protections A shred’s work can be more or less than a method Enter a shred using: shred_enter Exit a shred using: shred_exit A shred can access secrets in a private memory pool (s-pool)

Shred protection guarantees Exclusive access to an s-pool Only associated shreds can access data in an s-pool Other threads, unassociated shreds cannot access No leaks S-pool data cannot be copied w/o sanitization Use static analysis to identify outgoing data flows Untampered execution Shred code cannot be changed or routed elsewhere i.e., control-flow integrity w/i shred (including libs) S-driver S-compiler

Shred protection guarantees Exclusive access to an s-pool Only associated shreds can access data in an s-pool Other threads, unassociated shreds cannot access S-driver

S-driver ARM memory domains Three access levels specified in DACR Provide memory-access control independent of page protections 4-bit tags specified in top-level page tables Access level for each domain set via per-core register (DACR) MMU checks domain access for virtual memory address Three access levels specified in DACR No-access mode: all accesses to domain memory fail Manager mode: all access to domain memory are allowed Client mode: all accesses obey page protections

4-bit flag in top-level entry Two-level table 4-bit flag in top-level entry 1 … 1023 ? Level 1 Domain Domain Domain NULL NULL Level 2 0: PhysPage, Res, Prot 0: PhysPage, Res, Prot 1: PhysPage, Res, Prot 1: PhysPage, Res, Prot … … 1023: PhysPage, Res, Prot ? 1023: PhysPage, Res, Prot ?

All memory under domain 0 Two-level table All memory under domain 0 1 … 1023 ? Level 1 Domain Domain Domain NULL NULL Level 2 0: PhysPage, Res, Prot 0: PhysPage, Res, Prot 1: PhysPage, Res, Prot 1: PhysPage, Res, Prot … … 1023: PhysPage, Res, Prot ? 1023: PhysPage, Res, Prot ?

Two-level table … … ? ? 1 … 1023 ? Level 1 Level 2 Top 4MB now in domain 1 1 … 1023 ? Level 1 Domain Domain Domain 1 NULL NULL Level 2 0: PhysPage, Res, Prot 0: PhysPage, Res, Prot 1: PhysPage, Res, Prot 1: PhysPage, Res, Prot … … 1023: PhysPage, Res, Prot ? 1023: PhysPage, Res, Prot ?

Can put other regions under other domains Two-level table Can put other regions under other domains 1 … 1023 ? Level 1 Domain 2 Domain Domain 1 NULL NULL Level 2 0: PhysPage, Res, Prot 0: PhysPage, Res, Prot 1: PhysPage, Res, Prot 1: PhysPage, Res, Prot … … 1023: PhysPage, Res, Prot ? 1023: PhysPage, Res, Prot ?

Each core has a 32-bit DACR Two-level table 1 … 1023 ? Level 1 Domain 2 Domain Domain 1 NULL NULL Level 2 0: PhysPage, Res, Prot 0: PhysPage, Res, Prot 1: PhysPage, Res, Prot 1: PhysPage, Res, Prot … … 1023: PhysPage, Res, Prot ? 1023: PhysPage, Res, Prot ? Each core has a 32-bit DACR

Two-level table … … ? ? 1 … 1023 ? Level 1 Level 2 1 … 1023 ? Level 1 Domain 2 Domain Domain 1 NULL NULL Level 2 0: PhysPage, Res, Prot 0: PhysPage, Res, Prot 1: PhysPage, Res, Prot 1: PhysPage, Res, Prot … … 1023: PhysPage, Res, Prot ? 1023: PhysPage, Res, Prot ? 16 domains x 2 mode bits

Mode bits encode client, no-access, manager modes Two-level table 1 … 1023 ? Level 1 Domain 2 Domain Domain 1 NULL NULL Level 2 0: PhysPage, Res, Prot 0: PhysPage, Res, Prot 1: PhysPage, Res, Prot 1: PhysPage, Res, Prot … … 1023: PhysPage, Res, Prot ? 1023: PhysPage, Res, Prot ? Mode bits encode client, no-access, manager modes

Note changing DACR does not require TLB flush! Two-level table 1 … 1023 ? Level 1 Domain 2 Domain Domain 1 NULL NULL Level 2 0: PhysPage, Res, Prot 0: PhysPage, Res, Prot 1: PhysPage, Res, Prot 1: PhysPage, Res, Prot … … 1023: PhysPage, Res, Prot ? 1023: PhysPage, Res, Prot ? Note changing DACR does not require TLB flush!

Mode change should be faster than page change Two-level table 1 … 1023 ? Level 1 Domain 2 Domain Domain 1 NULL NULL Level 2 0: PhysPage, Res, Prot 0: PhysPage, Res, Prot 1: PhysPage, Res, Prot 1: PhysPage, Res, Prot … … 1023: PhysPage, Res, Prot ? 1023: PhysPage, Res, Prot ? Mode change should be faster than page change

Problem with only 16 domains? Two-level table Problem with only 16 domains? 1 … 1023 ? Level 1 Domain 2 Domain Domain 1 NULL NULL Level 2 0: PhysPage, Res, Prot 0: PhysPage, Res, Prot 1: PhysPage, Res, Prot 1: PhysPage, Res, Prot … … 1023: PhysPage, Res, Prot ? 1023: PhysPage, Res, Prot ?

Every s-pool is assigned a memory domain (e.g., 3-7) Linux No access Page permissions Unused 15 14 … 10 9 8 7 6 5 4 3 2 1 Every s-pool is assigned a memory domain (e.g., 3-7)

Assignments change depending on if shred is active and on which core. Linux No access Page permissions Unused 15 14 … 10 9 8 7 6 5 4 3 2 1 Assignments change depending on if shred is active and on which core.

Each core gets an exclusive domain Linux No access Page permissions Unused 15 14 … 10 9 8 7 6 5 4 3 2 1 Each core gets an exclusive domain

All inactive shreds’ s-pools are tagged w/ red domain. 15 14 … 10 9 8 7 6 5 4 3 2 1 While shred is active on core, its s-pool is tagged w/ core’s green domain.

Say we load shred 0 onto core 0. 15 14 … 10 9 8 7 6 5 4 3 2 1 S0

Which domain does S0’s s-pool need to be tagged with? 15 14 … 10 9 8 7 6 5 4 3 2 1 S0

Why will S0 be able to access its s-pool? 15 14 … 10 9 8 7 6 5 4 3 2 1 S0

Why can’t other shreds access S0’s s-pool? 15 14 … 10 9 8 7 6 5 4 3 2 1 S0

Can S0 run in parallel on > 1 core? 15 14 … 10 9 8 7 6 5 4 3 2 1 S0

How do we tag S0’s s-pool when it’s done? Why? 15 14 … 10 9 8 7 6 5 4 3 2 1 S0

What happens to the TLB on when an s-pool’s tag changes? 15 14 … 10 9 8 7 6 5 4 3 2 1 S0

Where does a shred’s stack have to reside? Why? 15 14 … 10 9 8 7 6 5 4 3 2 1 S0

int http_request_parse(server *srv, connection *con) { … /* inside the request parsing loop */ char *cur; /* current parsing offset */ char auth_str[] = "Authorization"; int auth_str_len = strlen(auth_str); if (strncmp(cur, auth_str, auth_str_len)==0){ shred_enter(AUTH_PASSWD_POOL); /* object holding passwd in spool */ data_string *ds = s_ds_init(); int pw_len = get_passwd_length(cur); cur += auth_str_len + 1; buffer_copy_string_len(ds->key, auth_str, auth_str_len); buffer_copy_string_len(ds->value, cur, pw_len); /* add ds to header pointer array */ array_insert_unique(parsed_headers, ds); /* only related shreds can deref ds */ /* wipe out passwd from input stream */ memset(cur, 0, pw_len); cur += pw_len; shred_exit(); }

Parse authorization request. int http_request_parse(server *srv, connection *con) { … /* inside the request parsing loop */ char *cur; /* current parsing offset */ char auth_str[] = "Authorization"; int auth_str_len = strlen(auth_str); if (strncmp(cur, auth_str, auth_str_len)==0){ shred_enter(AUTH_PASSWD_POOL); /* object holding passwd in spool */ data_string *ds = s_ds_init(); int pw_len = get_passwd_length(cur); cur += auth_str_len + 1; buffer_copy_string_len(ds->key, auth_str, auth_str_len); buffer_copy_string_len(ds->value, cur, pw_len); /* add ds to header pointer array */ array_insert_unique(parsed_headers, ds); /* only related shreds can deref ds */ /* wipe out passwd from input stream */ memset(cur, 0, pw_len); cur += pw_len; shred_exit(); } Parse authorization request.

Enter authorization shred. int http_request_parse(server *srv, connection *con) { … /* inside the request parsing loop */ char *cur; /* current parsing offset */ char auth_str[] = "Authorization"; int auth_str_len = strlen(auth_str); if (strncmp(cur, auth_str, auth_str_len)==0){ shred_enter(AUTH_PASSWD_POOL); /* object holding passwd in spool */ data_string *ds = s_ds_init(); int pw_len = get_passwd_length(cur); cur += auth_str_len + 1; buffer_copy_string_len(ds->key, auth_str, auth_str_len); buffer_copy_string_len(ds->value, cur, pw_len); /* add ds to header pointer array */ array_insert_unique(parsed_headers, ds); /* only related shreds can deref ds */ /* wipe out passwd from input stream */ memset(cur, 0, pw_len); cur += pw_len; shred_exit(); } Enter authorization shred.

Allocate a new string in the s-pool. int http_request_parse(server *srv, connection *con) { … /* inside the request parsing loop */ char *cur; /* current parsing offset */ char auth_str[] = "Authorization"; int auth_str_len = strlen(auth_str); if (strncmp(cur, auth_str, auth_str_len)==0){ shred_enter(AUTH_PASSWD_POOL); /* object holding passwd in spool */ data_string *ds = s_ds_init(); int pw_len = get_passwd_length(cur); cur += auth_str_len + 1; buffer_copy_string_len(ds->key, auth_str, auth_str_len); buffer_copy_string_len(ds->value, cur, pw_len); /* add ds to header pointer array */ array_insert_unique(parsed_headers, ds); /* only related shreds can deref ds */ /* wipe out passwd from input stream */ memset(cur, 0, pw_len); cur += pw_len; shred_exit(); } Allocate a new string in the s-pool.

Move input to the password data. int http_request_parse(server *srv, connection *con) { … /* inside the request parsing loop */ char *cur; /* current parsing offset */ char auth_str[] = "Authorization"; int auth_str_len = strlen(auth_str); if (strncmp(cur, auth_str, auth_str_len)==0){ shred_enter(AUTH_PASSWD_POOL); /* object holding passwd in spool */ data_string *ds = s_ds_init(); int pw_len = get_passwd_length(cur); cur += auth_str_len + 1; buffer_copy_string_len(ds->key, auth_str, auth_str_len); buffer_copy_string_len(ds->value, cur, pw_len); /* add ds to header pointer array */ array_insert_unique(parsed_headers, ds); /* only related shreds can deref ds */ /* wipe out passwd from input stream */ memset(cur, 0, pw_len); cur += pw_len; shred_exit(); } Move input to the password data.

Copy the password into the s-pool int http_request_parse(server *srv, connection *con) { … /* inside the request parsing loop */ char *cur; /* current parsing offset */ char auth_str[] = "Authorization"; int auth_str_len = strlen(auth_str); if (strncmp(cur, auth_str, auth_str_len)==0){ shred_enter(AUTH_PASSWD_POOL); /* object holding passwd in spool */ data_string *ds = s_ds_init(); int pw_len = get_passwd_length(cur); cur += auth_str_len + 1; buffer_copy_string_len(ds->key, auth_str, auth_str_len); buffer_copy_string_len(ds->value, cur, pw_len); /* add ds to header pointer array */ array_insert_unique(parsed_headers, ds); /* only related shreds can deref ds */ /* wipe out passwd from input stream */ memset(cur, 0, pw_len); cur += pw_len; shred_exit(); } Copy the password into the s-pool

Mark the authorization header as parsed. int http_request_parse(server *srv, connection *con) { … /* inside the request parsing loop */ char *cur; /* current parsing offset */ char auth_str[] = "Authorization"; int auth_str_len = strlen(auth_str); if (strncmp(cur, auth_str, auth_str_len)==0){ shred_enter(AUTH_PASSWD_POOL); /* object holding passwd in spool */ data_string *ds = s_ds_init(); int pw_len = get_passwd_length(cur); cur += auth_str_len + 1; buffer_copy_string_len(ds->key, auth_str, auth_str_len); buffer_copy_string_len(ds->value, cur, pw_len); /* add ds to header pointer array */ array_insert_unique(parsed_headers, ds); /* only related shreds can deref ds */ /* wipe out passwd from input stream */ memset(cur, 0, pw_len); cur += pw_len; shred_exit(); } Mark the authorization header as parsed.

Delete the password int http_request_parse(server *srv, connection *con) { … /* inside the request parsing loop */ char *cur; /* current parsing offset */ char auth_str[] = "Authorization"; int auth_str_len = strlen(auth_str); if (strncmp(cur, auth_str, auth_str_len)==0){ shred_enter(AUTH_PASSWD_POOL); /* object holding passwd in spool */ data_string *ds = s_ds_init(); int pw_len = get_passwd_length(cur); cur += auth_str_len + 1; buffer_copy_string_len(ds->key, auth_str, auth_str_len); buffer_copy_string_len(ds->value, cur, pw_len); /* add ds to header pointer array */ array_insert_unique(parsed_headers, ds); /* only related shreds can deref ds */ /* wipe out passwd from input stream */ memset(cur, 0, pw_len); cur += pw_len; shred_exit(); } Delete the password

Advance the input data. int http_request_parse(server *srv, connection *con) { … /* inside the request parsing loop */ char *cur; /* current parsing offset */ char auth_str[] = "Authorization"; int auth_str_len = strlen(auth_str); if (strncmp(cur, auth_str, auth_str_len)==0){ shred_enter(AUTH_PASSWD_POOL); /* object holding passwd in spool */ data_string *ds = s_ds_init(); int pw_len = get_passwd_length(cur); cur += auth_str_len + 1; buffer_copy_string_len(ds->key, auth_str, auth_str_len); buffer_copy_string_len(ds->value, cur, pw_len); /* add ds to header pointer array */ array_insert_unique(parsed_headers, ds); /* only related shreds can deref ds */ /* wipe out passwd from input stream */ memset(cur, 0, pw_len); cur += pw_len; shred_exit(); } Advance the input data.

Exit the shred. int http_request_parse(server *srv, connection *con) { … /* inside the request parsing loop */ char *cur; /* current parsing offset */ char auth_str[] = "Authorization"; int auth_str_len = strlen(auth_str); if (strncmp(cur, auth_str, auth_str_len)==0){ shred_enter(AUTH_PASSWD_POOL); /* object holding passwd in spool */ data_string *ds = s_ds_init(); int pw_len = get_passwd_length(cur); cur += auth_str_len + 1; buffer_copy_string_len(ds->key, auth_str, auth_str_len); buffer_copy_string_len(ds->value, cur, pw_len); /* add ds to header pointer array */ array_insert_unique(parsed_headers, ds); /* only related shreds can deref ds */ /* wipe out passwd from input stream */ memset(cur, 0, pw_len); cur += pw_len; shred_exit(); } Exit the shred.

Switch performance Comparing switch times Shred exit Intra-process thread switch Process switch How does this compare to lwCs? Which approach is better? When?

Course administration Mid-term exam One week from today Will cover everything through today Project 2 Will be released later today (due after the mid-term) Research Project Suggestions have been posted