1. Shreds: Fine-grained Execution Units with Private Memory
Landon Cox
February 21, 2018

2. Parts of a process Threads Address space File descriptors
Sequences of executing instructions
Active: do things
Address space
Data the process uses as it runs
Passive: acted upon by threads
File descriptors
Communication channels (e.g., files, sockets, pipes)
Passive: read-from/written-to by threads

3. Parts of a process
Process
Thread 1
Thread 2
Page table
File descriptors
Stack 1
Stack 2
Readable pages
Writable pages
TCP
socket
Open file
Each thread has its own stack, but everything else is share: same pages, file descriptors

4. Parts of a process Threads are not well isolated from each other.
Page table
File descriptors
Stack 1
Stack 2
Readable pages
Writable pages
TCP
socket
Open file
Threads can run in parallel on different cores.

5. Parts of a process (revised)
Thread 1
Thread 2
Page table
File descriptors
Stack 1
Stack 2
Readable pages
Writable pages
TCP
socket
Open file
If one thread fails, entire process fails.

6. Parts of a process (revised)
Thread 2
Page table
File descriptors
Stack 1
Stack 2
Readable pages
Writable pages
TCP
socket
Open file
Compromising a thread, compromises all data.

7. Parts of a process (revised)
Thread 2
Stack 1
Stack 2
Readable pages
Writable pages
TCP
socket
Open file
Compromising a thread, compromises all data.

8. Parts of a process (revised)
Thread 2
Stack 1
Stack 2
Readable pages
Writable pages
TCP
socket
Open file
See: heartbleed, cloudbleed, etc.

9. Heartbleed

10. Threads vs shreds How do you define a thread’s work?
Pass in a starting method
The thread exits when the starting function exits
What part of the address space can a thread access?
A thread can access any read- or write-enabled page
All threads are bound by the same protections
A shred’s work can be more or less than a method
Enter a shred using: shred_enter
Exit a shred using: shred_exit
A shred can access secrets in a private memory pool (s-pool)

11. Shred protection guarantees
Exclusive access to an s-pool
Only associated shreds can access data in an s-pool
Other threads, unassociated shreds cannot access
No leaks
S-pool data cannot be copied w/o sanitization
Use static analysis to identify outgoing data flows
Untampered execution
Shred code cannot be changed or routed elsewhere
i.e., control-flow integrity w/i shred (including libs)
S-driver
S-compiler

12. Shred protection guarantees
Exclusive access to an s-pool
Only associated shreds can access data in an s-pool
Other threads, unassociated shreds cannot access
S-driver

13. S-driver ARM memory domains Three access levels specified in DACR
Provide memory-access control independent of page protections
4-bit tags specified in top-level page tables
Access level for each domain set via per-core register (DACR)
MMU checks domain access for virtual memory address
Three access levels specified in DACR
No-access mode: all accesses to domain memory fail
Manager mode: all access to domain memory are allowed
Client mode: all accesses obey page protections

14. 4-bit flag in top-level entry
Two-level table
4-bit flag in top-level entry 1 …
1023 ?
Level 1
Domain
Domain
Domain
NULL
NULL
Level 2
0: PhysPage, Res, Prot
0: PhysPage, Res, Prot
1: PhysPage, Res, Prot
1: PhysPage, Res, Prot … …
1023: PhysPage, Res, Prot ?
1023: PhysPage, Res, Prot ?

15. All memory under domain 0
Two-level table
All memory under domain 0 1 …
1023 ?
Level 1
Domain
Domain
Domain
NULL
NULL
Level 2
0: PhysPage, Res, Prot
0: PhysPage, Res, Prot
1: PhysPage, Res, Prot
1: PhysPage, Res, Prot … …
1023: PhysPage, Res, Prot ?
1023: PhysPage, Res, Prot ?

16. Two-level table … … ? ? 1 … 1023 ? Level 1 Level 2
Top 4MB now in domain 1 1 …
1023 ?
Level 1
Domain
Domain
Domain 1
NULL
NULL
Level 2
0: PhysPage, Res, Prot
0: PhysPage, Res, Prot
1: PhysPage, Res, Prot
1: PhysPage, Res, Prot … …
1023: PhysPage, Res, Prot ?
1023: PhysPage, Res, Prot ?

17. Can put other regions under other domains
Two-level table
Can put other regions under other domains 1 …
1023 ?
Level 1
Domain 2
Domain
Domain 1
NULL
NULL
Level 2
0: PhysPage, Res, Prot
0: PhysPage, Res, Prot
1: PhysPage, Res, Prot
1: PhysPage, Res, Prot … …
1023: PhysPage, Res, Prot ?
1023: PhysPage, Res, Prot ?

18. Each core has a 32-bit DACR
Two-level table 1 …
1023 ?
Level 1
Domain 2
Domain
Domain 1
NULL
NULL
Level 2
0: PhysPage, Res, Prot
0: PhysPage, Res, Prot
1: PhysPage, Res, Prot
1: PhysPage, Res, Prot … …
1023: PhysPage, Res, Prot ?
1023: PhysPage, Res, Prot ?
Each core has a 32-bit DACR

19. Two-level table … … ? ? 1 … 1023 ? Level 1 Level 21 …
1023 ?
Level 1
Domain 2
Domain
Domain 1
NULL
NULL
Level 2
0: PhysPage, Res, Prot
0: PhysPage, Res, Prot
1: PhysPage, Res, Prot
1: PhysPage, Res, Prot … …
1023: PhysPage, Res, Prot ?
1023: PhysPage, Res, Prot ?
16 domains x 2 mode bits

20. Mode bits encode client, no-access, manager modes
Two-level table 1 …
1023 ?
Level 1
Domain 2
Domain
Domain 1
NULL
NULL
Level 2
0: PhysPage, Res, Prot
0: PhysPage, Res, Prot
1: PhysPage, Res, Prot
1: PhysPage, Res, Prot … …
1023: PhysPage, Res, Prot ?
1023: PhysPage, Res, Prot ?
Mode bits encode client, no-access, manager modes

21. Note changing DACR does not require TLB flush!
Two-level table 1 …
1023 ?
Level 1
Domain 2
Domain
Domain 1
NULL
NULL
Level 2
0: PhysPage, Res, Prot
0: PhysPage, Res, Prot
1: PhysPage, Res, Prot
1: PhysPage, Res, Prot … …
1023: PhysPage, Res, Prot ?
1023: PhysPage, Res, Prot ?
Note changing DACR does not require TLB flush!

22. Mode change should be faster than page change
Two-level table 1 …
1023 ?
Level 1
Domain 2
Domain
Domain 1
NULL
NULL
Level 2
0: PhysPage, Res, Prot
0: PhysPage, Res, Prot
1: PhysPage, Res, Prot
1: PhysPage, Res, Prot … …
1023: PhysPage, Res, Prot ?
1023: PhysPage, Res, Prot ?
Mode change should be faster than page change

23. Problem with only 16 domains?
Two-level table
Problem with only 16 domains? 1 …
1023 ?
Level 1
Domain 2
Domain
Domain 1
NULL
NULL
Level 2
0: PhysPage, Res, Prot
0: PhysPage, Res, Prot
1: PhysPage, Res, Prot
1: PhysPage, Res, Prot … …
1023: PhysPage, Res, Prot ?
1023: PhysPage, Res, Prot ?

24. Every s-pool is assigned a memory domain (e.g., 3-7)
Linux
No access
Page permissions
Unused 15 14 … 10 9 8 7 6 5 4 3 2 1
Every s-pool is assigned a memory domain (e.g., 3-7)

25. Assignments change depending on if shred is active and on which core.
Linux
No access
Page permissions
Unused 15 14 … 10 9 8 7 6 5 4 3 2 1
Assignments change depending on if shred is active and on which core.

26. Each core gets an exclusive domain
Linux
No access
Page permissions
Unused 15 14 … 10 9 8 7 6 5 4 3 2 1
Each core gets an exclusive domain

27. All inactive shreds’ s-pools are tagged w/ red domain.15 14 … 10 9 8 7 6 5 4 3 2 1
While shred is active on core, its s-pool is tagged w/ core’s green domain.

28. Say we load shred 0 onto core 0.15 14 … 10 9 8 7 6 5 4 3 2 1 S0

29. Which domain does S0’s s-pool need to be tagged with?15 14 … 10 9 8 7 6 5 4 3 2 1 S0

30. Why will S0 be able to access its s-pool?15 14 … 10 9 8 7 6 5 4 3 2 1 S0

31. Why can’t other shreds access
S0’s s-pool? 15 14 … 10 9 8 7 6 5 4 3 2 1 S0

32. Can S0 run in parallel on > 1 core?15 14 … 10 9 8 7 6 5 4 3 2 1 S0

33. How do we tag S0’s s-pool when it’s done? Why?15 14 … 10 9 8 7 6 5 4 3 2 1 S0

34. What happens to the TLB on when an
s-pool’s tag changes? 15 14 … 10 9 8 7 6 5 4 3 2 1 S0

35. Where does a shred’s stack have to reside? Why?15 14 … 10 9 8 7 6 5 4 3 2 1 S0

36. int http_request_parse(server *srv,
connection *con) { …
/* inside the request parsing loop */
char *cur; /* current parsing offset */
char auth_str[] = "Authorization";
int auth_str_len = strlen(auth_str);
if (strncmp(cur, auth_str, auth_str_len)==0){
shred_enter(AUTH_PASSWD_POOL);
/* object holding passwd in spool */
data_string *ds = s_ds_init();
int pw_len = get_passwd_length(cur);
cur += auth_str_len + 1;
buffer_copy_string_len(ds->key, auth_str, auth_str_len);
buffer_copy_string_len(ds->value, cur, pw_len);
/* add ds to header pointer array */
array_insert_unique(parsed_headers, ds);
/* only related shreds can deref ds */
/* wipe out passwd from input stream */
memset(cur, 0, pw_len);
cur += pw_len;
shred_exit(); }

37. Parse authorization request.
int http_request_parse(server *srv,
connection *con) { …
/* inside the request parsing loop */
char *cur; /* current parsing offset */
char auth_str[] = "Authorization";
int auth_str_len = strlen(auth_str);
if (strncmp(cur, auth_str, auth_str_len)==0){
shred_enter(AUTH_PASSWD_POOL);
/* object holding passwd in spool */
data_string *ds = s_ds_init();
int pw_len = get_passwd_length(cur);
cur += auth_str_len + 1;
buffer_copy_string_len(ds->key, auth_str, auth_str_len);
buffer_copy_string_len(ds->value, cur, pw_len);
/* add ds to header pointer array */
array_insert_unique(parsed_headers, ds);
/* only related shreds can deref ds */
/* wipe out passwd from input stream */
memset(cur, 0, pw_len);
cur += pw_len;
shred_exit(); }
Parse authorization request.

38. Enter authorization shred.
int http_request_parse(server *srv,
connection *con) { …
/* inside the request parsing loop */
char *cur; /* current parsing offset */
char auth_str[] = "Authorization";
int auth_str_len = strlen(auth_str);
if (strncmp(cur, auth_str, auth_str_len)==0){
shred_enter(AUTH_PASSWD_POOL);
/* object holding passwd in spool */
data_string *ds = s_ds_init();
int pw_len = get_passwd_length(cur);
cur += auth_str_len + 1;
buffer_copy_string_len(ds->key, auth_str, auth_str_len);
buffer_copy_string_len(ds->value, cur, pw_len);
/* add ds to header pointer array */
array_insert_unique(parsed_headers, ds);
/* only related shreds can deref ds */
/* wipe out passwd from input stream */
memset(cur, 0, pw_len);
cur += pw_len;
shred_exit(); }
Enter authorization shred.

39. Allocate a new string in the s-pool.
int http_request_parse(server *srv,
connection *con) { …
/* inside the request parsing loop */
char *cur; /* current parsing offset */
char auth_str[] = "Authorization";
int auth_str_len = strlen(auth_str);
if (strncmp(cur, auth_str, auth_str_len)==0){
shred_enter(AUTH_PASSWD_POOL);
/* object holding passwd in spool */
data_string *ds = s_ds_init();
int pw_len = get_passwd_length(cur);
cur += auth_str_len + 1;
buffer_copy_string_len(ds->key, auth_str, auth_str_len);
buffer_copy_string_len(ds->value, cur, pw_len);
/* add ds to header pointer array */
array_insert_unique(parsed_headers, ds);
/* only related shreds can deref ds */
/* wipe out passwd from input stream */
memset(cur, 0, pw_len);
cur += pw_len;
shred_exit(); }
Allocate a new string in the s-pool.

40. Move input to the password data.
int http_request_parse(server *srv,
connection *con) { …
/* inside the request parsing loop */
char *cur; /* current parsing offset */
char auth_str[] = "Authorization";
int auth_str_len = strlen(auth_str);
if (strncmp(cur, auth_str, auth_str_len)==0){
shred_enter(AUTH_PASSWD_POOL);
/* object holding passwd in spool */
data_string *ds = s_ds_init();
int pw_len = get_passwd_length(cur);
cur += auth_str_len + 1;
buffer_copy_string_len(ds->key, auth_str, auth_str_len);
buffer_copy_string_len(ds->value, cur, pw_len);
/* add ds to header pointer array */
array_insert_unique(parsed_headers, ds);
/* only related shreds can deref ds */
/* wipe out passwd from input stream */
memset(cur, 0, pw_len);
cur += pw_len;
shred_exit(); }
Move input to the password data.

41. Copy the password into the s-pool
int http_request_parse(server *srv,
connection *con) { …
/* inside the request parsing loop */
char *cur; /* current parsing offset */
char auth_str[] = "Authorization";
int auth_str_len = strlen(auth_str);
if (strncmp(cur, auth_str, auth_str_len)==0){
shred_enter(AUTH_PASSWD_POOL);
/* object holding passwd in spool */
data_string *ds = s_ds_init();
int pw_len = get_passwd_length(cur);
cur += auth_str_len + 1;
buffer_copy_string_len(ds->key, auth_str, auth_str_len);
buffer_copy_string_len(ds->value, cur, pw_len);
/* add ds to header pointer array */
array_insert_unique(parsed_headers, ds);
/* only related shreds can deref ds */
/* wipe out passwd from input stream */
memset(cur, 0, pw_len);
cur += pw_len;
shred_exit(); }
Copy the password into the s-pool

42. Mark the authorization header as parsed.
int http_request_parse(server *srv,
connection *con) { …
/* inside the request parsing loop */
char *cur; /* current parsing offset */
char auth_str[] = "Authorization";
int auth_str_len = strlen(auth_str);
if (strncmp(cur, auth_str, auth_str_len)==0){
shred_enter(AUTH_PASSWD_POOL);
/* object holding passwd in spool */
data_string *ds = s_ds_init();
int pw_len = get_passwd_length(cur);
cur += auth_str_len + 1;
buffer_copy_string_len(ds->key, auth_str, auth_str_len);
buffer_copy_string_len(ds->value, cur, pw_len);
/* add ds to header pointer array */
array_insert_unique(parsed_headers, ds);
/* only related shreds can deref ds */
/* wipe out passwd from input stream */
memset(cur, 0, pw_len);
cur += pw_len;
shred_exit(); }
Mark the authorization header as parsed.

43. Delete the password int http_request_parse(server *srv,
connection *con) { …
/* inside the request parsing loop */
char *cur; /* current parsing offset */
char auth_str[] = "Authorization";
int auth_str_len = strlen(auth_str);
if (strncmp(cur, auth_str, auth_str_len)==0){
shred_enter(AUTH_PASSWD_POOL);
/* object holding passwd in spool */
data_string *ds = s_ds_init();
int pw_len = get_passwd_length(cur);
cur += auth_str_len + 1;
buffer_copy_string_len(ds->key, auth_str, auth_str_len);
buffer_copy_string_len(ds->value, cur, pw_len);
/* add ds to header pointer array */
array_insert_unique(parsed_headers, ds);
/* only related shreds can deref ds */
/* wipe out passwd from input stream */
memset(cur, 0, pw_len);
cur += pw_len;
shred_exit(); }
Delete the password

44. Advance the input data. int http_request_parse(server *srv,
connection *con) { …
/* inside the request parsing loop */
char *cur; /* current parsing offset */
char auth_str[] = "Authorization";
int auth_str_len = strlen(auth_str);
if (strncmp(cur, auth_str, auth_str_len)==0){
shred_enter(AUTH_PASSWD_POOL);
/* object holding passwd in spool */
data_string *ds = s_ds_init();
int pw_len = get_passwd_length(cur);
cur += auth_str_len + 1;
buffer_copy_string_len(ds->key, auth_str, auth_str_len);
buffer_copy_string_len(ds->value, cur, pw_len);
/* add ds to header pointer array */
array_insert_unique(parsed_headers, ds);
/* only related shreds can deref ds */
/* wipe out passwd from input stream */
memset(cur, 0, pw_len);
cur += pw_len;
shred_exit(); }
Advance the input data.

45. Exit the shred. int http_request_parse(server *srv, connection *con) {…
/* inside the request parsing loop */
char *cur; /* current parsing offset */
char auth_str[] = "Authorization";
int auth_str_len = strlen(auth_str);
if (strncmp(cur, auth_str, auth_str_len)==0){
shred_enter(AUTH_PASSWD_POOL);
/* object holding passwd in spool */
data_string *ds = s_ds_init();
int pw_len = get_passwd_length(cur);
cur += auth_str_len + 1;
buffer_copy_string_len(ds->key, auth_str, auth_str_len);
buffer_copy_string_len(ds->value, cur, pw_len);
/* add ds to header pointer array */
array_insert_unique(parsed_headers, ds);
/* only related shreds can deref ds */
/* wipe out passwd from input stream */
memset(cur, 0, pw_len);
cur += pw_len;
shred_exit(); }
Exit the shred.

46. Switch performance Comparing switch times
Shred exit
Intra-process thread switch
Process switch
How does this compare to lwCs?
Which approach is better? When?

47. Course administration
Mid-term exam
One week from today
Will cover everything through today
Project 2
Will be released later today (due after the mid-term)
Research Project
Suggestions have been posted