1. Policies and Mechanisms for Operating System Security
Vinod Ganapathy
Associate Professor of Computer Science
Rutgers, The State University of New Jersey

2. Layered computer system design
Modern computer systems are built using layers of abstraction
Memory
I/O devices
CPU
Hardware
Vinod Ganapathy - Policies and Mechanisms for OS Security

3. Layered computer system design
Modern computer systems are built using layers of abstraction
Operating
System
Syscalls
IDT
Kernel
Code
Process
List …
Memory
I/O devices
CPU
Hardware
Vinod Ganapathy - Policies and Mechanisms for OS Security

4. Layered computer system design
Modern computer systems are built using layers of abstraction
Utilities &
Libraries
ls, ps, &
bash utilities
libc
gcc …
Operating
System
Syscalls
IDT
Kernel
Code
Process
List …
Memory
I/O devices
CPU
Hardware
Vinod Ganapathy - Policies and Mechanisms for OS Security

5. Layered computer system design
Modern computer systems are built using layers of abstraction
User
app
User
app …
Utilities &
Libraries
ls, ps, &
bash utilities
libc
gcc …
Operating
System
Syscalls
IDT
Kernel
Code
Process
List …
Memory
I/O devices
CPU
Hardware
Vinod Ganapathy - Policies and Mechanisms for OS Security

6. Fundamental principle in security
The lower you go, the more control you have
User
app
User
app
Least control …
Utilities &
Libraries
ls, ps, &
bash utilities
libc
gcc …
Operating
System
Syscalls
IDT
Kernel
Code
Process
List …
Memory
I/O devices
CPU
Hardware
Most control
Vinod Ganapathy - Policies and Mechanisms for OS Security

7. Example: Malware detection
User
app
Utilities &
Libraries
Operating
System
Hardware
Vinod Ganapathy - Policies and Mechanisms for OS Security

8. Example: Malware detection
User
app
Malware detector
Utilities &
Libraries
Operating
System
Hardware
Vinod Ganapathy - Policies and Mechanisms for OS Security

9. Example: Malware detection
User
app
Malware detector
Trusted
Layer
Utilities &
Libraries …
TCB
cat ps ls
Operating
System
Hardware
Vinod Ganapathy - Policies and Mechanisms for OS Security

10. But utilities may be compromised!
User
app
Malware detector
Utilities &
Libraries
cat ps ls
Operating
System
Hardware
Vinod Ganapathy - Policies and Mechanisms for OS Security

11. But utilities may be compromised!
Show me
file contents 1
User
app
Malware detector 1
Utilities &
Libraries
cat ps ls
Operating
System
Hardware
Vinod Ganapathy - Policies and Mechanisms for OS Security

12. But utilities may be compromised!
Show me
file contents 1 2
Fake, benign
content
User
app
Malware detector 2
Utilities &
Libraries
cat ps ls
Operating
System
Hardware
Vinod Ganapathy - Policies and Mechanisms for OS Security

13. Vinod Ganapathy - Policies and Mechanisms for OS Security
Solution: Query the OS
Query with syscall 1
User
app
Malware detector
Utilities &
Libraries 1
Operating
System
System call API
TCB
Hardware
Vinod Ganapathy - Policies and Mechanisms for OS Security

14. Vinod Ganapathy - Policies and Mechanisms for OS Security
Solution: Query the OS
Query with syscall 1 2
OS reads file
User
app
Malware detector
Utilities &
Libraries 2
Operating
System
System call API
TCB
Hardware
Vinod Ganapathy - Policies and Mechanisms for OS Security

15. Vinod Ganapathy - Policies and Mechanisms for OS Security
Solution: Query the OS
Query with syscall 1 2
OS reads file
User
app
Malware detector 3
Returns true
file content
Utilities &
Libraries 3
Operating
System
System call API
TCB
Hardware
Vinod Ganapathy - Policies and Mechanisms for OS Security

16. OS detects malicious utilities too
cat file B
Read file
User
app
Malware detector A B
diff vs ?
Utilities &
Libraries A
cat B
Operating
System
System call API
TCB
Hardware
Vinod Ganapathy - Policies and Mechanisms for OS Security

17. What if the OS is malicious?
User
app
Malware detector
Utilities &
Libraries
Operating
System
System call API
Is it game over?
Hardware
Vinod Ganapathy - Policies and Mechanisms for OS Security

18. Rootkit = Malware that infects OS
Rootkits hide malware from detectors  Long-term stealth
Malware detector …
Utilities &
Libraries
Operating
System
System call API
Is it game over?
Hardware
Vinod Ganapathy - Policies and Mechanisms for OS Security

19. How does an OS get infected?
Exploits of kernel vulnerabilities:
Injecting malicious code by exploiting a memory error in the kernel
Privilege escalation attacks:
Exploit a root process and use resulting administrative privileges to update the kernel
Social engineering attacks:
Trick user into installing fake kernel updates
Defeated via signature verification of kernel updates
Trivial to perform prior to the Windows Vista OS
Vinod Ganapathy - Policies and Mechanisms for OS Security

20. How prevalent are rootkits?
2010 Microsoft report: 7% of all infections from client machines due to rootkits[1]
2016 HummingBad Android rootkit:[2]
Up to 85 million Android devices infected?
Earns malware authors $300,000 each week through fraudulent mobile advertisements
Used in many high-profile incidents:
Torpig and Storm botnets
Sony BMG (2005), Greek wiretapping (2004/5)
[1]
Microsoft Malware Protection Center, “Some Observations on Rootkits,” January 2010,
[2]
CheckPoint Software, “From HummingBad to Worse,” July 2016,

21. How can we detect rootkits?
Ask for help from the layers below
User
app
Malware detector
Utilities &
Libraries
Operating
System
System call API
Is it game over?
TCB
Hypervisor (a.k.a. Virtual Machine Monitor)
Hardware
Vinod Ganapathy - Policies and Mechanisms for OS Security

22. Vinod Ganapathy - Policies and Mechanisms for OS Security
How low can we go?
User
app
Malware detector
Utilities &
Libraries
Operating
System
Is it game over?
Hypervisor
[Bluepill, Subvert]
TCB
Hardware
Vinod Ganapathy - Policies and Mechanisms for OS Security

23. Vinod Ganapathy - Policies and Mechanisms for OS Security
How low can we go?
User
app
Malware detector
Utilities &
Libraries
Operating
System
Is it game over?
Hardware
???
[Stuxnet, Trojaned ICs]
TCB
Vinod Ganapathy - Policies and Mechanisms for OS Security

24. Vinod Ganapathy - Policies and Mechanisms for OS Security
Today’s talk
User
apps
Detecting OS-level rootkit infections (with some help from the hardware)
In two parts:
Policies: How do we know that the OS is infected?
Mechanisms: How can the hardware help us?
Utilities & Libraries
Operating System
Hardware
TCB
Same methods can be used to detect hypervisor rootkits too.
Vinod Ganapathy - Policies and Mechanisms for OS Security

25. My contributions to OS security
2008
----
Detecting rootkits using data structure invariants [ACSAC’08]
Re-architecting device drivers for better isolation [ASPLOS’08]
2009
Securing OSes from malicious device drivers [ACSAC’09]
2010
Exploring rootkits on smartphones [HotMobile’10]
2011
Security/energy tradeoffs in rootkit detection [ACM MobiSys’11]
2012
Rootkit detection on cloud platforms [ACM CCS’12]
2013
Adapting multicore hardware for rootkit detection [TIFS’13]
2016
Rootkit detection with ARM TrustZone [ACM MobiSys’16]
Rootkit detection using 3D-stacked hardware [Submitted]
Vinod Ganapathy - Policies and Mechanisms for OS Security

26. Covered in today’s talk
2008
----
Detecting rootkits using data structure invariants [ACSAC’08]
Re-architecting device drivers for better isolation [ASPLOS’08]
2009
Securing OSes from malicious device drivers [ACSAC’09]
2010
Exploring rootkits on smartphones [HotMobile’10]
2011
Security/energy tradeoffs in rootkit detection [ACM MobiSys’11]
2012
Rootkit detection on cloud platforms [ACM CCS’12]
2013
Adapting multicore hardware for rootkit detection [TIFS’13]
2016
Rootkit detection with ARM TrustZone [ACM MobiSys’16]
Rootkit detection using 3D-stacked hardware [Submitted]
Vinod Ganapathy - Policies and Mechanisms for OS Security

27. Modus operandi
Analysis of memory snapshots obtained from target machine
Target machine
Potentially rootkit infected
User app
User app …
Utilities &
Libraries
Operating
System
Kernel
Code
Process
List
Syscall
Hardware
Physical Memory
TCB
Vinod Ganapathy - Policies and Mechanisms for OS Security

28. Modus operandi
Analysis of memory snapshots obtained from target machine
Target machine
Potentially rootkit infected
Analysis machine
Trusted
User app
User app …
Utilities &
Libraries
Operating
System
Kernel
Code
Process
List
Syscall
Hardware
Physical Memory
TCB
Vinod Ganapathy - Policies and Mechanisms for OS Security

29. Modus operandi
Analysis of memory snapshots obtained from target machine
Target machine
Potentially rootkit infected
Analysis machine
Trusted
User app
User app …
Utilities &
Libraries
Operating
System
Kernel
Code
Process
List
Syscall
Snapshot of
memory pages
Hardware …
Physical Memory
TCB
Vinod Ganapathy - Policies and Mechanisms for OS Security

30. Vinod Ganapathy - Policies and Mechanisms for OS Security
Research questions
RQ1: What algorithm should we use for memory snapshot analysis?
Concerns our security policy
Answer: Formulate rootkit detection problem as one of detecting invariant violations
RQ2: How can we fetch memory pages without involving the target’s OS?
Concerns our mechanism
Answer: Leverage hardware advances
Vinod Ganapathy - Policies and Mechanisms for OS Security

31. Vinod Ganapathy - Policies and Mechanisms for OS Security
Research questions
RQ1: What algorithm should we use for memory snapshot analysis?
Concerns our security policy
Answer: Formulate rootkit detection problem as one of detecting invariant violations
RQ2: How can we fetch memory pages without involving the target’s OS?
Concerns our mechanism
Answer: Leverage hardware advances
Vinod Ganapathy - Policies and Mechanisms for OS Security

32. Example 1: Linux Adore rootkit
sys_open(...) {
... }
int main() {
open(…)
...
return(0) }
sys_open
Mention that attacks are not only for control data, even for non-control data.
System call table
User app
OS kernel
Vinod Ganapathy - Policies and Mechanisms for OS Security 32

33. Example 1: Linux Adore rootkit
sys_open(...) {
... }
int main() {
open(…)
...
return(0) }
evil_open
evil_open(...) {
malicious();
sys_open(...) }
Mention that attacks are not only for control data, even for non-control data.
System call table
User app
OS kernel
Vinod Ganapathy - Policies and Mechanisms for OS Security 33

34. Example 1: Linux Adore rootkit
Violated: Function pointer values in
system call table should not change
sys_open(...) {
... }
int main() {
open(…)
...
return(0) }
evil_open
evil_open(...) {
malicious();
sys_open(...) }
Mention that attacks are not only for control data, even for non-control data.
System call table
User app
OS kernel
Vinod Ganapathy - Policies and Mechanisms for OS Security 34

35. Example 2: Windows Fu rootkit
run-list: Used by the scheduler to select processes for execution
Process A
Process B
Process C
run_list
run_list
run_list
next_task
next_task
next_task
all-tasks: Used for process accounting
Vinod Ganapathy - Policies and Mechanisms for OS Security

36. Example 2: Windows Fu rootkit
run-list: Used by the scheduler to select processes for execution
Process A
Hidden process
Process B
Process C
run_list
run_list
run_list
run_list
next_task
next_task
next_task
next_task
all-tasks: Used for process accounting
Vinod Ganapathy - Policies and Mechanisms for OS Security

37. Example 2: Windows Fu rootkit
Violated: run-list ⊆ all-tasks
run-list: Used by the scheduler to select processes for execution
Process A
Hidden process
Process B
Process C
run_list
run_list
run_list
run_list
next_task
next_task
next_task
next_task
all-tasks: Used for process accounting
Vinod Ganapathy - Policies and Mechanisms for OS Security

38. Example 3: Kernel PRNG corruptor
Secondary
Entropy Pool
(128 bytes)
/dev/random
External Entropy Sources
Primary
Entropy Pool
(512 bytes)
Look up tcp syn attack – how generating poor sequence numbers can put the system at risk.
Urandom
Entropy Pool
(128 bytes)
/dev/urandom
Vinod Ganapathy - Policies and Mechanisms for OS Security

39. Example 3: Kernel PRNG corruptor
Attack: Modify coefficients of polynomials used to stir the entropy pools. Weaken quality of random numbers
Secondary
Entropy Pool
(128 bytes)
/dev/random
External Entropy Sources
Primary
Entropy Pool
(512 bytes)
Look up tcp syn attack – how generating poor sequence numbers can put the system at risk.
Urandom
Entropy Pool
(128 bytes)
/dev/urandom
Vinod Ganapathy - Policies and Mechanisms for OS Security

40. Example 3: Kernel PRNG corruptor
Violated:
poolinfo.tap1 ∈ {26, 103}
poolinfo.tap2 ∈ {20, 76}
poolinfo.tap3 ∈ {14, 51}
poolinfo.tap4 ∈ {7, 25}
poolinfo.tap5 == 1
Secondary
Entropy Pool
(128 bytes)
/dev/random
External Entropy Sources
Primary
Entropy Pool
(512 bytes)
Look up tcp syn attack – how generating poor sequence numbers can put the system at risk.
Urandom
Entropy Pool
(128 bytes)
/dev/urandom
Vinod Ganapathy - Policies and Mechanisms for OS Security

41. Key technical challenges
Vast attack surface:
The kernel has thousands of data structures
Specifying correctness properties:
Infeasible to supply properties manually
Vinod Ganapathy - Policies and Mechanisms for OS Security

42. Key technical challenges
Vast attack surface:
The kernel has thousands of data structures
Solution: Use memory snapshots to analyze all kernel data structures
Specifying correctness properties:
Infeasible to supply properties manually
Solution: Infer invariants by adapting methods from dynamic program analysis
Vinod Ganapathy - Policies and Mechanisms for OS Security

43. Offline training phase
Clean reference machine
Not rootkit infected
Analysis machine
User app
User app …
Utilities &
Libraries
Operating
System
Kernel
Code
Process
List
Syscall
Snapshot of
memory pages
Hardware …
Physical Memory
TCB
Vinod Ganapathy - Policies and Mechanisms for OS Security

44. Offline training phase
Clean reference machine
Not rootkit infected
Analysis machine
User app
User app
Invariant DB …
Utilities &
Libraries
Inference
Operating
System
Kernel
Code
Process
List
Syscall
Snapshot of
memory pages
Hardware …
Physical Memory
TCB
Vinod Ganapathy - Policies and Mechanisms for OS Security

45. Online enforcement phase
Target machine
Potentially rootkit infected
Analysis machine
User app
User app
Invariant DB …
Compare
Utilities &
Libraries
Inference
Operating
System
Kernel
Code
Process
List
Syscall
Snapshot of
memory pages
Hardware …
Physical Memory
TCB
Vinod Ganapathy - Policies and Mechanisms for OS Security

46. Prior work on inferring invariants
Daikon: Dynamic program analysis tool to infer data invariants [Ernst et al., 2000]
T1, T2, … , Tn = Traces from execution of a target program, recording variable values
Values/properties invariant in in T1, T2, … , Tn
(e.g., foo == 5, foo ≤ bar + baz)
Vinod Ganapathy - Policies and Mechanisms for OS Security

47. Adapting to memory snapshots
S1, S2, … Sn = Snapshots from reference machine for (i ∈ [1 .. n]) { Di = Reconstruct kernel data structures in Si } D1, D2, … , Dn
sys_open == 0x3ee210fb
run-list ⊆ all-tasks
poolinfo.tap1 ∈ {26, 103}
poolinfo.tap2 ∈ {20, 76} …
Data Structure Invariants
Vinod Ganapathy - Policies and Mechanisms for OS Security

48. Reconstructing data structures
Kernel data structure type definitions
(2) Entry-points into the kernel
struct task_struct {...}
struct list_head {...}
struct siginfo {...}
...
ffffe400 init_task ffffe410 phys_base ffffe420 loops_per_jiffy
... …
(3) Snapshot of memory pages
Vinod Ganapathy - Policies and Mechanisms for OS Security

49. Reconstructing data structures
Kernel data structure type definitions
(2) Entry-points into the kernel
struct task_struct {...}
struct list_head {...}
struct siginfo {...}
...
ffffe400 init_task ffffe410 phys_base ffffe420 loops_per_jiffy
...
Definition of task_struct
Data at 0xffffe400
struct task_struct {
int state;
int counter;
struct task_struct *next;
... }
0034ea23
ac3456bc
... …
init_task.state = 1
init_task.counter = 0x34ea23
init_task.next = 0xac3456bc
(3) Snapshot of memory pages
Vinod Ganapathy - Policies and Mechanisms for OS Security

50. Reconstructing data structures
Kernel data structure type definitions
(2) Entry-points into the kernel
struct task_struct {...}
struct list_head {...}
struct siginfo {...}
...
ffffe400 init_task ffffe410 phys_base ffffe420 loops_per_jiffy
...
Definition of task_struct
Data at 0xac3456bc
struct task_struct {
int state;
int counter;
struct task_struct *next;
... }
0056ae71
bf6723ae
... …
init_task.next.state = 0
init_task.next.counter = 0x56ae71
init_task.next.next = 0xbf6723ae
(3) Snapshot of memory pages
Vinod Ganapathy - Policies and Mechanisms for OS Security

51. Experimental evaluation
How effective is our approach at detecting rootkits? i.e., what is the false negative rate?
What is the quality of automatically-generated invariants? i.e., what is the false positive rate?
Target machine ran Linux Used same machine as reference machine as well.
Vinod Ganapathy - Policies and Mechanisms for OS Security

52. Vinod Ganapathy - Policies and Mechanisms for OS Security
Training phase
Ran LMBench on reference machine:
Collected 15 complete memory snapshots (including reboots): took 25 minutes
Inferred invariants using Daikon in 31 minutes
Inferred 236,444 invariants across the memory snapshots
Vinod Ganapathy - Policies and Mechanisms for OS Security

53. False negative evaluation
Conducted experiments with 23 Linux rootkits:
14 rootkits from PacketStorm
9 advanced rootkits, discussed in the literature
Installed rootkits one at a time on the target machine and tested effectiveness of our approach at detecting the infection
Vinod Ganapathy - Policies and Mechanisms for OS Security

54. Data structures affected Detected?
Rootkit name
Data structures affected
Detected? 1.
Adore-0.42
System call table (from PacketStorm) 2.
All-root 3.
Kbd 4.
Kis-0.9 5.
Linspy2 6.
Modhide 7.
Phide 8.
Rial 9.
Rkit-1.01
10.
Shtroj2
11.
Synapsys-0.4
12.
THC Backdoor
13.
Adore-ng
VFS hooks/UDP recvmsg (from PacketStorm)
14.
Knark-2.4.3
System call table, proc hooks (from PacketStorm)
15.
Disable Firewall
Netfilter hooks (Baliga et al., 2007)
16.
Disable PRNG
VFS hooks (Baliga et al., 2007)
17.
Altering RTC
18.
Defeat signature scans
19.
Entropy pool
struct poolinfo (Baliga et al., 2007)
20.
Hidden process
Process lists (Petroni et al., 2006)
21.
Linux Binfmt
Shellcode.com
22.
Resource waste
struct zone_struct (Baliga et al., 2007)
23.
Intrinsic DOS
int max_threads (Baliga et al., 2007)
November 30, 2009 54

55. False positive evaluation
Ran a benign workload for 42 minutes
Copying Linux kernel source code
Editing a text document
Compiling the Linux kernel
Downloading eight videos from Internet
Perform file system operations using the IOZone benchmark
Only 82 out of 236,444 invariants spuriously violated during execution
Can be improved with more training
Vinod Ganapathy - Policies and Mechanisms for OS Security

56. Current status of this approach
Adopted widely in community for memory snapshot-based rootkit detection
Has led to numerous follow-on projects by other research groups (200+ citations)
More accurate data structure reconstruction
Better ways to express invariants
Improving accuracy of inferred invariants
Vinod Ganapathy - Policies and Mechanisms for OS Security

57. Vinod Ganapathy - Policies and Mechanisms for OS Security
Research questions
RQ1: What algorithm should we use for memory snapshot analysis?
Concerns our security policy
Answer: Formulate rootkit detection problem as one of detecting invariant violations
RQ2: How can we fetch memory pages without involving the target’s OS?
Concerns our mechanism
Answer: Leverage hardware advances
Vinod Ganapathy - Policies and Mechanisms for OS Security

58. Snapshot acquisition mechanism
Tamper
resistance
Performance isolation
Snapshot consistency 1 2 3
Vinod Ganapathy - Policies and Mechanisms for OS Security

59. Target should not interfere with snapshot acquisition
Tamper resistance
Tamper
resistance
Performance isolation
Snapshot consistency
Target should not interfere with snapshot acquisition
Vinod Ganapathy - Policies and Mechanisms for OS Security

60. Target should not interfere with snapshot acquisition
Tamper resistance
Tamper
resistance
Performance isolation
Snapshot consistency
Virtualization
Target should not interfere with snapshot acquisition
Operating
System
Hypervisor can fetch memory from virtual machine without OS involvement
Virtual Hardware
Physical Memory
Hypervisor
Vinod Ganapathy - Policies and Mechanisms for OS Security

61. Target should not interfere with snapshot acquisition
Tamper resistance
Tamper
resistance
Performance isolation
Snapshot consistency
Virtualization
Co-processor
Target should not interfere with snapshot acquisition
Operating
System
Co-processor uses DMA
OS on target involved in DMA setup
Malicious OS can hide portions of memory with malicious content
Hardware
Physical Memory
Vinod Ganapathy - Policies and Mechanisms for OS Security

62. Performance isolation
Tamper
resistance
Performance isolation
Snapshot consistency
Virtualization
Co-processor
Do not halt the target during snapshot acquisition
Necessary for situations where continuous snapshot acquisition is necessary
Hypervisor-based acquisition requires pausing the virtual machine
Co-processor can operate in concert with target
Vinod Ganapathy - Policies and Mechanisms for OS Security

63. Vinod Ganapathy - Policies and Mechanisms for OS Security
Snapshot consistency
Tamper
resistance
Performance isolation
Snapshot consistency
Virtualization
Co-processor
Snapshot should faithfully represent target’s state at a given instant in time
Operating
System
CONSISTENT … T F1 F2
Hardware
Physical Memory
CONSISTENT
NULL …
T + δ F1 F2
Vinod Ganapathy - Policies and Mechanisms for OS Security

64. Vinod Ganapathy - Policies and Mechanisms for OS Security
Snapshot consistency
Tamper
resistance
Performance isolation
Snapshot consistency
Virtualization
Co-processor
Snapshot should faithfully represent target’s state at a given instant in time
Operating
System
INCONSISTENT … F1 F2 T
T + δ
Hardware
Co-processor cannot pause target. Snapshot may contain pages obtained at different instants in time
Physical Memory
Vinod Ganapathy - Policies and Mechanisms for OS Security

65. Vinod Ganapathy - Policies and Mechanisms for OS Security
Our contribution
Tamper
resistance
Performance isolation
Snapshot consistency
Virtualization
Co-processor
3D-stacking
Based on 3D-stacked technology:
New hardware manufacturing technology that “stacks” memory/processing logic atop the chip
Early versions of 3D-stacked hardware already on market, e.g., AMD Radeon series
Vinod Ganapathy - Policies and Mechanisms for OS Security

66. Vinod Ganapathy - Policies and Mechanisms for OS Security
3D-stacked chip
On-chip memory
(high-speed)
CPU and
Memory controller
Picture courtesy of AMD
Vinod Ganapathy - Policies and Mechanisms for OS Security

67. Traditional (off-chip)
3D-stacked chip
Traditional (off-chip)
DRAM memory
On-chip memory
(high-speed)
Memory bus
CPU and
Memory controller
Design space of methods to use the on-chip memory still a topic of active debate in the computer architecture community.
Picture courtesy of AMD
Vinod Ganapathy - Policies and Mechanisms for OS Security

68. Vinod Ganapathy - Policies and Mechanisms for OS Security
Our use of 3D-stacking
On-chip DRAM treated as a page-granularity cache of off-chip DRAM memory
Every address accessed by the CPU will result in the page frame being fetched to on-chip DRAM
Cache of off-chip
DRAM memory
Off-chip DRAM
On-chip DRAM
Memory bus
Memory controller
Crypto logic
CPU
Vinod Ganapathy - Policies and Mechanisms for OS Security

69. Triggering snapshot acquisition
Off-chip DRAM
On-chip DRAM
Memory bus
Memory controller
Crypto logic
CPU
Trigger = Device that communicates to the CPU to enter snapshot acquisition mode:
Physical device attached to South/NorthBridge that sends a non-maskable interrupt
NIC with Wake-on-LAN-like feature
Vinod Ganapathy - Policies and Mechanisms for OS Security

70. Snapshot acquisition mode1
Off-chip DRAM
CoW
Cache
Memory bus
Memory controller
Crypto logic
CPU
Memory controller splits on-chip DRAM into two parts:
Cache of off-chip DRAM memory
Copy-on-Write (CoW) area
Vinod Ganapathy - Policies and Mechanisms for OS Security

71. Snapshot acquisition mode2
Off-chip DRAM
CoW
Cache Fi Fi
Memory bus
Memory controller
Crypto logic
CPU
Hardware brings one page frame of off-chip DRAM at a time to on-chip DRAM cache
Vinod Ganapathy - Policies and Mechanisms for OS Security

72. Snapshot acquisition mode3
Off-chip DRAM
+ Page#
+ Rand# Fi
CoW
Cache
Memory bus
Memory controller
Crypto logic
CPU
Crypto logic digitally signs contents of page:
Random nonce used to prevent replay attacks
Same nonce used for all pages in snapshot
Vinod Ganapathy - Policies and Mechanisms for OS Security

73. Snapshot acquisition mode4
Off-chip DRAM
+ Page#
+ Rand# Fi
CoW
Cache
Memory bus
Memory controller
Crypto logic
CPU
Disk
Hardware instructs OS to write signed page to external medium:
Even if OS is infected, it cannot cheat, since integrity of page is protected by the hardware
Vinod Ganapathy - Policies and Mechanisms for OS Security

74. Snapshot acquisition mode5
Off-chip DRAM Fj
CoW
Cache Fj
Memory bus
Memory controller
Crypto logic
CPU
CPU continues to execute concurrently:
If it writes to page Fj that has not yet been copied  Memory controller makes a copy of the original page in the Copy-on-Write area
When hardware ready to snapshot Fj, copy created from Copy-on-Write area
Vinod Ganapathy - Policies and Mechanisms for OS Security

75. At conclusion of acquisition…
+ 0
+ R F0
+ 1
+ R F1
+ N
+ R FN
Consistent snapshot of off-chip memory at instant when acquisition was initiated
Snapshot is tamper-resistant even to a corrupted OS
Obtained without pausing target machine
If OS attempts to hide malicious activity, will be evident because CoW will capture original page.
Vinod Ganapathy - Policies and Mechanisms for OS Security

76. Vinod Ganapathy - Policies and Mechanisms for OS Security
Security analysis …
+ 0
+ R F0
+ 1
+ R F1
+ N
+ R FN
Malicious OS cannot:
Corrupt pages in snapshot: Integrity
Hide pages from snapshot: Completeness
Replay old snapshot: Freshness
“Clean” itself during snapshot acquisition because Copy-on-Write stores original page: External control
If OS attempts to hide malicious activity, will be evident because CoW will capture original page.
Vinod Ganapathy - Policies and Mechanisms for OS Security

77. Vinod Ganapathy - Policies and Mechanisms for OS Security
Evaluation
Atop 3D-stacked hardware emulator
Evaluated:
Impact of 3D-stacked memory available
Effectiveness of performance-isolation claim
Used canneal, memcached, graph500, mcf
Time to procure full snapshot of memory:
~ seconds, depending on external medium
Complexity of hardware modifications:
Evaluated using CACTI and Aladdin
Negligible area/energy overheads
Vinod Ganapathy - Policies and Mechanisms for OS Security

78. Evidence of performance isolation
Only showing you two noteworthy data points: Elided results for graph500 and mcf benchmarks
Applications make progress as long as space available in on-chip CoW area. Space in CoW area dependent on speed of external medium that stores snapshot
Vinod Ganapathy - Policies and Mechanisms for OS Security

79. Other research projects…
Generic theme: Computer Systems Security
Improving cloud platform security
[ACSAC’08, RAID’10, CCS’12a, SOCC’14]
Security for mobile devices (and other IoT devices)
[MobiSys’11, TIFS’13, MobiSys’16]
Hardware support for software and system security [CCS’08, ECOOP’12a, TIFS’13, MobiSys’16, RU-DCS-TR724]
Web application and Web browser security
[ACSAC’08, ACSAC’09, ECOOP’12a, ECOOP’12b, ECOOP’14, FSE’14]
Tools for cross-platform mobile app development [ICSE’13, ASE’15]
Retrofitting legacy software for security
[CCS’05, Oakland’06, ASPLOS’06, ICSE’07, CCS’08, CCS’12b]
Proofs of security for retrofitting transformations [Work in progress]
Vinod Ganapathy - Policies and Mechanisms for OS Security

80. A big thank you to my students
Graduated PhDs
Dr. Mohan Dhawan (IBM Research India)
Dr. Saman Zarandioon (Amazon.com)
Dr. Shakeel Butt (NVidia  now at Google)
Dr. Liu Yang (HP Labs  now at Baidu)
Dr. Rezwana Karim (Samsung Research America)
Dr. Amruta Gokhale (Teradata)
Former Postdocs
Dr. Arati Baliga (AT&T Security Labs)
Graduated MS students
Jeffrey Bickford (AT&T Research)
Yogesh Padmanaban (Microsoft)
Current PhD students
Jay P. Lim, Hai Nguyen, Daeyoung Kim.
Vinod Ganapathy - Policies and Mechanisms for OS Security

81. URL: http://www.cs.rutgers.edu/~vinodg
URL: