HEY, YOU, GET OFF OF MY CLOUD: EXPLORING INFORMATION LEAKAGE IN THIRD-PARTY COMPUTE CLOUDS Eran Tromer MIT Hovav Shacham UCSD Stefan Savage UCSD ACM CCS 2009 Thomas Ristenpart UCSD Presented by: Yun Liaw

Outline  Introduction  Threat Model  The EC2 Service  Cloud Cartography  Determine Co-Residence  Exploiting Placement in EC2  Cross-VM Information Leakage  Conclusions & Comments 2009/11/20Presented by: Yun Liaw 2

Introduction  Cloud Computing: The next generation infrastructure for hosting data and deploying software and services  Amazon, Google, Microsoft, …  Advantages: economies of scale, dynamic provisioning, low capital expenditure…  In cloud infrastructure, in order to maximize the efficiency, physical resources are shared between virtual machines  Here introduces the new risk! 2009/11/20Presented by: Yun Liaw 3

Threat Model 2009/11/20Presented by: Yun Liaw 4  Cloud customer ↔ Malicious cloud provider  The cloud provider violates customer confidentiality or integrity Insider threats within the cloud provider  But this is not in the scope of this paper!  Cloud Customer ↔ Non-provider-affiliated malicious parties  Direct compromise  A malicious party can run its instance on the same physical machines as the potential victim ← co-residence The attacker might manipulate shared physical resources to learn the victim’s confidential information Here’s this paper’s focus!

Threat Model  Can one Determine where in the cloud infrastructure an instance is located?  Can one easily determine if two instances are co- resident on the same physical machine?  Can an adversary launch instances that will be co- resident with other user’s instance?  Can an adversary exploit cross-VM information leakage once co-resident? 2009/11/20Presented by: Yun Liaw 5

The EC2 Service  Provided by Amazon  VM: Xen hypervisor [1]  To monitor VMs  Provide isolation between VMs  Intermediating access to physical devices  Domain0 (Dom0): A privileged VM in Xen hypervisor to manage: guest images physical resource provisioning access control right Dom0 is used for route packets of the guest image Can be tracked by traceroute 2009/11/20Presented by: Yun Liaw 6 [1]P.Barham, et al., “Xen and The Art of virtualization,” ACM SOSP, 2003

The EC2 Service  Three “degrees of freedom” in specifying the physical infrastructure of an instance  Region: United State or Europe  Availability Zone: Distinct locations that are engineered to be insulated from failures each region has three availability zones e.g., us-east-1a  Instance Type: Five types: “m1.small”, “m1.large”, “m1.xlarge”, “c1.medium”, “c1.xlarge” /11/20Presented by: Yun Liaw 7

The EC2 Service and Network Probing  Each instance in EC2 are given Internet connectivity by:  External IPv4 address and domain name e.g., with domain name ec compute-1.amazonaws.com  Internal RFC 1918 private address and domain name e.g., with domain name domU D-C6.compute-1.internal  The DNS resolution queries from an EC2 instance  Can determine the external name of an instance  Can determine the internal IP of an instance’s external IP 2009/11/20Presented by: Yun Liaw 8

Cloud Cartography  To “map” the EC2 service to understand where the instances are located in the cloud  To understand the instance creation parameter needed to attempt establishing co-residence instance  Assumption: Different availability zones and instance types are likely to correspond to different internal IP ranges  Validation: Create a number of EC2 instances with different zones and types 2009/11/20Presented by: Yun Liaw 9

Cloud Cartography – Experiment 1  Experiment 1:  Using single account A to create 300 instances  Each zone/type pair are with 20 instances  If internal IPs are statically assigned to physical machines… Availability zones use separate physical infrastructure! 2009/11/20Presented by: Yun Liaw 10

Cloud Cartography – Experiment 2  Experiment 2:  Using another account B to launch 100 instances only in zone 3  Each type are with 20 instances  Result:  In 100 instances in account A 92 instances had unique /24 prefix 4 of /24 prefixes had two instances, but with same type  In 100 instances in account B 88 instances had unique /24 prefix 6 of /24 prefixes had two instances Only single /24 prefixes had two different type instances 2009/11/20Presented by: Yun Liaw 11

Cloud Cartography – Heuristic Rules  Heuristic Rule for label /24 prefixes with zones and types:  All IPs from a /16 are from the same zone  A /24 inherits any included sampled instance type. If there are multiple observed instances with distinct type, we label this /24 with each types  A /24 containing a Dom0 IP only contains Dom0 IP. We associate this /24 to the type of Dom0’s associated instance  All /24’s between two consecutive Dom0 /24’s inherit the former’s associated type (?) Dom0 IPs are consistently assigned a prefix that immediately precedes the instance IPs they are associated with (?) 2009/11/20Presented by: Yun Liaw 12

Cloud Cartography - Prevention  In EC2, local IP are statically associated to zones and types  randomly assign internal IP address  It would make administrator’s task more challenging  Cloud provider may isolate each account’s view of the internal IP address space (restrict DNS query) 2009/11/20Presented by: Yun Liaw 13

Determining Co-Residence  The two instances are likely co-resident if they are  Matching Dom0 IP address  Small packet round-trip times  Numerically close internal IP address  Network-based co-residence check  Verification:  The ground truth: If two instances can successfully transmit via a convert channel, they are co-resident  [slide 2009/11/20Presented by: Yun Liaw 14

Determining Co-Residence - Experiment  Experiment: 1. Three accounts: control, victim and probe 2. Two instances were launched by control in each zones 3. Victim and probe both launched 20 instances in zone 3 4. Performed Dom0 IP check for each victim/probe pair 5. If passed, let probe probed victim to calculate RTT 6. Control probed probe and victim to calculate RTT 7. Probe sent 5-bit to victim through the covert channel 2009/11/20Presented by: Yun Liaw 15

Determining Co-Residence - Experiment  Experiment Result:  There are 31 pairs passed the Dom0 IP test (62 ordered pairs)  60 pairs passed convert-channel test Two failures due to a single bit error, and these two are not for the same pair of instance  Result of RTT: 2009/11/20Presented by: Yun Liaw 16

Determining Co-Residence  Simple co-residence check: 1. One compares the internal IP of the two instances 2. If they are numerically close, perform Dom0 IP test  Obfuscating co-residence check  Let Dom0 not respond in traceroute  Randomly assign internal IP  VLAN to isolate accounts 2009/11/20Presented by: Yun Liaw 17

Exploiting Placement in EC2  Observations from the data:  Single count was never seen to have two instances that is simultaneously co-resident  No more than 8 m1.small instances were observed to be simultaneously co-resident due to the physical limitation that single physical machine can only support at most 8 m1.small instances  Sequential placement locality: two instances run sequentially are often assigned to the same machine  Parallel placement locality: two instances run at roughly the same time are often assigned to the same machine 2009/11/20Presented by: Yun Liaw 18

Exploiting Placement in EC2 – Brute-Forcing Placement  Strategy: 1. Attacker enumerates a set of potential targets (should be big) 2. Infer the zone and type of these targets 3. Launch probe instances in the zone and type that targets belongs to 4. Each probe instance probes each target to see if there is any co-resident the target 5. If not, terminate the instance immediately  Experiment Result:  8.4% co-residence coverage over 1785 probe instances, probing 1686 public web servers hosted on EC2  The first 510 launched probe instances had achieved 90% co- residence coverage … strongly sequential placement locality! 2009/11/20Presented by: Yun Liaw 19

Exploiting Placement in EC2 – Abusing Placement Locality  Strategy: Instance Flooding 1. Once the attacker observe (or trigger) the victim launch his instance… The servers in cloud often terminate when it is not in use The attacker can monitor a server’s state to observe whether the victim’s instance is re-launched Attackers can also trigger new victim instance due to the auto scaling systems 2. Attacker launching as many instances in parallel as possible, in the appropriate zone and type 2009/11/20Presented by: Yun Liaw 20

Exploiting Placement in EC2 – Abusing Placement Locality  Experiment: the effect of zone and account  A victim account launch either 1, 10, 20 instance simultaneously  No sooner than 5 minutes, an attacker account launch 20 instances simultaneously 2009/11/20 21

Exploiting Placement in EC2 – Abusing Placement Locality  Experiment: the effect of increased time lag  40 victim instances were launched in zone 3 throughout the experiment (36 unique machines)  Every hour a set of 20 attack instances were launched and perform the co-residence check 2009/11/20Presented by: Yun Liaw 22 Total co-resident: # of probe instances that were co-residence with the victim at that hour New co-resident: # of victim instances that were collided with for the first time at that hour

Exploiting Placement in EC2  Instance flooding! But when fails?  Target instance were being assigned to machines with high instance density or even full  Patching placement vulnerabilities  Inhibit cartography and co-residence check  Let users request placement of their VMs on machines that can only be populated by VMs from their (trusted) accounts 2009/11/20Presented by: Yun Liaw 23

Cross-VM Information Leakage  Time-shared caches allow an attacker to measure when other instances are experiencing computation load (through cache access)  Co-residence detection  Web traffic detection  Keystroke timing attack 2009/11/20Presented by: Yun Liaw 24

Cross-VM Information Leakage – Prime+Trigger+Probe  Prime+Trigger+Probe 1. The probing instance allocates a contiguous buffer B of b bytes 2. Let s be the cache line size (in bytes) 3. Prime: the probing instance read buffer B at s-byte offsets into the cache 4. Trigger: the probing instance perform a busy-loop until the CPU’s cycle counter jumps by a large value 5. Probe: the probing instance measure the time it takes to again read B at s-byte offsets 2009/11/20Presented by: Yun Liaw 25

Cross-VM Information Leakage – Load-Based Co-Residence Detection  We could detect co-residence when adversary has the knowledge of computational load variation on the target instance  Publicly-accessible services on the target instance  A priori information about load variation on the target 2009/11/20Presented by: Yun Liaw 26

Cross-VM Information Leakage – Load-Based Co-Residence Detection  Experiment:  Environment: A web server on EC2 “m1.small” instance Hosting a single 1024 byte text HTML page 1. The attacker VM took 100 load samples 2. After 10 sec, the attacker VM took 100 samples while simultaneously making numerous HTTP request to the web server 27

Cross-VM Information Leakage – Estimating Traffic Rates  HTTP traffic rate estimation to a co-resident web server experiment  Perform four separate runs of 1000 cache load measurement on a web server hosted on EC2 “m1.small” simultaneously with Sent no HTTP request Sent 50 HTTP request per min Sent 100 HTTP request per min Sent 200 HTTP request per min 2009/11/20Presented by: Yun Liaw 28

Cross-VM Information Leakage – Keystroke Timing Attack  The adversary's goal:  To measure the time between keystrokes made by a victim typing a password  The keystroke time can be used to perform recovery of the password [2]  Assumption: on an otherwise idle machine, a spike in load corresponds to a letter being typed into the co-resident VM’s terminal 2009/11/20Presented by: Yun Liaw 29 [2] D. Song, et. al., “Timing Analysis of Keystrokes and SSH Timing Attacks,” USENIX Security Symposium, 2001

Cross-VM Information Leakage – Keystroke Timing Attack  Experiment setup: Not on EC2!  A testbed similar to EC2  In order to “pinned” the VM with the CPU core  EC2 occasionally migrates VM’s virtual CPUs between the machine’s cores  Reason: Ensure that machine is completely idle other than the test core Allows us to know the relation between VMs involved in the attack 2009/11/20Presented by: Yun Liaw 30

Cross-VM Information Leakage – Keystroke Timing Attack  Keystroke activity side channel  Repeatedly perform load measurement  Report a keystroke when the measurement indicates momentarily high cache usage  Experiment Result:  Clear signal with 5%missed keystrokes  0.3 false triggers per second  Timing resolution: 13ms  Clear difference between keys with different effects (shell command vs enter)  Inhibiting side-channel attacks  Blinding techniques 2009/11/20Presented by: Yun Liaw 31

Conclusions & Comments  Conclusions:  Studied the risk arise from sharing physical infrastructure between mutually distrustful users Cloud cartography Co-residence detection Placement exploitation Cross-VM information leakage  Comments:  The risk mentioned is more likely the system-wide problems  How to configure the cloud internal network to blind the infrastructure and placement?  How to log the cloud behavior to support further investigation? 2009/11/20Presented by: Yun Liaw 32

Cross-VM Information Leakage – Cache-Based Covert Channel  The senders idles to transmit “0”, and frantically access the memory to transmit “1”  Set-associative cache:  The pool of cache lines is partitioned into associative sets  Each memory address is mapped into a specific associative set  We partition the cache sets into 2 classes: “odd sets” and “even sets”  Odd sets with odd memory addresses; even set with even addresses  Differential encoding:  In order to mitigate the noise in the load sample  The signal is carried in the load difference between two classes  The noise would be balanced between the two classes 2009/11/20Presented by: Yun Liaw 33

Cross-VM Information Leakage – Cache-Based Covert Channel  Sender: (to transmit “0”) 1. Allocates a contiguous buffer A of a bytes 2. Reads the even addresses in A  Receiver: 1. Allocate a contiguous buffer B of b bytes 2. Sleep briefly (to build up credit with Xen’s scheduler) 3. Prime: read all B to make sure it’s fully cached 4. Trigger: the probing instance perform a busy-loop until the CPU’s cycle counter jumps by a large value 5. Probe: Measure the time to read all even addresses in B, likewise for the odd addresses 6. Decide “0” iff the difference is positive 2009/11/20Presented by: Yun Liaw 34

Cross-VM Information Leakage – Cache-Based Covert Channel  Sender: (to transmit “1”) 1. Allocates a contiguous buffer A of a bytes 2. Reads the odd addresses in A  Receiver: 1. Allocate a contiguous buffer B of b bytes 2. Sleep briefly (to build up credit with Xen’s scheduler) 3. Prime: read all B to make sure it’s fully cached 4. Trigger: the probing instance perform a busy-loop until the CPU’s cycle counter jumps by a large value 5. Probe: Measure the time to read all even addresses in B, likewise for the odd addresses 6. Decide “1” iff the difference is negative 2009/11/20Presented by: Yun Liaw 35