1. Srikanth Kandula, Sudipta Sengupta, Albert Greenberg, Parveen Patel, Ronnie Chaiken Microsoft Research IMC November, 2009 Abhishek Ray raya@cs.ucr.edu THE NATURE OF DATACENTER: MEASUREMENTS & ANALYSIS

2. Outline Introduction Data & Methodology Application Traffic Characteristics Tomography Conclusion

3. Introduction Analysis and mining of data sets Processing around some petabytes of data This paper has tried to describe characteristics of traffic Detailed view of traffic Congestion conditions and patterns

4. Contribution Measurement Instrumentation Measures traffic at data centers rather than switches Traffic characteristics Flow, congestion and rate of change of traffic mix. Tomography Inference Accuracy Performs  Clusters =1500 servers  Rack = 20 2 months

5. Data & Methodology ISPs SNMP Counters Sampled Flow Deep packet Inspection Data Center Measurements at Server  Servers, Storage and network  Linkage of network traffic with application level logs

6. Socket level events at each servers ETW – Event Tracing for Windows One per application read or write Aggregates over several packets http://msdn.microsoft.com/en-us/magazine/cc163437.aspx#S1

7. ETW – Event tracing for Windows

8. Application Workload SQL Programming language like Scope 3 phases of different types Extract Partition Aggregate Combine Short interactive programs to long running programs

9. Traffic Characteristics

10. Patterns

11. Work-Seeks-BW and Scatter-Gather patterns in datacenter traffic exchanged b/w server pairs

12. Work-seeks-bandwidth Within same servers Within servers in same rack Within servers in same VLAN Scatter-gather-patterns Data is divided into small parts and each servers works on particular part Aggregated

13. How much traffic is exchanged between server pairs?

14. Server pair with same rack are more likely to exchange more bytes Probability of exchanging no traffic 89 % - servers within same rack 99.5 % - servers in different rack

15. How many other servers does a server correspond with?

16. Sever either talks to all other servers with the same rack Servers doesn’t talk to servers outside the rack or talks 1-10 % outside servers.

17. Congestion within the Datacenter Congestion within the Datacenter

18. N/W at as high an utilization as possible without adversely affecting throughput Low network utilization indicate Application by nature demands more of other resources such as CPU and disk than the network Applications can be re-written to make better use of available network bandwidth

19. Where and when the congestion happens in data center

20. Congestion Rate  86 % - 10 seconds  15 % - 100 seconds Short congestion periods are highly correlated across many tens of links and are due to brief spurts of high demand from the application Long lasting congestion periods tend to be more localized to a small set of links

21. Length of Congestion Events

22. Compares the rates of flows that overlap high utilization periods with the rates of all flows

23. Impact of high utilization

24. Read failure - Job is killed Congestion To attribute network traffic to the applications that generate it, they merge the network event logs with logs at the application-level that describe which job and phase were active at that time

25. Reduce phase - Data in each partition that is present at multiple servers in the cluster has to be pulled to the server that handles the reduce for the partition e.g. count the number of records that begin with ‘A’ Extract phase – Extracting the data Largest amount of data Evaluation phase – Problem Conclusion – High utilization epochs are caused by application demand and have a moderate negative impact to job performance

26. Flow Characteristics Flow Characteristics

27. Traffic mix changes frequently

28. How traffic changes over time within the data center

29. Change in traffic 10 th and 90 th percentiles are 37 % and 149 % the median change in traffic is roughly 82 %  even when the total traffic in the matrix remains the same, the server pairs that are involved in these traffic exchanges change appreciably

30. Short bursts cause spikes at the shorter time-scale (in dashed line) that smooth out at the longer time scale (in solid line) whereas gradual changes appear conversely, smoothed out at shorter time-scales yet pronounced on the longer time-scale Variability - key aspect for data center

31. Inter-arrival times in the entire cluster, at Top-of-Rack switches and at servers

32. Inter-arrivals at both servers and top-of-rack switches have spaced apart by roughly 15 ms This is likely due to the stop-and-go behavior of the application that rate-limits the creation of new flows  Median arrival rate of all flows in the cluster is 10 5 fl ows per second or 100 flows in every millisecond

33. Tomography N/W tomography methods to infer traffic matrices If the methods used in ISP n/w is applicable to datacenters, it would help to unravel the nature of traffic Why? Data flow volume is quadratic n(n - 1) – no. of links measurements are fewer Assumptions - Gravity model - Amount of traffic a node (origin) would send to another node (destination) is proportional to the traffic volume received by the destination Scalability

34. Methodology Computes ground truth TM and measure how well the TM estimated by tomography from these link counts approximates the true TM

35. Tomogravity and Spare Maximization

36. Tomogravity - Communication likely to be B/W nodes with same job rather than all nodes, whereas gravity model, not being aware of these job-clusters, introduces traffic across clusters, resulting in many non-zero TM entries Spare maximization – Error rate starts from several hundreds

37. Comparison the TMs by various tomography methods with the ground truth

38. Ground TMs are sparser than tomogravity estimated TMs, and denser than sparsity maximized estimated TMs

39. Conclusion Capture both Macroscopic patterns – which servers talk to which others, when and for what reasons Microscopic characteristics – flow durations, inter-arrival times Tighter coupling between network, computing, and storage in datacenter applications Congestion and negative application impact do occur, demanding improvement - better understanding of traffic and mechanisms that steer demand

40. My Take More data should be examined over a period of 1 year instead of 2 months I would certainly like to see some mining of data and application running at datacenters of companies like Google, Yahoo etc

41. Related Work T. Benson, A. Anand, A. Akella, andM. Zhang: Understanding Datacenter Traffic Characteristics, In SIGCOMMWREN workshop, 2009. A. Greenberg, N. Jain, S. Kandula, C. Kim, P. Lahiri, D. Maltz, P. Patel, and S. Sengupta: VL 2 : A Scalable and Flexible Data Center Network, In ACM SIGCOMM, 2009.

42. Thank You