Cloud Benchmarking, Tools, and Challenges Advisor: Dr. Yong Chen Student: Elham Hojati This research is supported by the Cloud and Autonomic Computing site at Texas Tech University via the StackVelocity membership contribution and the Aerospace Corporation technical partnerships.

Outline INTRODUCTION TOOLS AND FRAMEWORKS CHALLENGES OUR PROJECT PLAN Benchmarking Cloud benchmarking TOOLS AND FRAMEWORKS CHALLENGES OUR PROJECT PLAN QUESTION AND FEEDBACKS

Team and Leverage This research is supported by the Cloud and Autonomic Computing site at Texas Tech University via the StackVelocity membership contribution and the Aerospace Corporation technical partnerships.

PART 1: Introduction 4

BENCHMARKING (Generic Concept) What is a benchmark? How does a benchmark do its tasks? What are benchmark`s requirements? 5

Project Overview Benchmarking of a system is a process of assessing its performance and other characteristics, to be able to compare them with other systems. Benchmark tools try to answer the question “which is the best system in a given domain?” Cloud is one of the systems need benchmarking process.

Project Overview On one hand, considering the number of cloud computing providers and the different services each provider offers, cloud users need benchmark information to select the best service and provider for their needs. On the other hand, cloud providers and cloud architects need benchmarking results to create optimized architectures. The problem is that cloud is a very complex and dynamic environment. Therefore there are some important differences between benchmarking in dynamic cloud environment with traditional benchmarking methods in static systems.

Project Overview In this research we have studied and compared several tools and frameworks of cloud benchmarking. We have performed a survey research about benchmarking in cloud and various challenges about creating and deploying cloud benchmarking; The most important problem for cloud users and providers are: data isolation, security unreliable performance inference challenging and performance isolation. It is possible to improve performance isolation by considering fairness property especially in current implementations, mainly in I/O intensive scenarios.

WHAT ARE BENCHMARK`S REQUIREMENTS? 1. General Requirements Strong Target Audience Relevant Economical Simple Implementation Requirements Fair Repeatable Configurable Workload Requirements Scalable Metric 9

PART 2 Tools and Frameworks 10

11 CloudCmp CloudStone HiBench YCSB CloudSuite Perfkit price-performance benchmarking Target Application Legacy application Web 2.0 MapReduce Database Media streaming Servers functionality-performance Server price performance functionality Test environment Multiple instance types Amazon EC2 instances Hadoop cluster Data serving system Characterize scale-out workloads Public and private cloud Amazon, Google, Microsoft, Rackspace, IBM, HP and Linode Service IaaS PaaS Target Estimate performance and costs of running a legacy application on a cloud Social web applications Hadoop (MapReduce) programs including real-world applications Performance comparisons of the new generation of cloud data serving systems Server 11

PART 3 Challenges 12

The Art of Building and Running a Good Cloud Benchmark Main steps for building and running a good benchmark in the Cloud: Meaningful Metric Workload Design Workload Implementation Creating Trust 13

Challenges Step 1: Meaningful Metric Step 2: Workload Design Challenge 1: defining meaningful metric with considering elasticity and fairness Step 2: Workload Design Challenge 2: Resources allocation/ Scalability and performance isolation Step 3: Workload Implementation Challenge 3: Workload Generation. Challenge 4: Fairness Step 4: Creating Trust Challenge 5: Location. Challenge 6: Ownership. Challenge 7: security 14

Fairness Property A system is fair if it has the below conditions: 1. Customers working within their assigned quota should not suffer from customers exceeding their quotas. 2. Customers exceeding their quotas should suffer performance degradation if they have a negative impact on others. 3. Customers with higher quotas should be provided with better performance than customers with lower quotas.” 15

System Types based on performance isolation [7] 16

performance isolation Having different customers and hosting their application, and isolating their tasks from each other as much as possible and reducing the effect of sharing resources on the performance of each user is one of the major goals of cloud service providers. It is important to have a metric for measuring performance isolation in cloud systems. Architects and developers need to have a mechanism to evaluate performance isolation between users and the effectiveness of their methodology to guarantee the quality of the artefact and avoid distrust. Performance isolation is providing a constant Quality of Service for one user which is independent from the load used by the other users. Limitations of this method The current method and metrics does not consider all three aspects of fairness property. As a feature work we are going to improve this method and develop an advanced isolation approach to consider all the aspects of fairness property.

using game theory Resource allocation using game theory by considering fairness property. Limitations of this method Do not consider all the 3 part definition of fairness property. the problem modeling is static here.  In this project, our goal is improving this method for considering complex dynamic environment, and using that in cloud benchmarking process. It should consider elasticity. And also, our new method should consider all three aspects of fairness property, which are not covered by the existing approaches.

Conclusion / future Project Plan We have performed a survey research about benchmarking in cloud and various challenges about creating and deploying cloud benchmarking. Existing cloud benchmarks and metrics mostly focus on traditional metrics like throughput in a virtualized environment, or just consider single aspects like databases, or just focus on some cloud features. For example there are some methods for benchmarking resource allocation which consider fairness or elasticity. But there is not a method which consider both of them. Our goal is considering various aspects such as performance isolation along with fairness and elasticity features for dynamic shared resources environment.

Experimental plan Study and compare several tools and frameworks of Cloud benchmarking. Evaluate and understand challenges about creating and deploying cloud benchmarking Develop new methods for Cloud benchmarking for better performance isolation Develop new methods to improve benchmarking resource allocation process and develop an advanced isolation approach to consider all the aspects of fairness property.

References [1] Alexandru Iosup, Radu Prodan, Dick Epema ,"IaaS Cloud Benchmarking: Approaches, Challenges, and Experience", Cloud Computing for Data-Intensive Applications 2014, pp 83-104 [2] Enno Folkerts, Alexander Alexandrov, Kai Sachs, Alexandru Iosup, Volker Mark, Cafer Tosun, "Benchmarking in the Cloud: What it Should, Can, and Cannot Be", Lecture Notes in Computer Science Volume 7755, 2013, pp 173-188. [3] C.Vazquez, R. Krishnan, and E. John, "Cloud Computing Benchmarking: A Survey", 2014. [4] Edward Wustenhoff, CTO, Burstorm, "Cloud Computing Benchmark RB-A, the 1st step to continuous price-performance benchmarking of the cloud", Rice Burstorm Price Performance Benchmark Report (RB-A) June 2015. [5] Bin Sun, Brian Hall, Hu Wang, Da Wei Zhang, Kai Ding, "Benchmarking Private Cloud Performance with User- Centric Metrics", 2014 IEEE International Conference on Cloud Engineering. [6] GuiyiWei · Athanasios V. Vasilakos · Yao Zheng ·, Naixue Xiong, "A game-theoretic method of fair resource allocation for cloud computing services", Springer Science+Business Media, LLC 2009. [7] RouvenKrebs, ChristofMomm, SamuelKounev,"Metrics and techniques for quantifying performance isolation in cloud environments", Elsevier 2013. 21