Q. Huang, J. Xia, M. Sun, K. Liu, J. Li, Z. Gui, C. Xu, C. Yang, Chapter 14 How to test the readiness of open-source cloud computing solutions, In Spatial Cloud Computing: a practical approach, edited by C.Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp Chapter 14 How to Test the Readiness of Open Source Cloud Computing Solutions Qunying Huang, Jizhe Xia, Min Sun, Kai Liu, Jing Li, Zhipeng Gui, Chen Xu, and Chaowei Yang

Q. Huang, J. Xia, M. Sun, K. Liu, J. Li, Z. Gui, C. Xu, C. Yang, Chapter 14 How to test the readiness of open-source cloud computing solutions, In Spatial Cloud Computing: a practical approach, edited by C.Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp Learning Objectives How to test the readiness of open-source cloud solutions

Q. Huang, J. Xia, M. Sun, K. Liu, J. Li, Z. Gui, C. Xu, C. Yang, Chapter 14 How to test the readiness of open-source cloud computing solutions, In Spatial Cloud Computing: a practical approach, edited by C.Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp Learning Materials Videos: o Chapter_14-Video_1.wmvChapter_14-Video_1.wmv o Chapter_14-Video_2.wmvChapter_14-Video_2.wmv o Chapter_14-Video_3.wmvChapter_14-Video_3.wmv o Chapter_14-Video_4.wmvChapter_14-Video_4.wmv o Chapter_14-Video_5.mp4Chapter_14-Video_5.mp4 Scripts, Files and others: o ubench-patch.txt ubench-patch.txt 3

Q. Huang, J. Xia, M. Sun, K. Liu, J. Li, Z. Gui, C. Xu, C. Yang, Chapter 14 How to test the readiness of open-source cloud computing solutions, In Spatial Cloud Computing: a practical approach, edited by C.Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp Learning Modules 1.Open-source cloud solutions 2.Test environment 3.Cloud readiness test Tests of cloud operations Tests of virtual computing resource Tests of general applications Cloud Readiness Test for GEOSS Clearinghouse Cloud Readiness Test for Dust Storm Forecasting 4.Conclusion and discussions

Q. Huang, J. Xia, M. Sun, K. Liu, J. Li, Z. Gui, C. Xu, C. Yang, Chapter 14 How to test the readiness of open-source cloud computing solutions, In Spatial Cloud Computing: a practical approach, edited by C.Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp Open-source cloud solutions Solution Partial List of Customers Maturation (years in business) Feedback from Community Installation Complexities Eucalyptus Cloudera, DoD, NASA, HP, SONY, FDA, Puma, USDA The first version was released in May, 2008 While the software is easy to install, cloud consumers have reported challenges of network configuration Installation is relatively simple for first time installer; Network configuration is difficult OpenNebula IBM, Hexagrid, CloudWeavers, CERN The first public release was in 2008 New stable version has been contributed to the project Steep learning curve for first time installer CloudStackZynga, Edmunds.com, Nokia Research Center CloudStack was released in May, 2010 Well documented; May have problems with creating instances with Xen Easy to install because packages are prepared for guiding installers through the process Table 14.1 Customer and evaluations of the software solutions

Q. Huang, J. Xia, M. Sun, K. Liu, J. Li, Z. Gui, C. Xu, C. Yang, Chapter 14 How to test the readiness of open-source cloud computing solutions, In Spatial Cloud Computing: a practical approach, edited by C.Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp Open-source cloud solutions Solution Open Source Version Cost (Enterprise edition) OS that can support the cloud OS Images that can be supported EucalyptusYes Licensing based on number of physical hosts Red Hat Enterprise; CentOS; openSUSE-11; Debian; Fedora; Ubuntu Windows; Major Linux OS distributions OpenNebulaYesFreeUbuntu and CentOS Windows; Major Linux OS distributions CloudStackYesFreeRed Hat Enterprise; CentOS 5; CentOS 6.0; CentOS 6.1; Ubuntu 10.04; Fedora 14 Windows; Major Linux OS distributions Table 14.2 The general characteristics of the selected solutions

Q. Huang, J. Xia, M. Sun, K. Liu, J. Li, Z. Gui, C. Xu, C. Yang, Chapter 14 How to test the readiness of open-source cloud computing solutions, In Spatial Cloud Computing: a practical approach, edited by C.Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp Learning Modules 1.Open-source cloud solutions 2.Test environment 3.Cloud readiness test Tests of cloud operations Tests of virtual computing resource Tests of general applications Cloud Readiness Test for GEOSS Clearinghouse Cloud Readiness Test for Dust Storm Forecasting 4.Conclusion and discussions

Q. Huang, J. Xia, M. Sun, K. Liu, J. Li, Z. Gui, C. Xu, C. Yang, Chapter 14 How to test the readiness of open-source cloud computing solutions, In Spatial Cloud Computing: a practical approach, edited by C.Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp Test Environment Figure 14.1.The open source solution test environment Figure Conceptual cloud architecture

Q. Huang, J. Xia, M. Sun, K. Liu, J. Li, Z. Gui, C. Xu, C. Yang, Chapter 14 How to test the readiness of open-source cloud computing solutions, In Spatial Cloud Computing: a practical approach, edited by C.Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp Cloud VM configurations Instance TypeArchitectureOS RAM [GB] Virtual Cores Disk [GB] Small64-bitCentOS Medium64-bitCentOS Large64-bitCentOS XLarge64-bitCentOS XXLarge64-bitCentOS

Q. Huang, J. Xia, M. Sun, K. Liu, J. Li, Z. Gui, C. Xu, C. Yang, Chapter 14 How to test the readiness of open-source cloud computing solutions, In Spatial Cloud Computing: a practical approach, edited by C.Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp Tests of cloud operations Single resource allocation and release performance are tested. The startup and release of resources, including VM and storage volume are tested. Each operation test is repeated 5 times for each instance type with a resource acquisition followed by a release. Time costs for starting, pausing, restarting, and deleting a VM with different virtual computing resources are recorded and compared.

Q. Huang, J. Xia, M. Sun, K. Liu, J. Li, Z. Gui, C. Xu, C. Yang, Chapter 14 How to test the readiness of open-source cloud computing solutions, In Spatial Cloud Computing: a practical approach, edited by C.Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp Tests of virtual computing resource CPU Video: Chapter_12_Video_1.mp4Chapter_12_Video_1.mp4 Video: Chapter_12_Video_2.mp4Chapter_12_Video_2.mp4 Video: Chapter_12_Video_3.mp4Chapter_12_Video_3.mp4 Video: Chapter_12_Video_4.mp4Chapter_12_Video_4.mp4 I/O performance Memory hierarchy performance Networking performance

Q. Huang, J. Xia, M. Sun, K. Liu, J. Li, Z. Gui, C. Xu, C. Yang, Chapter 14 How to test the readiness of open-source cloud computing solutions, In Spatial Cloud Computing: a practical approach, edited by C.Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp Learning Modules 1.Open-source cloud solutions 2.Test environment 3.Cloud readiness test Tests of cloud operations Tests of virtual computing resource Tests of general applications Cloud Readiness Test for GEOSS Clearinghouse Cloud Readiness Test for Dust Storm Forecasting 4.Conclusion and discussions

Q. Huang, J. Xia, M. Sun, K. Liu, J. Li, Z. Gui, C. Xu, C. Yang, Chapter 14 How to test the readiness of open-source cloud computing solutions, In Spatial Cloud Computing: a practical approach, edited by C.Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp Virtual Computing Resource Test Benchmarkers BenchmarkerTest Items Virtual computing resource UBench(UBench, 2012)CPU Bonnie++ (Coker, 2008) Hard drive Cachebench Memory Hierarchy Performance Iperf (Tirumala et al., 2012) Network General applicationsql-bench MySQL SPECjvm2008 JAVA Geoscience applicationSpatial web portal(database, tomcat) GEOSS Clearinghouse MPI application Dust storm model Table General benchmarkers for testing the performance of VMs

Q. Huang, J. Xia, M. Sun, K. Liu, J. Li, Z. Gui, C. Xu, C. Yang, Chapter 14 How to test the readiness of open-source cloud computing solutions, In Spatial Cloud Computing: a practical approach, edited by C.Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp Virtual Computing Resource Test Workflow Figure Virtual computing resource benchmark workflow Step 1: Building an image with different virtual computing resource benchmark software package installed for each open-source solution. To build an image, a VM needs to be launched as configured in Table 14.3 for further customization. The virtual computing resource benchmark software package should be installed on the basic VM.

Q. Huang, J. Xia, M. Sun, K. Liu, J. Li, Z. Gui, C. Xu, C. Yang, Chapter 14 How to test the readiness of open-source cloud computing solutions, In Spatial Cloud Computing: a practical approach, edited by C.Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp Virtual Computing Resource Test Workflow Figure Virtual computing resource benchmark workflow Step 2: Start different types of VMs. The five VM types (Table 14.3) can be launched from the images built in Step 1 on each solution. Step 3: Run the test. This step requires creating the script to run each benchmarker sequentially. Typically, each benchmarker will be repeated three times, and wait for 60 seconds between each test. Step 4: Collect and analyze the results. When the benchmarking is completed, check the log files (e.g., ubench.log [Box 14.5]) for the benchmarking result.

Q. Huang, J. Xia, M. Sun, K. Liu, J. Li, Z. Gui, C. Xu, C. Yang, Chapter 14 How to test the readiness of open-source cloud computing solutions, In Spatial Cloud Computing: a practical approach, edited by C.Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp Virtual Computing Resource Test Result Figure UBench output Figure 14.5 A Bonnie++ benchmarking report sample Figure Cachebench output Figure Iperf output

Q. Huang, J. Xia, M. Sun, K. Liu, J. Li, Z. Gui, C. Xu, C. Yang, Chapter 14 How to test the readiness of open-source cloud computing solutions, In Spatial Cloud Computing: a practical approach, edited by C.Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp Tests of general applications JRE SQL

Q. Huang, J. Xia, M. Sun, K. Liu, J. Li, Z. Gui, C. Xu, C. Yang, Chapter 14 How to test the readiness of open-source cloud computing solutions, In Spatial Cloud Computing: a practical approach, edited by C.Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp JRE performance SPECjvm2008 contains several real-world applications and benchmarks focusing on java functionality. SPECjvm2008 focuses on the performance benchmark of JRE in a single machine. MySQL performance MySQL-Bench : MySQL-Bench is used to test the performance of MySQL database. It can test various types of database operations such as creating tables, inserting and deleting records and querying on tables, which ensures a comprehensive test on the capabilities of MySQL database supported by different solutions. General Application Test Benchmarkers Video: Chapter_12_Video_5.mp4Chapter_12_Video_5.mp4

Q. Huang, J. Xia, M. Sun, K. Liu, J. Li, Z. Gui, C. Xu, C. Yang, Chapter 14 How to test the readiness of open-source cloud computing solutions, In Spatial Cloud Computing: a practical approach, edited by C.Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp General Application Test Workflow Figure benchmark workflow Step 2: Start different types of VMs on different solutions. The five VM types (Table 14.3) can be launched from the images built in step 1 on each solution. Step 3: Run the test. Step 4: Collect and analyze the results. The log file (SPECjvm2008.log) would contain the three times test results for JRE test. The log file “sql- bench.log” would contain the test results for SQL-Bench test.

Q. Huang, J. Xia, M. Sun, K. Liu, J. Li, Z. Gui, C. Xu, C. Yang, Chapter 14 How to test the readiness of open-source cloud computing solutions, In Spatial Cloud Computing: a practical approach, edited by C.Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp General Application Test Result Figure A sample test result of SPECjvm2008 Figure Sample test result

Q. Huang, J. Xia, M. Sun, K. Liu, J. Li, Z. Gui, C. Xu, C. Yang, Chapter 14 How to test the readiness of open-source cloud computing solutions, In Spatial Cloud Computing: a practical approach, edited by C.Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp General Application Test Workflow Figure benchmark workflow Step 1: Build images with JRE benchmark software package or MySQl- Bench installed on each cloud platform.

Q. Huang, J. Xia, M. Sun, K. Liu, J. Li, Z. Gui, C. Xu, C. Yang, Chapter 14 How to test the readiness of open-source cloud computing solutions, In Spatial Cloud Computing: a practical approach, edited by C.Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp Learning Modules 1.Open-source cloud solutions 2.Test environment 3.Cloud readiness test Tests of cloud operations Tests of virtual computing resource Tests of general applications Cloud Readiness Test for GEOSS Clearinghouse Cloud Readiness Test for Dust Storm Forecasting 4.Conclusion and discussions

Q. Huang, J. Xia, M. Sun, K. Liu, J. Li, Z. Gui, C. Xu, C. Yang, Chapter 14 How to test the readiness of open-source cloud computing solutions, In Spatial Cloud Computing: a practical approach, edited by C.Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp GEOSS Clearinghouse Test Design Figure Matrix test of GEOSS Clearinghouse

Q. Huang, J. Xia, M. Sun, K. Liu, J. Li, Z. Gui, C. Xu, C. Yang, Chapter 14 How to test the readiness of open-source cloud computing solutions, In Spatial Cloud Computing: a practical approach, edited by C.Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp GEOSS Clearinghouse Test Workflow Figure 12.5 Workflow of Load balancing and auto- scaling test

Q. Huang, J. Xia, M. Sun, K. Liu, J. Li, Z. Gui, C. Xu, C. Yang, Chapter 14 How to test the readiness of open-source cloud computing solutions, In Spatial Cloud Computing: a practical approach, edited by C.Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp GEOSS Clearinghouse Test Result Figure 12.6 Test results of GEOSS TestPlan_1

Q. Huang, J. Xia, M. Sun, K. Liu, J. Li, Z. Gui, C. Xu, C. Yang, Chapter 14 How to test the readiness of open-source cloud computing solutions, In Spatial Cloud Computing: a practical approach, edited by C.Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp Dust Storm Test Design HPC Vs. Clouds : The virtual cluster built from VMs is compared to traditional clusters to quantify the overhead of transforming physical infrastructure into clouds. The result of this experiment would indicate how well the solutions support large scale scientific computing. Within this experiment, different numbers of virtualized (from one to four VMs) and non-virtualized computing resources are compared to investigate the impact of virtualized computing power, storage and networking. Open-source Solution Comparison: This experiment tests the capability of different solutions in supporting the computing- and communication-intensive applications with different numbers of VMs on the physical machines and three cloud solutions respectively. The performance results indicate the relative performance of these cloud solutions for scientific computing. Virtualization Technology: This experiment compares the performance of the same amount of computing resources virtualized by KVM and Xen. In order to exclude the impact of cloud computing solution, the resources virtualized by Xen and KVM are managed by the same cloud solution.

Q. Huang, J. Xia, M. Sun, K. Liu, J. Li, Z. Gui, C. Xu, C. Yang, Chapter 14 How to test the readiness of open-source cloud computing solutions, In Spatial Cloud Computing: a practical approach, edited by C.Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp Dust Storm Test Workflow Figure The workflow of testing the performance of three different platforms with 2, 4, and 8 VMs using dust storm model 2. Prepare the testing script 3. Start up the same number of VMs on EC2 and Nebula (e.g., 2VMs ) 1.Build up images for cloud platforms and set up dust storm environment on local cluster 1.Build up images for cloud platforms and set up dust storm environment on local cluster 4. Transfer the model source code and data to the master node of EC2, Nebula and local cluster 4. Transfer the model source code and data to the master node of EC2, Nebula and local cluster 5. Transfer the testing script to the master node of EC2, Nebula and local cluster 5. Transfer the testing script to the master node of EC2, Nebula and local cluster 10. Recycle VMs 9. Add four more VMs for EC2 and Nebula platforms, and use eight nodes to local cluster to test the performance by repeating step 6 and 7 9. Add four more VMs for EC2 and Nebula platforms, and use eight nodes to local cluster to test the performance by repeating step 6 and 7 8. Add two more VMs to EC2 and Nebula platforms, and use four nodes to local cluster to test the performance by repeating Step 6 and 7 8. Add two more VMs to EC2 and Nebula platforms, and use four nodes to local cluster to test the performance by repeating Step 6 and 7 7. Check and analzye testing results 6. Run the testing script on each platform

Q. Huang, J. Xia, M. Sun, K. Liu, J. Li, Z. Gui, C. Xu, C. Yang, Chapter 14 How to test the readiness of open-source cloud computing solutions, In Spatial Cloud Computing: a practical approach, edited by C.Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp Dust Storm Test Result ****** Test type EC2.2VMs********* ************Begin to run**************** /mnt/mirror/performancetest/test.1/nmmgmu3km.iop.0/model.parallel.8/exe Fri Nov 11 20:58:57 UTC 2011 **************** after 128 time is************* Fri Nov 11 21:25:18 UTC 2011 **************** after 120 time is************* Fri Nov 11 21:50:14 UTC 2011 **************** after 112 time is************* Fri Nov 11 22:13:45 UTC 2011 **************** after 104 time is************* Fri Nov 11 22:37:07 UTC 2011 **************** after 96 time is************* Fri Nov 11 22:59:59 UTC 2011 **************** after 80 time is************* Fri Nov 11 23:22:04 UTC 2011 **************** after 64 time is************* Fri Nov 11 23:44:44 UTC 2011 **************** after 48 time is************* Sat Nov 12 00:11:35 UTC 2011 **************** after 32 time is************* Sat Nov 12 00:39:50 UTC 2011 **************** after 16 time is************* Sat Nov 12 01:14:56 UTC 2011 Finish tar begin copy data Finish copy data Wed May 23 19:48:13 EDT 2012 ***** finish********** Figure Model test output on EC2 with two instances and different process numbers

Q. Huang, J. Xia, M. Sun, K. Liu, J. Li, Z. Gui, C. Xu, C. Yang, Chapter 14 How to test the readiness of open-source cloud computing solutions, In Spatial Cloud Computing: a practical approach, edited by C.Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp Dust Storm Test Result Figure Dust storm model performances with different platforms and process numbers

Q. Huang, J. Xia, M. Sun, K. Liu, J. Li, Z. Gui, C. Xu, C. Yang, Chapter 14 How to test the readiness of open-source cloud computing solutions, In Spatial Cloud Computing: a practical approach, edited by C.Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp Discussion questions 1.What are the aspects that should be considered to test the virtual computing resources? 2.Please enumerate the tools that can be used to test the virtual computing resources? 3.What is the general workflow of testing the virtual cloud computing resources? 4.How to test the open source solutions with general applications? 5.What are the aspects that should be considered when testing the open source solutions? 6.How to test the capability of an open-source solution in supporting concurrent intensity? 7.How to test the capability of an open-source solution in supporting computing intensity? 8.Read the results paper (Huang et al., 2013) and describe the results in 500 words. Discuss the dynamics of the results, i.e., how the results may change?

Q. Huang, J. Xia, M. Sun, K. Liu, J. Li, Z. Gui, C. Xu, C. Yang, Chapter 14 How to test the readiness of open-source cloud computing solutions, In Spatial Cloud Computing: a practical approach, edited by C.Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp Barham, P., Dragovic, B., Fraser, K., Hand, S., Harris, T., Ho, A., Neugebauer, R., Pratt, I., Warfield, A., Xen and the art of virtualization. In: Proceedings of the 19th ACM Symposium on Operating Systems Principles, Bolton Landing, New York, NY, pp , 2.Cachebench, tools/?page=detail&get_id=34&category=10. tools/?page=detail&get_id=34&category=10 3.CloudStack, Coker, R., coker.com.au/bonnie+. 5.Huang Q., Yang C., Liu K., Xia J., Xu C., Li J., Gui Z., Sun M., Li Z., Comparing Open Source Cloud Computing Solutions for Geosciences, Computers & Geosciences. 6.KVM, 2010.Kernel-based Virtual Machine. 7.Mucci, P., Llcbench (low-level characterization benchmarks). 8.Tirumala, A., Qin, F., Dugan, J., Ferguson, J., Gibbs, K., Iperf: The TCP/UDP bandwidth measurement tool. 9.UBench, Reference