1. Software Testing in the Cloud Leah Riungu-Kalliosaari

2. Introduction  The study looks at an intersection of cloud computing and software testing  Applications are tested as services by use of cloud- based resources.  Daily operation, maintenance, and testing support through web-based browsers, testing frameworks and servers  Testing is seen as an arena for piloting cloud computing adoption

3. Objective  To understand how organizations can successfully use the cloud for testing.  Observe the adoption of cloud computing in different organizational contexts  Impact of cloud computing on testing; testing as a service  Use of empirical observations, with qualitative research methods

4. Testing in the Cloud  Testing in the cloud affects  The acquisition model (cloud based testing emphasizes services  The business model (cloud based testing emphasizes pay per use instead of license fees  The access model (services are accessed over the internet  The technical models of testing (e.g. scalability)

5. What has been done so far  Conditions that influence software testing as a service  Requirements, benefits, and challenges for software testing in the Cloud.  Research issues for software testing in the cloud  Facets of testing in the cloud  Effects of testing in the cloud  On actual testing work  On delivery and support of testing services  Roadmap towards testing in the cloud

6. Testing in the Cloud (1) 1a. SaaS software 3. Testing the cloud 2. Testing environments in the cloud 1b. Non-SaaS software 1.The system or application under test is available online 2.Testing infrastructure and platforms are hosted in the cloud (Including crowdsourcing/Human as a Service-(Haas)) 3.Testing of the cloud itself Facets of testing in the cloud Source: L.M. Riungu, O. Taipale, K. Smolander, “Research Issues for Software Testing the Cloud, ”2nd International Conference on Cloud Computing Technology and Science, 2010.

7. Testing in the Cloud (2) Requirements Domain knowledgeImportant especially for mission critical systems. InfrastructureTesting resources, e.g. servers. These can be hosted in and obtained from the cloud SecurityData security across networks, confidentiality of customer data PricingTransparent pricing models, Service level agreements CommunicationMeetings, video conferences, telephone calls, formal software test management systems Testers’ skillsNeed for new/better skills, e.g. communication and global project management skills Source: L.M. Riungu, O. Taipale, K. Smolander, “Software Testing as an Online Service: Observations from Practice, ”3rd International Conference on Software Testing, Verification and Validation Workshops, 2010.

8. Testing in the Cloud (3) Benefits Reduced costsInvestment costs for testing resources are low, pay per use, and reduced license fees Access to global markets Market base for both providers and customer becomes larger FlexibilityA customer can obtain testing services only when needed, and pay only for what is used Improved testingTesting of certain quality attributes (e.g. scalability, performance) Challenges Test data management Who owns data? Where is it stored? Project and change management How to manage testing projects across different platforms, customers and/or providers Service level agreements How reliable are the services? What happens if data is lost? Source: L.M. Riungu, O. Taipale, K. Smolander, “Software Testing as an Online Service: Observations from Practice, ” 3rd International Conference on Software Testing, Verification and Validation Workshops, 2010.

9. Effect of Testing in the Cloud (1)  On actual testing work  More efficient performance testing – e.g. running 40 Amazon instances for generating about 20,000 realistic user experiences  Quicker testing – e.g. being able to test run a developed feature in say 10 different environments and getting (almost) immediate results  More realistic testing results Source: L. Riungu-Kalliosaari, O. Taipale, K. Smolander, “Testing in the Cloud: Exploring the Practice, Accepted, ” Special issue on Software Engineering for Cloud Computing, IEEE Software, March/April 2012.

10. Effect of Testing in the Cloud (2)  On support and delivery of testing services  Better availability of testing tools and options – e.g. being able to carry out technical feasibility studies when choosing the tools to use.  Improved developer-tester communication  due to equal access to the system  Enhanced service delivery for vendors – quicker time to market and meeting of customer demands Source: L. Riungu-Kalliosaari, O. Taipale, K. Smolander, “Testing in the Cloud: Exploring the Practice, Accepted, ” Special issue on Software Engineering for Cloud Computing, IEEE Software, March/April 2012.

11. Roadmap towards testing in the Cloud  Develop an understanding of cloud computing Understand the risks and prepare to address them.  Carry out pilot projects. Explore the viability of testing in the cloud and the potential benefits.  Come up with elaborate strategies For example, criteria for the selection of applications suitable for cloud-based testing; criteria for the selection of potential cloud vendors.  Enhance team interaction and prepare for complexities Organizations need to be prepared for additional testing brought about by the complexities and new requirements for cloud-based applications and systems.  Enhance co-operation between research and industry Focus on addressing cloud related issues that are relevant for the software industry (including testing) Source: L. Riungu-Kalliosaari, O. Taipale, K. Smolander, “Testing in the Cloud: Exploring the Practice, Accepted, ” Special issue on Software Engineering for Cloud Computing, IEEE Software, March/April 2012.

12. Strategy for Testing in the Cloud  Evaluate the line of business  Critical vs. non-critical  Organization size  Assess the need  Goals  Security risks/threats  Trade-offs  Identify and select  The delivery approach  The cloud service provider  Service level agreements (SLAs), Terms of service (TOS) etc  Utilize the cloud service i.e. test  Pay attention to security, SLAs  Re-evaluate  Achieved vs. non-achieved needs, security, tradeoffs, maintainability  Change management e.g. skills development

13. Aspects of Security (1)  Trust  An entity A is considered to trust another entity B when entity A believes that entity B will behave exactly as expected and required (Artz, et al., 2011)  Level of certainty to the customer that the cloud provider is capable of providing the subscribed service properly and accurately  Governance  Management and control over policies, defining roles and responsibilities, standards for application development and special attention for managing security risks/threats (CSA 2009)  Design, identification and implementation of organizational structures along with monitoring, control and testing of deployed services in the public cloud (Jansen, et al., 2011)  Compliance  Compliance is the process of ensuring adherence to policies derived from internal directives, procedures and requirements, or from external laws, regulations, standards and agreements (Proctor, 2011).  Involves measuring the effectiveness and adherence of the rules and understanding the followed process.  Cloud providers need to provide assurance and proof to the subscribers that they have control over security. Customers need to verify their own internal security measures with their own auditors.  Identity and Access Management  Provision of privacy and protection of data sensitivity  Who has access to the data?

14. Aspects of Security (2)  Availability  Service interruption e.g. g-mail had one-day outage in 2008.  Distributed denial of services (DDOS) - servers and networks are brought down by the flood of network traffic and prevent users to access the internet based services  Incompatibility between the cloud provider’s storage services and applications that need to be tested  Data Security  Data protection and confidentiality especially in shared multi-tenant environments  Change management e.g. skills development  Instance Isolation and its Failure  Ensure that different instances running on the same physical machine are detached from each other (Ertaul, et al., 2009).  Architecture  A public cloud may enable one vendor’s SaaS to be hosted within some other vendor’s PaaS or IaaS service.  Nested hosting  platform and network risks  lack of transparency between the customers and actual point of operations even during testing (Lumley, 2010).

15. Security Approach (1)

16. Security Approach (2)  Define a strategic cloud security roadmap  Mainly guided by the requirements of the organization  Evaluate the cloud provider’s risks and various types of risk assessment methods  Define the business and IT strategy  Evaluate the information: what can be public/private?  The organization type  Identify the risks  Point out the risks, threats and vulnerabilities  Design some initial control mechanisms to deal with the risks  Document the plan  Detail all the important aspects in a plan that can be disseminated and effectively communicated  Assess the cloud security requirements  Map the customer’s security needs to the provider’s ability to meet them  Identify the gaps and how to resolve them

17. Comparing security strategies of popular cloud providers CSP/Security Issues AmazonGoogle App Engine Microsoft Azure Rackspace TrustVery strongStrong Very strong GovernanceStrongSatisfiedStrongSatisfied ComplianceStrongSatisfiedStrong Identity and Access Management Strong AvailabilityStrong Data SecurityStrong Very strong Instance isolation and its failure Strong ArchitectureStrong

18. Work in Progress  Software testing standard 29119 Part 4 contain information on how to map quality characteristics to test design techniques and how to map quality characteristics to types of testing  This is used as guide and reference in this study.  Evaluating the important quality attributes as expressed by organizations that are delivering cloud services  The study contains 5 organizations interviewed during Autumn 2011