1. N ORTH D AKOTA S TATE U NIVERSITY D EPARTMENT OF C OMPUTER S CIENCE © NDSU S OFTWARE T ESTING R ESEARCH G ROUP Tariq M. King STITC 2011 Co-located with Towards a Self-Testing Cloud Computing Infrastructure Markham, Ontario, Canada

2. © NDSU S OFTWARE T ESTING R ESEARCH G ROUP Outline  Introduction  Research Questions  Cloud Testability  Test Support as-a-Service  Vision: Self-Testing in the Cloud  Summary  Open Discussion 2

3. © NDSU S OFTWARE T ESTING R ESEARCH G ROUP  Cloud provides ubiquitous, on-demand access to computing resources “as services” via the Internet  Operational Perspective  Availability, Scalability, Efficiency, …  Development Perspective  Reusable Components, Info. Hiding  Highly Complex Local Infrastructure  Testing Perspective? Introduction 3 CLOUD

4. © NDSU S OFTWARE T ESTING R ESEARCH G ROUP  Cloud brings new testing challenges and benefits  Software Testing In the Cloud (STITC)  STITC 2010 Workshop focused on three areas:  Testing IN the Cloud – leveraging the cloud for test execution and testing in a virtualized environment  Testing OF the Cloud – validating applications that are hosted and deployed cloud  Testing TO the Cloud – moving the testing process and other assets to the cloud Introduction 4 STITC

5. © NDSU S OFTWARE T ESTING R ESEARCH G ROUP  Services in cloud computing typically fall into three categories: Software-as-a-Service | Google Docs, Office Web Apps Platform-as-a-Service | Google App Engine, Windows Azure Infrastructure-as-a-Service | Amazon EC 2, Rackspace Introduction 5 CLOUD SERVICES

6. © NDSU S OFTWARE T ESTING R ESEARCH G ROUP  Many cloud platforms offer Virtualization support  Creating abstract computing resources from more powerful physical resources Hosted Architecture Introduction 6 VIRTUALIZATION Hypervisor Architecture

7. © NDSU S OFTWARE T ESTING R ESEARCH G ROUP Introduction  Cloud Computing shares characteristics with Autonomic Computing (AC), Grid, Utility, …  Central theme of AC is self-management to reduce burden of human administration and integration  King et. al 2008 introduce an implicit self-test characteristic  Regardless of self-* features implemented, system should validate adaptations at runtime 7 AUTONOMIC COMPUTING Self- Configure Self- Optimize Self- Heal Self- Protect Self- Test

8. © NDSU S OFTWARE T ESTING R ESEARCH G ROUP  “Migrating Autonomic Self-Testing to the Cloud”  Tariq M. King and Annaji S. Ganti | STITC 2010  High-level position on how a self-testing approach for autonomic software can be applied to the cloud  In today’s talk, we highlight our recent progress and developments from the above perspective  Extended version of King et al. | CASCON 2011  Summarizes contents of book chapter submitted to STITC: Perspectives on an Emerging Discipline Introduction 8 ABOUT THIS TALK

9. © NDSU S OFTWARE T ESTING R ESEARCH G ROUP  How does cloud computing affect our ability to test applications that are either built or deployed in a cloud environment, or depend on cloud services? Research Question # 1 9 Testability Investigates impact of cloud development model on software testability (Positive, Negative)

10. © NDSU S OFTWARE T ESTING R ESEARCH G ROUP  How can the test artifacts and tools used to validate a cloud application service during development, be reused to help test derivative applications? Research Question # 2 10 Test Reuse Investigates ways to reuse outputs of pre-deployment testing effort (Artifacts, Tooling)

11. © NDSU S OFTWARE T ESTING R ESEARCH G ROUP  How can built-in cloud computing technologies such as virtualization be used to improve the process of testing cloud applications? Research Question # 3 11 Cloud Leverage Investigates ways in which the cloud infrastructure itself can be harnessed for Testing OF the Cloud (Power, Virtualization)

12. © NDSU S OFTWARE T ESTING R ESEARCH G ROUP Cloud Testability: The Negative 12 Complexity Remoteness Information Hiding AutonomyAdaptiveness Statefulness High Performance Criteria High Dependability Criteria (Paradigm Infancy)

13. © NDSU S OFTWARE T ESTING R ESEARCH G ROUP 13 Computational Power Storage Testing requires lots of resources and the cloud is certainly powerful enough to handle it Virtualization Cloud Testability: The Positive

14. © NDSU S OFTWARE T ESTING R ESEARCH G ROUP  As a step towards Research Questions 2 and 3, we investigate a novel approach to software testing of the cloud, in the cloud | King et al., CASCON ‘ 11  Scenario: One host (Provider B) is developing an application service that will extend functionality of a service offered by another host (Provider A) Testing of the Cloud, In the Cloud 14

15. © NDSU S OFTWARE T ESTING R ESEARCH G ROUP Integration Testing Challenges 15

16. © NDSU S OFTWARE T ESTING R ESEARCH G ROUP  Our proposed solution introduces the notion of Test Support as-a-Service (TSaaS) in the cloud  Prior to the deployment of Service A, Provider A would have tested it, and created:  Test Artifacts – Test Drivers, Test Stubs, Test Data  Local Test Bed – Hardware/Software Environment  TSaaS seeks to reuse existing test artifacts, tools and configurations of the local test bed, to aid the rapid development and delivery of test support services for collaborating partners Proposed Solution 16

17. © NDSU S OFTWARE T ESTING R ESEARCH G ROUP  Provider B has full access to a test copy of Service A  Test hooks, instrumentation, and scaffolding reuse  Test data (non-confidential) can be made available  Testing does not interrupt service in production  Concerns: Maintenance, Scalability, Security Test Support as-a-Service (TSaaS) 17 Provider A exposes TSaaS to Provider B

18. © NDSU S OFTWARE T ESTING R ESEARCH G ROUP Hypervisor-Based Virtualization and Secure Access TSaaS Architecture 18

19. © NDSU S OFTWARE T ESTING R ESEARCH G ROUP TSaaS Control Flow 19 Control Flow When Realizing Test Support Services

20. © NDSU S OFTWARE T ESTING R ESEARCH G ROUP Using Existing Test Automation to Guide Development TSaaS Development Method 20

21. © NDSU S OFTWARE T ESTING R ESEARCH G ROUP  Designed with goal of making its components easily deployable to Windows Azure ®  Web & Worker Roles (ASP.NET, IIS 7, WCF Services)  VM Role (Windows 2008 R 2 Server Image)  Application Domain: Credit Reporting/Banking TSaaS Prototype 21 DETAILED DESIGN

22. © NDSU S OFTWARE T ESTING R ESEARCH G ROUP  We were able to develop and expose a cross section of test support operations via the TSaaS API  Test Specification – user defines fixture operations  Test Execution – runs specified fixture operations  Code Coverage – analyzes branch and lines covered  Server-Side Diagnostics – facilitates configuring web server, including performing a reset after failures  Runtime Virtualization – create, start, stop, delete, take snapshot, and update VHD configuration TSaaS Prototype 22 PROGRAMMING INTERFACE

23. © NDSU S OFTWARE T ESTING R ESEARCH G ROUP TSaaS Prototype 23 GRAPHICAL USER INTERFACE

24. © NDSU S OFTWARE T ESTING R ESEARCH G ROUP Vision: Self-Testing In The Cloud 24 Service A (Production Env.) Monitor AnalyzePlan Execute Knowledge Autonomic Test Manager TSaaS Sensor Provider A Provider B Test Harness Service B (Test Environment) Service B (Test Environment) Provider A builds a sensor for Service A Provider B monitors Service A for changes If a change occurs, Provider B generates a plan for regression testing of Service B Provider B reruns tests on latest build of Service B, which is set to use new version of Service A via TSaaS

25. © NDSU S OFTWARE T ESTING R ESEARCH G ROUP Test Support as-a-Service  Improved Testability and Confidence (Partner)  Control/Observation, White Box Testing, Metrics  Improved Product (Provider)  Partners test your component, bug reporting/fixes  Virtualization Benefits  Snapshots: Speed Up, Bug Replication/Debugging  Scalability, Ease of Maintenance, Hardware Savings, and Environment Testing (OS, Applications) Foreseen Benefits 25

26. © NDSU S OFTWARE T ESTING R ESEARCH G ROUP Self-Testing in the Cloud via TSaaS  Automatic Regression Testing of changes due to:  Software maintenance and evolution  Dynamic adaptation and updating  Improved Autonomic Decision-Making  Use of validation to determine which service may be the best candidate to select during adaptation  As a proactive form of self-healing (fault detection through formalized testing techniques) Foreseen Benefits (cont.) 26

27. © NDSU S OFTWARE T ESTING R ESEARCH G ROUP  Standardization  Providing standardized interfaces and guidelines for performing testing in the cloud, and developing test support infrastructures  Evaluation  Providing convincing evidence that these approaches are not only feasible to implement but can have real value to practitioners  Validation  Showing virtualization is consistent with production Key Challenges 27

28. © NDSU S OFTWARE T ESTING R ESEARCH G ROUP  Described our progress towards our grand vision of migrating autonomic self-testing in the cloud  Proposed an approach to Test Support as-a-Service, which uses the cloud to tackle testing of the cloud  Described the design and implementation of a a prototype of TSaaS  Discussed benefits and key challenges associated with TSaaS and use for self-testing OF/IN the cloud  In terms of our progress, we have barely scratched the surface of the research problem under investigation Summary 28

29. © NDSU S OFTWARE T ESTING R ESEARCH G ROUP Acknowledgements 29  Annaji Ganti | PhD Student, NDSU/Microsoft  Neha Kale | MS Student, NDSU  David Froslie | Test Architect, Microsoft  Alisson Sol | Knowledge Engineer, Microsoft  Dr. Scott Tilley | Professor, Florida Tech  Participants and Reviewers  STITC 2010  CASCON 2011

30. © NDSU S OFTWARE T ESTING R ESEARCH G ROUP Thank You! 30 Questions? ¿Preguntas? 問題 Sawwal вопросы 質問 domande ερωτήσεις