N ORTH D AKOTA S TATE U NIVERSITY D EPARTMENT OF C OMPUTER S CIENCE © NDSU S OFTWARE T ESTING R ESEARCH G ROUP Safe Runtime Validation of Behavioral Adaptations in Autonomic Software Presenter: T ariq M. King ATC 2011 September 2-4, Banff, Alberta, Canada

© NDSU S OFTWARE T ESTING R ESEARCH G ROUP Outline  Motivation  Introduction  Testing Approach  Component Design  Prototype  Related Work  Conclusion 2

© NDSU S OFTWARE T ESTING R ESEARCH G ROUP Motivation  Self-* features in autonomic computing systems may incorporate dynamic software adaptation  System adds, removes, replaces its own components at runtime  Raises reliability concerns as new faults may be introduced due to dynamic adaptation  Most AC research neglects that self-test should be an integral part of these types of systems 3 A NEGLECTED SELF-* CHARACTERISTIC Self- Configure Self- Optimize Self- Heal Self- Protect Self- Test FAULTS? FAILURES?

© NDSU S OFTWARE T ESTING R ESEARCH G ROUP  To address this issue, we introduced the notion of Autonomic Self-Testing (King et. al 2007)  Idea: Investigate how Autonomic Managers (AMs) can be tailored for purpose of testing the system itself  Test Managers (TMs) Monitor, intercept and validate the adaptive changes requests of AMs Managed Resource Introduction 4 AUTONOMIC SELF-TESTING Monitor AnalyzePlan Execute Knowledge Effector Sensor Autonomic Manager Monitor AnalyzePlan Execute Knowledge Autonomic Manager Test Manager

© NDSU S OFTWARE T ESTING R ESEARCH G ROUP Initial work on AST concentrated on:  Defining validation strategies for TMs 1.Replication with Validation (RV) – tests changes using separate copies of managed resources 2.Safe Adaptation with Validation (SAV) – tests changes in-place, directly on managed resources  Developing prototypes to investigate the feasibility of the RV strategy  Lightweight, focused on localized validation Introduction 5 AUTONOMIC SELF-TESTING (CONT.)

© NDSU S OFTWARE T ESTING R ESEARCH G ROUP In this research, we extend previous work on AST by overcoming two of its limitations. Major Contributions: 1.Addresses the need for system-wide validation in autonomic software through the description of an automated runtime integration testing approach 2.Investigates AST of a real-world application in which it is too expensive to maintain copies of managed resources for testing – Validates SAV Introduction 6 CONTRIBUTIONS

© NDSU S OFTWARE T ESTING R ESEARCH G ROUP S ystem-Wide V alidation of B ehavioral A daptations Using S elf- T estable A utonomic C omponents (STACS) Overview of Testing Approach 7

© NDSU S OFTWARE T ESTING R ESEARCH G ROUP State-Based Safety Model 8 1. Extends Zhang (2004) “Enabling Safe Component-Based Software Adaptation” in WADS ‘04

© NDSU S OFTWARE T ESTING R ESEARCH G ROUP S elf- T estable A utonomic C omponent (T, A, R, I, K) Component Design 9 STAC DEFINITION

© NDSU S OFTWARE T ESTING R ESEARCH G ROUP  Abstract MAPE  Generics Managers (T, A) 10  Multi-Threading  Safety Mechanisms

© NDSU S OFTWARE T ESTING R ESEARCH G ROUP Full Access to Source Code  Derive suspend, resume, save and restore state operations from AbstractResource class  Loose coupling of concrete objects to facilitate stub injection at runtime Limited Access to Source Code  Use wrapper class (carefully)  Research on improving COTS testability Resources (R) 11

© NDSU S OFTWARE T ESTING R ESEARCH G ROUP Test Interface (I T ) 12  Test Case Execution  Code Coverage Analysis  Performance

© NDSU S OFTWARE T ESTING R ESEARCH G ROUP  Communication Virtual Machine (CVM) is a model-driven platform for realizing user- centric communication services  Collaboration: FIU & Miami Children’s Hospital  CVM separates the concerns of modeling, synthesis, coordination, and delivery of communication services into self-contained layers  Layer in which communication in communication takes place is called the Network Communication Broker (NCB). Prototype 13 COMMUNICATION VIRTUAL MACHINE

© NDSU S OFTWARE T ESTING R ESEARCH G ROUP  Allen (2009) leverage AC and open-platform APIs for integrating communication services into NCB Prototype 14 COMMUNICATION VIRTUAL MACHINE CVM NCB LAYER OF

© NDSU S OFTWARE T ESTING R ESEARCH G ROUP  Conducted two sets of experimental runs as part of the prototype evaluation:  Test Quality Experiments – compared the effectiveness of test sets in revealing faults and exercising the NCB program code  Performance Experiments – investigates ways to reduce the performance overhead of having a runtime testing process interleaved with the NCB  In both cases, a Mutation Analysis technique was used to simulate faulty change requests, induce self-testing, and produce the results Experiment Overview 15

© NDSU S OFTWARE T ESTING R ESEARCH G ROUP  39 mutants were generated for the prototype by planting artificial faults into the interfaces and/or implementations of Skype and Smack  15 created manually, 24 automatically generated  Test Support Tools: JUnit, Cobertura, MuJava, Eclipse TPTP  Experiments were performed on a windows-based, webcam-enabled Intel® Core 2 Duo 2.4GHz PCs with 2 GB RAM or greater Setup Environment 16

© NDSU S OFTWARE T ESTING R ESEARCH G ROUP  Fault Detection  35 out of the 39 mutants were detected, producing a mutation score of 89.7% (Threat, CVM Dev/Test)  Code Coverage  Recorded 63% statement coverage and 57% branch coverage of the NCB implementation  100% method coverage of the communication service interfaces (SkypeAdapter, SmackAdapter) Results 17 TEST QUALITY

© NDSU S OFTWARE T ESTING R ESEARCH G ROUP Results (cont.) 18 PERFORMANCE SKYPE TWO-WAY TO SMACK THREE-WAY

© NDSU S OFTWARE T ESTING R ESEARCH G ROUP  System-wide SAV is feasible and beneficial for preventing runtime faults in the context of a real- world application  Testing adaptive changes in-place using TMs was challenging: high complexity, synchronization  Significant performance overhead, further research  Safety mechanisms (locks) were instrumental during development and runtime testing  Selecting and tailoring open-source testing tools presented unnecessary difficulties Lessons Learned 19

© NDSU S OFTWARE T ESTING R ESEARCH G ROUP  Da Costa et al. (2010)  JAAF+T: Java Self-Adaptive Framework + Test  Modify MAPE structure to include a self-test activity  Stevens et al. (2007) and Ramirez et al. (2008)  Prototypes of Replication with Validation  Autonomic Container & Autonomic Job Scheduler  Zhang et al. (2009) and Zhao et al. (2005)  Runtime verification of self-* systems Related Work 20

© NDSU S OFTWARE T ESTING R ESEARCH G ROUP  Presented a system-wide approach to runtime integration testing in autonomic software  Still many open research issues:  Dynamic test generation  Dynamic regression test scheduling  Runtime test case maintenance  Reducing overhead of runtime testing  Latest Direction: Cloud (CASCON 2011) Conclusion 21

© NDSU S OFTWARE T ESTING R ESEARCH G ROUP Acknowledgements 22  Dr. Masoud Milani, FIU  Dr. S. Masoud Sadjadi, FIU  Participants of the REU Summer 2006 and 2007 Programs on Autonomic Computing at FIU  Graduate students and research collaborators on CVM project This work has been supported in part by the NSF under grant IIS

© NDSU S OFTWARE T ESTING R ESEARCH G ROUP Thank You! 23 Questions? ¿Preguntas? 問題 Sawwal вопросы 質問 domande ερωτήσεις