Shufeng Wang National Laboratory for Parallel and Distributed Processing National University of Defense Technology Changsha, 410073, China

Slides:



Advertisements
Similar presentations
Author: Carlos Pacheco, Shuvendu K. Lahiri, Michael D. Ernst, Thomas Ball MIT CSAIL.
Advertisements

Hong Zhu Department of Computing and Communication Technologies Oxford Brookes University, Oxford OX33 1HX, UK COMPSAC 2012 PANEL.
Translation-Based Compositional Reasoning for Software Systems Fei Xie and James C. Browne Robert P. Kurshan Cadence Design Systems.
Towards Self-Testing in Autonomic Computing Systems Tariq M. King, Djuradj Babich, Jonatan Alava, and Peter J. Clarke Software Testing Research Group Florida.
Some questions o What are the appropriate control philosophies for Complex Manufacturing systems? Why????Holonic Manufacturing system o Is Object -Oriented.
ASTA Aspect Software Testing Assistant Juha Gustafsson, Juha Taina, Jukka Viljamaa University of Helsinki.
Paraμ A Partial and Higher-Order Mutation Tool with Concurrency Operators Pratyusha Madiraju AdVanced Empirical Software Testing and Analysis (AVESTA)
Alternate Software Development Methodologies
Presented by: Thabet Kacem Spring Outline Contributions Introduction Proposed Approach Related Work Reconception of ADLs XTEAM Tool Chain Discussion.
1 Concurrency Specification. 2 Outline 4 Issues in concurrent systems 4 Programming language support for concurrency 4 Concurrency analysis - A specification.
Pedro Reales Mateo University of Castilla-La Mancha Alarcos Research Group University of Castilla-La Mancha Macario Polo Usaola University of Castilla-La.
Adding Organizations and Roles as Primitives to the JADE Framework NORMAS’08 Normative Multi Agent Systems, Matteo Baldoni 1, Valerio Genovese 1, Roberto.
NaLIX: A Generic Natural Language Search Environment for XML Data Presented by: Erik Mathisen 02/12/2008.
SE 450 Software Processes & Product Metrics Reliability: An Introduction.
An Intelligent Broker Approach to Semantics-based Service Composition Yufeng Zhang National Lab. for Parallel and Distributed Processing Department of.
Software Issues Derived from Dr. Fawcett’s Slides Phil Pratt-Szeliga Fall 2009.
1 FM Overview of Adaptation. 2 FM RAPIDware: Component-Based Design of Adaptive and Dependable Middleware Project Investigators: Philip McKinley, Kurt.
Introduction to Software Testing
An Agile View of Process
Workflow API and workflow services A case study of biodiversity analysis using Windows Workflow Foundation Boris Milašinović Faculty of Electrical Engineering.
Formal Techniques for Verification Using SystemC By Nasir Mahmood.
©Ian Sommerville 2004Software Engineering, 7th edition. Chapter 18 Slide 1 Software Reuse 2.
Dr. Pedro Mejia Alvarez Software Testing Slide 1 Software Testing: Building Test Cases.
A Survey of Software Refactoring Tom Mens, Tom Tourwé
Lecture 6 Software Testing and jUnit CS140 Dick Steflik.
1 Autonomic Computing An Introduction Guenter Kickinger.
Chapter 2 The process Process, Methods, and Tools
© 2012 IBM Corporation Rational Insight | Back to Basis Series Chao Zhang Unit Testing.
C++ Code Analysis: an Open Architecture for the Verification of Coding Rules Paolo Tonella ITC-irst, Centro per la Ricerca Scientifica e Tecnologica
Software Engineering Prof. Dr. Bertrand Meyer March 2007 – June 2007 Chair of Software Engineering Static program checking and verification Slides: Based.
Zhiyong Wang In cooperation with Sisi Zlatanova
Chapter 8 – Software Testing Lecture 1 1Chapter 8 Software testing The bearing of a child takes nine months, no matter how many women are assigned. Many.
Mihir Daptardar Software Engineering 577b Center for Systems and Software Engineering (CSSE) Viterbi School of Engineering 1.
AN EXTENDED OPENMP TARGETING ON THE HYBRID ARCHITECTURE OF SMP-CLUSTER Author : Y. Zhao 、 C. Hu 、 S. Wang 、 S. Zhang Source : Proceedings of the 2nd IASTED.
Composing Adaptive Software Authors Philip K. McKinley, Seyed Masoud Sadjadi, Eric P. Kasten, Betty H.C. Cheng Presented by Ana Rodriguez June 21, 2006.
WSMX Execution Semantics Executable Software Specification Eyal Oren DERI
11 CORE Architecture Mauro Bruno, Monica Scannapieco, Carlo Vaccari, Giulia Vaste Antonino Virgillito, Diego Zardetto (Istat)
Verification and Validation in the Context of Domain-Specific Modelling Janne Merilinna.
The roots of innovation Future and Emerging Technologies (FET) Future and Emerging Technologies (FET) The roots of innovation Proactive initiative on:
Test Drivers and Stubs More Unit Testing Test Drivers and Stubs CEN 5076 Class 11 – 11/14.
1 Introduction to Software Engineering Lecture 1.
Hong Zhu Dept of Computing and Communication Technologies Oxford Brookes University Oxford, OX33 1HX, UK TOWARDS.
FDT Foil no 1 On Methodology from Domain to System Descriptions by Rolv Bræk NTNU Workshop on Philosophy and Applicablitiy of Formal Languages Geneve 15.
1 CSCD 326 Data Structures I Software Design. 2 The Software Life Cycle 1. Specification 2. Design 3. Risk Analysis 4. Verification 5. Coding 6. Testing.
CPSC 873 John D. McGregor Session 9 Testing Vocabulary.
Testing OO software. State Based Testing State machine: implementation-independent specification (model) of the dynamic behaviour of the system State:
Course: COMS-E6125 Professor: Gail E. Kaiser Student: Shanghao Li (sl2967)
August 2003 At A Glance The IRC is a platform independent, extensible, and adaptive framework that provides robust, interactive, and distributed control.
CPSC 871 John D. McGregor Module 8 Session 1 Testing.
Survey of Tools to Support Safe Adaptation with Validation Alain Esteva-Ramirez School of Computing and Information Sciences Florida International University.
Concepts and Realization of a Diagram Editor Generator Based on Hypergraph Transformation Author: Mark Minas Presenter: Song Gu.
Banaras Hindu University. A Course on Software Reuse by Design Patterns and Frameworks.
1 HPJAVA I.K.UJJWAL 07M11A1217 Dept. of Information Technology B.S.I.T.
Whole Test Suite Generation. Abstract Not all bugs lead to program crashes, and not always is there a formal specification to check the correctness of.
From Use Cases to Implementation 1. Structural and Behavioral Aspects of Collaborations  Two aspects of Collaborations Structural – specifies the static.
CS223: Software Engineering Lecture 18: The XP. Recap Introduction to Agile Methodology Customer centric approach Issues of Agile methodology Where to.
A Framework for Automated and Composable Testing of Component-based Services Miguel A. Jiménez, Ángela Villota, Norha M. Villegas, Gabriel Tamura, Laurence.
Tool Support for Testing Classify different types of test tools according to their purpose Explain the benefits of using test tools.
Copyright 1999 G.v. Bochmann ELG 7186C ch.1 1 Course Notes ELG 7186C Formal Methods for the Development of Real-Time System Applications Gregor v. Bochmann.
From Use Cases to Implementation 1. Mapping Requirements Directly to Design and Code  For many, if not most, of our requirements it is relatively easy.
CPSC 372 John D. McGregor Module 8 Session 1 Testing.
Laurea Triennale in Informatica – Corso di Ingegneria del Software I – A.A. 2006/2007 Andrea Polini XVIII. Software Testing.
Automated Software Testing
Testing Tutorial 7.
Generating Automated Tests from Behavior Models
Definition CASE tools are software systems that are intended to provide automated support for routine activities in the software process such as editing.
John D. McGregor Session 9 Testing Vocabulary
Software Testing An Introduction.
Introduction to Software Testing
Test Case Purification for Improving Fault Localization
Presentation transcript:

Shufeng Wang National Laboratory for Parallel and Distributed Processing National University of Defense Technology Changsha, , China Hong Zhu Dept of Computing and Communication Technologies Oxford Brookes University Oxford, OX33 1HX, UK CATest: A Test Automation Framework for Multi-agent Systems

OUTLINE  Motivation  Review of the current state of art  Overview of the proposed framework  Prototype tool CATest  Experiments results  Conclusion and further works

MOTIVATION  Software test automation  Testing is labour intensive and expensive  Test automation is imperative to reduce the cost and improve the effectiveness of testing  A great amount of research efforts has been reported and a significant progress has been made  Test automation has become a common practice in IT industry  Agent-oriented software development methodologies  Agents are autonomous, active and collaborative computational entities (such as services)  Widely perceived as a promising new paradigm suitable for the Internet-based computing  Extremely difficult to test  poor on both controllability and observability aspects of software testability  Research question  Can automated testing tools deal with the complexity of agent-oriented systems?

TEST AUTOMATION FRAMEWORKS (TAFs)  A TAF provides a facility for setting up the environment in which test methods and assertion methods are executed and enables test results to be reported.  associating each program unit (e.g. class) with a test unit that contains a collection of test methods, each for a test.  specifying the expected test results for each test in the form of calls to assertion methods in the test class;  aggregating a collection of tests into test suites that can be run as a single operation by calling the test methods;  executing test suites and reporting the results when the code of the program is tested. For OO programming languages, the test unit is a declared as a subclass of the class under test and called test class.

ARCHITECTURE OF TAFs Static View of Test Automation FrameworksDynamic View of Test Automation Frameworks [Meszaros, G., Xunit Test Patterns, Addison Wesley, 2007]

THE CURRENT STATE OF ART  Test Automation frameworks  Best practice of test automation in IT industry  Wide range of products are available, some as open source, e.g.,  JUnit for testing software written in Java  CppUnit for C++, NUnit for.NET  RUnit for Ruby, PyUnit for Python  VbUnit for Visual Basic  Selenium for Web Services, etc.  TAFs can significantly reduce test costs and increase test efficiency, especially when  the program code is revised frequently  testing is repeated for many times  in agile development processes  The test code is a valuable and tangible asset and can be sold to component customers

WEAKNESS OF EXISTING TAFs  Manual coding of test classes  write test code to represent test cases in test methods  translate specification into assertion methods  Lack of support to the measurement of test adequacy  There is no facility in the existing TAFs that enables the measurement of test adequacy.  Weak in the support to correctness checking  The assertion methods can only access the local variables and methods of the unit under test.  Implications:  correctness cannot be checked against the context in which the unit is called  correctness checking cannot across multiple executions of the unit under test This is not only labour intensive, but also error prone. Well, it is doable, but needs advanced programming to achieve this.

TESTING AGENT-BASED SOFTWARE  Research on testing agent-based systems have addressed the following aspects of testing MAS  correctness of interaction and communication [6]–[10]  correctness of processing internal states [11]–[14]  generation of test cases [12], [15]  control of test executions [16]–[18]  Adequacy criteria  Low et al. [1999] proposed a set of coverage criteria defined on the structure of plans for testing BDI (Belief-Desire-Intention) agents.  Test automation frameworks  SUnit for Seagent by extending JUnit [17]  JAT for Jade [7]  the testing facility in INGENIAS [18]  in Prometheus methodology [13] All of these are extensions of OO TAFs with slight additional features of agents.

WHY NEED A NEW TYPE OF TAFs  Insufficient support to correctness checking:  What the facility supports:  The mechanism replies on the internal information of the unit under test (i.e. object or agent) and the data at a single time point  What we require:  Agents are autonomous, proactive, context-aware and adaptive  They often deliver the functionality through emergent behaviours that involve multiple agents  The specifications of the required behaviours in a MAS are often hard to translate into assertion methods manually  Most MAS are continuous running systems.  to determine when to stop a test execution  to measure test adequacy during testing executions The correctness of agent’s behaviours must be judged in the context of the dynamic and open environments the histories that agents have experienced in previous executions

PROPOSED APPROACH 1. Division of testing objectives into 4 layers  Infrastructure level Devoting to the validation and verification of the correctness of the implementation of the infrastructure facilities that support agent communication and interactions  Caste level Focusing on validating and verifying the correctness of each individual agent’s behaviour  Cluster level Aiming at validating and verifying the correctness of the behaviours of a group of agents in interaction and collaboration processes  Global level Aiming at validating and verifying the correctness of the whole system’s behaviour, especially the emergent behaviour Equivalent to class in object-orientation

KEY COMPONENTS OF THE ARCHITECTURE  Runtime facility for behavior observation  A library provides support to the observation of the dynamic behaviors  Invocations of the library methods are inserted into the source code  When the AUT is executed, its behavior is observed and recorded  It enables both correctness checking and adequacy measurement  Test oracle  Takes a formal specification or model and recorded behaviors as input  Checks automatically the correctness of the recorded behaviors against the formal specification  Generic test coverage calculator  Takes a formal specification and a set of recorded behavior as input  Translates formal specification into test requirements according to user selected test adequacy criteria  Calculates specification coverage while checking the correctness  Test execution controller  Runs the coverage calculator in parallel to the system under test  Stops one test when an elemental adequacy criterion is satisfied  Stops the whole testing when satisfies a collective adequacy criterion In SLABS (Specification Language for Agent-Bases Systems)

A QUICK OVERVIEW OF SLABS Behaviour rules are in the form of Agents are instances of castes; An agent can be multiple castes; Agents can dynamically change their casteships by joining or quitting a caste; Environment determines a set of other agents in the system whose behaviour are the input to the specified agent For the sake of simplicity, here we write in the following form. See [Zhu 2001] for details.

TEST ADEQUACY CRITERIA  A set of adequacy criteria have been defined and implemented based on guard-condition semantics of behavior rules  The criteria have the following subsumption relations

CATEST: TAFs FOR CASTE LEVEL TESTING Architecture of CATest

CATest UGI For Set Test Parameters

CATest GUI: Report Test Results

EXPERIMENTS: THE SUBJECTS

EXPERIMENTS: PROCESS 1.Generation of mutants The muJava testing tool is used to generate mutants of the Java class that implements the caste under test. 2.Analysis of mutants Each mutant is compiled and those contain syntax errors are deleted. Those equivalent to the original are also removed. 3.Test on mutants The original class is replaced by the mutants one by one and tested using our tool. The test cases were generated at random. The test executions stop when the Rule Coverage Criterion is satisfied, or the execution stops abnormally when an interrupting exception occurs. 4.Classification of mutants A mutant is regarded as killed if an error is detected, i.e. when the specification is violated. Otherwise, the mutant is regarded as alive. This is different from traditional definition of dead mutants, which does not work because the non-deterministic nature of the system.

EXPERIMENTS: RESULTS

ANALYSIS OF EXPERIMENT RESULTS  Observations:  Mutants that represent faults at the caste level, such as in the behaviour rules, are detected 100% in our experiments using the rule coverage criterion.  The kinds of mutants that are not killed  Mutants that change the code that initializes the agent’s state  Mutants that change the code that sends/receives messages to/from the others agents  Mutants that change the code inside the functions/ methods of actions  Mutants that change the infrastructure code  Conclusions:  The method works well at caste level  Testing at other levels are necessary These mutants correspond to faults that are either at a higher or a lower level than caste level.

CONCLUSION AND FURTHER WORKS  Proposed a novel architecture of TAFs  Presented a prototype tool CATest for testing MAS  Conducted experiments with the CATest tool  Key features:  It automatically checks the correctness of software dynamic behaviours against formal specifications without the need to manually write assertion methods.  It fully supports automatic measurement of test adequacy and use the adequacy measurement to control test executions.  Applicability:  All levels of MAS testing  Can be easily adapted for testing OO software. Note: We have developed a test environment called CATE-Test that supports all levels of agent test. CATest is a part of CATE-Test. Work in Progress: Experiments with MAS testing at other levels. Note: 1. Overcoming the weakness of existing TAFs. 2. Test cases generation is not a part of the TAF, but can be easily integrated to the framework. Main Contribution 1: Architecture of TAFs: Further work: Experiments in larger scale

MAIN CONTRIBUTION 2: TESTING MAS  Proposed a new hierarchy of adequacy criteria for specification-based testing  Implemented these adequacy criteria in the CATest tool  Key features:  Treat guard-conditions differently from pre/post-conditions  Reflect better the semantics of guard conditions in testing  Take full consideration of non-determinism  Applicability  Applicable to MAS at caste level  All systems that  are running continuously, non-deterministically and event-driven  specified by a set of behaviour rules with guard-conditions  e.g. distributed and service-oriented systems Note: Other levels will need different adequacy criteria Work in Progress: Study of adequacy criteria and their effectiveness in detecting faults at other levels. Future Work : Testing service-oriented systems: TAFs and adaptation of the adequacy criteria

THANK YOU Questions?