1. Advanced Modeling Model Based Test Generation and Abstract state machine Language Keith Stobie Test Architect XML Web Services Microsoft Corporation

2. 2SASQAG 16Jan03 AsmL Modelling © 2003 Microsoft Corporation, All Rights Reserved What’s a Model? A model: Is an abstraction or simplified representation of the system from a particular perspective Supports investigation, discovery, explanation, prediction, or construction May be expressed as a description, table, graphical diagram, or quantitative mathematical model Is not necessarily comprehensive From: PNSQC 2002 “Modeling: A Picture's Worth 1000 Words” Copyright (c) 2002, Quality Tree Software, Inc.

3. 3SASQAG 16Jan03 AsmL Modelling © 2003 Microsoft Corporation, All Rights Reserved Models in Everyday Life Inspired by :“Modeling: A Picture's Worth 1000 Words” Copyright (c) 2002, Quality Tree Software, Inc.

4. 4SASQAG 16Jan03 AsmL Modelling © 2003 Microsoft Corporation, All Rights Reserved Examples of Models Flow Charts Flow Charts Data Flow Diagrams Data Flow Diagrams Entity- Relationship Diagrams Entity- Relationship Diagrams State Diagrams State Diagrams Deployment Diagrams Deployment Diagrams Class Diagrams Class Diagrams Use Cases Use Cases Activity Diagrams Activity Diagrams State Transition Tables State Transition Tables Inspired by :“Modeling: A Picture's Worth 1000 Words” Copyright (c) 2002, Quality Tree Software, Inc.

5. 5SASQAG 16Jan03 AsmL Modelling © 2003 Microsoft Corporation, All Rights Reserved Model Based Test Generation NOT just model based input generators (pair-wise, random data, grammars, etc.) NOT modeling tools (e.g. UML) set of Test Cases sequence of stimuli expected responses Formal Model Test Generation Directives Generator

6. 6SASQAG 16Jan03 AsmL Modelling © 2003 Microsoft Corporation, All Rights Reserved Modeling ModelingValidation Refinement Verification Specification AsmL Implementation C/C++/C# Product Idea / Specification Are you building the product right ? Are you building the right product? What product are you building?

7. 7SASQAG 16Jan03 AsmL Modelling © 2003 Microsoft Corporation, All Rights Reserved Benefits (GOTCHA) Testing pilot amounted to 10 person months, including the GOTCHA learning curve. This time is less than the time used to originally test the subsystem (12 person months). revealed 2 documentation changes defects 15 of the 18 defects found by the original function test effort would also have been found by this pilot test effort.” From: Using a model-based test generator to test for standard conformance Using a model-based test generator to test for standard conformanceUsing a model-based test generator to test for standard conformance

8. 8SASQAG 16Jan03 AsmL Modelling © 2003 Microsoft Corporation, All Rights Reserved Benefits (TestMaster) “In a typical application of this approach, test engineer productivity has increased by a factor of five to ten over conventional manual approaches (Figure 5)”. From: Model Based Testing Model Based TestingModel Based Testing

9. 9SASQAG 16Jan03 AsmL Modelling © 2003 Microsoft Corporation, All Rights Reserved FSM: Start State & action Start state (Entry Point) FilesShown: 0 FileSelect: NotSelected ExitPoint: False From: Intelligent Test Automation by Harry Robinson Software Testing & Quality Engineering (STQE) magazine. Sept/Oct 2000

10. 10SASQAG 16Jan03 AsmL Modelling © 2003 Microsoft Corporation, All Rights Reserved other States start

11. 11SASQAG 16Jan03 AsmL Modelling © 2003 Microsoft Corporation, All Rights Reserved Simple FSM Model

12. 12SASQAG 16Jan03 AsmL Modelling © 2003 Microsoft Corporation, All Rights Reserved File-in-a-directory Model

13. 13SASQAG 16Jan03 AsmL Modelling © 2003 Microsoft Corporation, All Rights Reserved Create Invert Selection Delete Walk Every State-Change

14. 14SASQAG 16Jan03 AsmL Modelling © 2003 Microsoft Corporation, All Rights Reserved File-in-a-directory Model C2 C1 C3 C4

15. 15SASQAG 16Jan03 AsmL Modelling © 2003 Microsoft Corporation, All Rights Reserved A: Invert Selection B: Create; Delete C: Create; Invert Selection; Invert Selection; Delete Test Every Walk < 5 Steps

16. 16SASQAG 16Jan03 AsmL Modelling © 2003 Microsoft Corporation, All Rights Reserved Test Model Toolkit Source: Testing Internet Explorer: From Underdog To Best-In-Class Browser

17. 17SASQAG 16Jan03 AsmL Modelling © 2003 Microsoft Corporation, All Rights Reserved Test Model Toolkit - Impact Used by over 20 teams at MS Time to automate fell by as much as 88% BizTalk - Took 1 week to generate a set of test cases that took 8 weeks by hand BizTalk - Took 1 week to generate a set of test cases that took 8 weeks by hand Code coverage increased by as much as 50% One subteam in Shell – In 2 weeks increased code coverage from 20% to 75%, while the number of test cases increased from ~75 to ~2000. One subteam in Shell – In 2 weeks increased code coverage from 20% to 75%, while the number of test cases increased from ~75 to ~2000. Many spec and implementation bugs are uncovered in the course of model creation Source: Testing Internet Explorer: From Underdog To Best-In-Class Browser

18. 18SASQAG 16Jan03 AsmL Modelling © 2003 Microsoft Corporation, All Rights Reserved The AsmL modeling toolkit Abstract state machine Language is a powerful modeling language AsmL provides Authoring via MS Word and XML Authoring via MS Word and XML Simulation and interoperability via COM and.NET Simulation and interoperability via COM and.NET Automatic test oracle Automatic test oracle Semi-automatic test case generation Semi-automatic test case generation Integration with Microsoft’s test platform Integration with Microsoft’s test platform

19. 19SASQAG 16Jan03 AsmL Modelling © 2003 Microsoft Corporation, All Rights Reserved AsmL: Getting Started, ‘sort’ example var A as Seq of Integer = [] Swap() choose i in indices(A), j in indices(A) where i A(j) A(i) := A(j) A(j) := A(i) sort() step until fixpoint swap() A = [2,3,1] A = [1,2,3] A = [2,1,3] Non-determinism A = [1,3,2] Parallelism

20. 20SASQAG 16Jan03 AsmL Modelling © 2003 Microsoft Corporation, All Rights Reserved AsmL: Predefined Data Types Sets menu = {"ham","cheese", "blt"} menu = {"ham","cheese", "blt"} Sequences random = [3, 6, 7, 1, 5] //as Seq of Integer Maps directory as Map of String to Integer = {“police” -> 112, “info” -> 118} Tuples nameAndAge = (“John”,39) // ( String, Integer)

21. 21SASQAG 16Jan03 AsmL Modelling © 2003 Microsoft Corporation, All Rights Reserved The cool part: Asml’s high-level-ness comprehension to pick 1 of the oldest filters in a Most Recently Used (MRU) cache. class CacheFilter var lastAccessed as DateTime var MruSet as Set of CacheFilter = {} let victim = any i | i in MruSet where (forall j in MruSet holds // j access time newer or equal i (Compare(j.lastAccessed, i.lastAccessed) >= 0))

22. 22SASQAG 16Jan03 AsmL Modelling © 2003 Microsoft Corporation, All Rights Reserved XML & XPath <DocumentRoot> <childElementA/><childElementB/></ParentElement></DocumentRoot> Child Axis Attribute Axis XPath: /DocumentRoot/ParentElement/childElementB /DocumentRoot/ParentElement/@attrib2

23. 23SASQAG 16Jan03 AsmL Modelling © 2003 Microsoft Corporation, All Rights Reserved XPath Model – axis nodes We interpret an axis as producing either 1) all of the children of a given node or 2) all of the attribute children of a given node. EvaluateAxis (a as Axis, c as Context) as Seq of Node let x = ContextNode(c) match a Child : return NodeChildren(x) Axis.Attribute : return NodeAttributes(x) _ : error "EvaluateAxis: internal error" a=ChildContext=ParentNode Returns: childElementAchildElementB

24. 24SASQAG 16Jan03 AsmL Modelling © 2003 Microsoft Corporation, All Rights Reserved XPath model – node test An axis path (for example, "child::para") has a two- step evaluation. The axis ("child") is used to generate a list of matching nodes, which is subsequently filtered to include only those nodes that pass the given node test ("para"). ap as AxisPath: let nodes = EvaluateAxis(ap.axis, c) return [nodes(i - 1) | i in [1..Length(nodes)] where PassesNodeTest(ap.test, Context(nodes, i))]

25. 25SASQAG 16Jan03 AsmL Modelling © 2003 Microsoft Corporation, All Rights Reserved Using models for test Model Implementation Test OracleTest Cases Are run by Provides actual results for Pass No pass Provides expected results for Generates User Info

26. 26SASQAG 16Jan03 AsmL Modelling © 2003 Microsoft Corporation, All Rights Reserved Tools & Techniques Tool&Language Test Technique TMT FSM ASMLASM State Walk Test Case Gen Queries e.g. all Nodes all Edges Test Properties => FSM => state walk Simulation Test Oracle Dynamic Traversal Model Manager Model Coverage via random Test Case Gen currently manual – random paths ASML test case generation with feedback

27. 27SASQAG 16Jan03 AsmL Modelling © 2003 Microsoft Corporation, All Rights Reserved AsmL-based test harness AsmL model Test harness Implementation under test Side-by-side runs Discrepancies automatically flagged Any client

28. 28SASQAG 16Jan03 AsmL Modelling © 2003 Microsoft Corporation, All Rights Reserved AsmL test case generation - 1 Solves two problems How to find “interesting” parameter values and combinations How to find “interesting” call sequences Approach Use information derived from the model to optimize the number of test cases

29. 29SASQAG 16Jan03 AsmL Modelling © 2003 Microsoft Corporation, All Rights Reserved AsmL test case generation - 2 Divide state space according to state predicates System “covers” the state predicates Driven by input generator (parameter selection) or Driven by input generator (parameter selection) or Driven by enabledness (call sequences) Driven by enabledness (call sequences) Tester provides inputs, gives interesting state predicates, directs the search

30. 30SASQAG 16Jan03 AsmL Modelling © 2003 Microsoft Corporation, All Rights Reserved AsmL Model M The dual role of ASM models The dual role of ASM models Test cases Guided generation (Param Select with Feedback) Reduce FSM tool IUT Call next action Get state AsmL Model M Testing harness Is S valid according to M? Fail: witness Pass S view FSM

31. 31SASQAG 16Jan03 AsmL Modelling © 2003 Microsoft Corporation, All Rights Reserved AsmL test case generation - 3 Generate only as many test cases as needed to have full coverage of the specification GeneratorModel Input Coverage criteria Branch: Every “branch” of the spec must be taken Branch: Every “branch” of the spec must be taken Modified condition: independent atomic conditions Modified condition: independent atomic conditions Off-by-One: off-by-one errors Off-by-One: off-by-one errors Hyperstate coverage: Every abstract state must be visited Hyperstate coverage: Every abstract state must be visited Feedback

32. 32SASQAG 16Jan03 AsmL Modelling © 2003 Microsoft Corporation, All Rights Reserved XPath Model coverage via random Param Test Generation Model Statistics: 1,003,638 tests generated (115Mb file) Only 37 tests are needed to cover all 113 (reachable 104+9) branches!

33. 33SASQAG 16Jan03 AsmL Modelling © 2003 Microsoft Corporation, All Rights Reserved AsmL Test (asmlt) asmlT can be used to: 1.Find interesting sequences of method calls finds the shortest route in a generated FSM that uses every edge to traverse the states finds the shortest route in a generated FSM that uses every edge to traverse the states 2.Find interesting parameters for each method call Access Driven Filtering (ADF) you write a predicate (boolean function) to decide if a given parameter is interesting you write a predicate (boolean function) to decide if a given parameter is interesting 3.Perform a conformance test Compare Model with Implementation

34. 34SASQAG 16Jan03 AsmL Modelling © 2003 Microsoft Corporation, All Rights Reserved References Model-Based Test Generation ToolsModel-Based Test Generation Tools, Alan Hartman, AGEDIS CONSORTIUM http://www.agedis.de/documents/ModelBasedTestGenerationTools_cs.pdf http://www.agedis.de/documents/ModelBasedTestGenerationTools_cs.pdf Model-Based Test Generation Tools http://www.agedis.de/documents/ModelBasedTestGenerationTools_cs.pdf Finite State Model-Based Testing on a ShoestringFinite State Model-Based Testing on a Shoestring, Harry Robinson http:// www.model-based-testing.org /shoestring.htm http:// www.model-based-testing.org /shoestring.htm Finite State Model-Based Testing on a Shoestring http:// www.model-based-testing.org /shoestring.htm Using a model-based test generator to test for standard conformance Using a model-based test generator to test for standard conformance E. Farchi, et al IBM SYSTEMS JOURNAL, VOL 41, NO 1, 2002 http://researchweb.watson.ibm.com/journal/sj/411/farchi.pdf http://researchweb.watson.ibm.com/journal/sj/411/farchi.pdf Using a model-based test generator to test for standard conformance http://researchweb.watson.ibm.com/journal/sj/411/farchi.pdf Model Based TestingModel Based Testing, L. Apfelbaum, et al, Software Quality Week Conference in May, 1997. http:// www.model-based-testing.org /sqw97.pdf http:// www.model-based-testing.org /sqw97.pdf Model Based Testing http:// www.model-based-testing.org /sqw97.pdf Modeling for Testers: A Picture’s Worth 1000 WordsModeling for Testers: A Picture’s Worth 1000 Words, Elisabeth Hendrickson, Quality Tree Software, Inc. http://www.raddue.com/pnsqc/Pnsqc02.pdf http://www.raddue.com/pnsqc/Pnsqc02.pdf Modeling for Testers: A Picture’s Worth 1000 Words http://www.raddue.com/pnsqc/Pnsqc02.pdf Good reference for existing tools

35. 35SASQAG 16Jan03 AsmL Modelling © 2003 Microsoft Corporation, All Rights Reserved More References Abstract State Machine LanguageAbstract State Machine Language, Microsoft Research http://research.microsoft.com/fse/asml/ http://research.microsoft.com/fse/asml/ Abstract State Machine Language http://research.microsoft.com/fse/asml/ The AsmL Test Generator toolThe AsmL Test Generator tool, Microsoft Research http:// research.microsoft.com/fse /AsmL/doc/AsmLTester.html http:// research.microsoft.com/fse /AsmL/doc/AsmLTester.html The AsmL Test Generator tool http:// research.microsoft.com/fse /AsmL/doc/AsmLTester.html Testing Internet Explorer: From Underdog To Best-In-Class Browser Jason Taylor, Microsoft Corporation at StarWest 2002 Intelligent Test AutomationIntelligent Test Automation, Harry Robinson Software Testing & Quality Engineering (STQE) magazine Sept/Oct 2000 http://www.model-based-testing.org/robinson.pdf http://www.model-based-testing.org/robinson.pdf Confessions of a Modeling Bigot: Parts Confessions of a Modeling Bigot: Parts III-IV Michael Corning, ASPToday Dec02, Jan03 http://www.asptoday.com/content.asp?id=2043 Michael Corning http://www.asptoday.com/content.asp?id=2043 Confessions of a Modeling Bigot: Parts Michael Corning http://www.asptoday.com/content.asp?id=2043 © 2003 Microsoft Corporation. All rights reserved. This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.