1. Yaxiong Lin TestOptimalTestOptimal, LLC TestOptimal Model-based Testing Effective Test Case Design And Test Automation Twin Cities Quality Assurance Association (TCQAA) September 12, 2013

2. Agenda  What is Model-Based Testing (MBT)  MBT in Action (demo)  Benefits of MBT  MBT Process  MBT Use Cases Testptimal LLC

3. Agenda  What is Model-Based Testing (MBT)  MBT in Action (demo)  Benefits of MBT  MBT Process  MBT Use Cases Testptimal LLC

4. Business Impacts of Software Defects Source: Crosstalk, the Journal of Defense Software Engineering U.S. alone: $60 billion per year (NIST 2002) Testptimal LLC

5. Where Defects are Found Testptimal LLC Source Unknown

6. Costs of Fixing Software Defects Testptimal LLC The Real Cost of Software Defects, LKP Consulting Group http://www.lkpgroup.com/Cost%20of%20Software%20Defects.pdf

7. Software Testing Challenges  Requirement/design defect impact: 64%  Non-developer testing catches 69% of defects  Costs 100x to fix defect in production  Gain higher confidence better test coverage requirement traceability Testptimal LLC

8. What is Model-Based Testing? “ Model-Based Testing is the Automation of Test Design of Black-box Tests” Testptimal LLC Practical Model-base Testing”, Mark Utting and Bruno Legeard

9. What is Model-Based Testing? “ Model-Based Testing is the Automation of Test Design of Black-box Tests” Testptimal LLC Practical Model-base Testing”, Mark Utting and Bruno Legeard Model-based testing is application of Model based design for designing and optionally also executing artifacts to perform software testing. Models can be used to represent the desired behavior of a System Under Test (SUT), or to represent testing strategies and a test environment.Model based designsoftware testing wikipedia

10. Why Model?  Visual representation of complex systems  Ask What-If to understand cause-effect  Important role in the technology advancement and software development  Software testing is a complex task Testptimal LLC

11. MBT Modeling Notations  Finite State Machine (FSM) State Diagram (UML)  Extended Finite State Machine (EFSM)  Message Sequence Chart (MSC) Sequence Diagram (UML)  Control Flow Graph (CFG) Activity Diagram (UML)  Event Flow Diagram  UML Testing Profile – UTP  … Testptimal LLC

12. Testing Vending Machine  Drink for $1  Accept Quarter and HalfDollar coins  Cancel any time for refund Testptimal LLC

13. Vending Machine Test Cases  Quarter + Quarter + Quarter + Quarter => Vend  Quarter + Quarter + HalfDollar => Vend  Quarter + HalfDollar + Quarter => Vend  Quarter + HalfDollar + HalfDollar => Vend  HalfDollar + Quarter + Quarter => Vend  HalfDollar + HalfDollar => Vend  HalfDollar + Quarter + HalfDollar => Vend  Quarter + Cancel => Refund  … Testptimal LLC

14. Example Model Testptimal LLC

15. Generated Test Sequence Testptimal LLC

16. Agenda  What is Model-Based Testing (MBT)  MBT in Action (demo)  Benefits of MBT  MBT Process  MBT Use Cases Testptimal LLC

17. Types of MBT  Offline Testing  Online Testing Testptimal LLC

18. Demo: Test Case Generation for Manual Testing or Offline Testing Testptimal LLC

19. Demo: Login Test Automation – Online Testing Testptimal LLC

20. Demo: WebStore Test Automation – Online Testing Testptimal LLC

21. Testing Web App Testptimal LLC

22. WebStore MBT Model © 2012 Testptimal LLC

23. Generated Test Sequence Testptimal LLC

24. Agenda  What is Model-Based Testing (MBT)  MBT in Action (demo)  Benefits of MBT  MBT Process  MBT Use Cases Testptimal LLC

25. Recent MBT User Survey by Robert Binder, System Verification Associates,2012  MBT reduced escaped bugs by 59%.  MBT reduced testing costs by 17%.  MBT reduced testing duration by 25%. Testptimal LLC

26. Benefits of MBT  Catch requirement / design defects  Catch defects earlier  Reduce defects in fields  Improve efficiency (Agility) handling changes better communication on defects  Gain higher confidence better test coverage requirement traceability Testptimal LLC

27. Agenda  What is Model-Based Testing (MBT)  MBT in Action (demo)  Benefits of MBT  MBT Process  MBT Use Cases Testptimal LLC

28. MBT Iterative Development Process Testptimal LLC Modeling Test Generation AutomationExecution Analysis MBT Models Requirement s User Stories

29. MBT Process Input/Output Testptimal LLC Input Requirements User Stories Test Objectives AUT/Spec Environment Configuration Execution Result MBT Step Modeling Test Generation Automation Execution Analysis Output Behavior Models Test Strategies Test Coverage Automation Scripts Test Coverage Defects, Perform. Stat Model Modification

30. Model-Based Testing Process Testptimal LLC http://journal.thedacs.com/issue/52/145 Department of Defense (DoD) Information Analysis Center (IAC)

31. Agenda  What is Model-Based Testing (MBT)  MBT in Action (demo)  Benefits of MBT  MBT Process  MBT Use Cases (more demo) Testptimal LLC

32. Applications of Model-Based Testing  Conformance Testing  Functional Testing  System Integration Testing  Security Vulnerability Testing Testptimal LLC

33. MBT Use Cases (demo)  Finding Temporal Defects  Realistic Load Simulation Performance Load/Stress Testing  Session /Scenario Based Testing  Risk Based Testing  Quick bug fix verification Testptimal LLC

34. Questions?  MBT Papers: http://testoptimal.com/MbtReferences.html http://testoptimal.com/MbtReferences.html  Email: yaxiong.lin@testoptimal.com Testptimal LLC