Yaxiong Lin TestOptimalTestOptimal, LLC TestOptimal Model-based Testing Effective Test Case Design And Test Automation Twin Cities Quality Assurance Association (TCQAA) September 12, 2013

Agenda  What is Model-Based Testing (MBT)  MBT in Action (demo)  Benefits of MBT  MBT Process  MBT Use Cases Testptimal LLC

Agenda  What is Model-Based Testing (MBT)  MBT in Action (demo)  Benefits of MBT  MBT Process  MBT Use Cases Testptimal LLC

Business Impacts of Software Defects Source: Crosstalk, the Journal of Defense Software Engineering U.S. alone: $60 billion per year (NIST 2002) Testptimal LLC

Where Defects are Found Testptimal LLC Source Unknown

Costs of Fixing Software Defects Testptimal LLC The Real Cost of Software Defects, LKP Consulting Group

Software Testing Challenges  Requirement/design defect impact: 64%  Non-developer testing catches 69% of defects  Costs 100x to fix defect in production  Gain higher confidence better test coverage requirement traceability Testptimal LLC

What is Model-Based Testing? “ Model-Based Testing is the Automation of Test Design of Black-box Tests” Testptimal LLC Practical Model-base Testing”, Mark Utting and Bruno Legeard

What is Model-Based Testing? “ Model-Based Testing is the Automation of Test Design of Black-box Tests” Testptimal LLC Practical Model-base Testing”, Mark Utting and Bruno Legeard Model-based testing is application of Model based design for designing and optionally also executing artifacts to perform software testing. Models can be used to represent the desired behavior of a System Under Test (SUT), or to represent testing strategies and a test environment.Model based designsoftware testing wikipedia

Why Model?  Visual representation of complex systems  Ask What-If to understand cause-effect  Important role in the technology advancement and software development  Software testing is a complex task Testptimal LLC

MBT Modeling Notations  Finite State Machine (FSM) State Diagram (UML)  Extended Finite State Machine (EFSM)  Message Sequence Chart (MSC) Sequence Diagram (UML)  Control Flow Graph (CFG) Activity Diagram (UML)  Event Flow Diagram  UML Testing Profile – UTP  … Testptimal LLC

Testing Vending Machine  Drink for $1  Accept Quarter and HalfDollar coins  Cancel any time for refund Testptimal LLC

Vending Machine Test Cases  Quarter + Quarter + Quarter + Quarter => Vend  Quarter + Quarter + HalfDollar => Vend  Quarter + HalfDollar + Quarter => Vend  Quarter + HalfDollar + HalfDollar => Vend  HalfDollar + Quarter + Quarter => Vend  HalfDollar + HalfDollar => Vend  HalfDollar + Quarter + HalfDollar => Vend  Quarter + Cancel => Refund  … Testptimal LLC

Example Model Testptimal LLC

Generated Test Sequence Testptimal LLC

Agenda  What is Model-Based Testing (MBT)  MBT in Action (demo)  Benefits of MBT  MBT Process  MBT Use Cases Testptimal LLC

Types of MBT  Offline Testing  Online Testing Testptimal LLC

Demo: Test Case Generation for Manual Testing or Offline Testing Testptimal LLC

Demo: Login Test Automation – Online Testing Testptimal LLC

Demo: WebStore Test Automation – Online Testing Testptimal LLC

Testing Web App Testptimal LLC

WebStore MBT Model © 2012 Testptimal LLC

Generated Test Sequence Testptimal LLC

Agenda  What is Model-Based Testing (MBT)  MBT in Action (demo)  Benefits of MBT  MBT Process  MBT Use Cases Testptimal LLC

Recent MBT User Survey by Robert Binder, System Verification Associates,2012  MBT reduced escaped bugs by 59%.  MBT reduced testing costs by 17%.  MBT reduced testing duration by 25%. Testptimal LLC

Benefits of MBT  Catch requirement / design defects  Catch defects earlier  Reduce defects in fields  Improve efficiency (Agility) handling changes better communication on defects  Gain higher confidence better test coverage requirement traceability Testptimal LLC

Agenda  What is Model-Based Testing (MBT)  MBT in Action (demo)  Benefits of MBT  MBT Process  MBT Use Cases Testptimal LLC

MBT Iterative Development Process Testptimal LLC Modeling Test Generation AutomationExecution Analysis MBT Models Requirement s User Stories

MBT Process Input/Output Testptimal LLC Input Requirements User Stories Test Objectives AUT/Spec Environment Configuration Execution Result MBT Step Modeling Test Generation Automation Execution Analysis Output Behavior Models Test Strategies Test Coverage Automation Scripts Test Coverage Defects, Perform. Stat Model Modification

Model-Based Testing Process Testptimal LLC Department of Defense (DoD) Information Analysis Center (IAC)

Agenda  What is Model-Based Testing (MBT)  MBT in Action (demo)  Benefits of MBT  MBT Process  MBT Use Cases (more demo) Testptimal LLC

Applications of Model-Based Testing  Conformance Testing  Functional Testing  System Integration Testing  Security Vulnerability Testing Testptimal LLC

MBT Use Cases (demo)  Finding Temporal Defects  Realistic Load Simulation Performance Load/Stress Testing  Session /Scenario Based Testing  Risk Based Testing  Quick bug fix verification Testptimal LLC

Questions?  MBT Papers:  Testptimal LLC