1. Thursday, November 1, 2001(c) 2001 Ibrahim K. El-Far. All rights reserved.1 Enjoying the Perks of Model-based Testing Ibrahim K. El-Far Florida Institute of Technology http://www.testingresearch.com

2. Thursday, November 1, 2001(c) 2001 Ibrahim K. El-Far. All rights reserved.2 In This Presentation … Overview of model-based testing tasks Advantages Choosing models Resources, training and tool support Tips on Model Building General MBT tips Inescapable evils of MBT Is MBT Worthwhile?

3. Thursday, November 1, 2001(c) 2001 Ibrahim K. El-Far. All rights reserved.3 What Model Based Testing Is Any software testing approach that bases tasks on a model of the software under test A model is an explicit representation of the program s functional behavior Industry does not recognize mental or structural models as a basis for MBT Industry is realizing the value of models MBT is increasingly understood but its worth is yet to be validated

4. Thursday, November 1, 2001(c) 2001 Ibrahim K. El-Far. All rights reserved.4 Mental Model Build Model Explicit Model Generate Tests Test Suites & Scripts Run Scripts Application Under Test Oracle Get expected result Get actual result Test Pass & Failure Data Analyze Data Decide whether to Generate more tests Modify the model Stop testing Estimate Reliability & other quality measures Test Objectives & Stopping Criteria Model Based Testing Tasks

5. Thursday, November 1, 2001(c) 2001 Ibrahim K. El-Far. All rights reserved.5 Obvious Advantages Automate generation of Large suites of tests and lengthy tests Complicated input sequences Likely to expose failures Caused by weird combinations of inputs Revealed only with time (memory leaks)

6. Thursday, November 1, 2001(c) 2001 Ibrahim K. El-Far. All rights reserved.6 Obvious Advantages Provides an additional basis for Coverage-based evaluation of test progress Measurement quality of the product Provides insight into effectiveness when no failures are found

7. Thursday, November 1, 2001(c) 2001 Ibrahim K. El-Far. All rights reserved.7 Not So Obvious Advantages A model is a precise communication tool Good vehicle of presentation to non- technical staff While building a model Develop a better understanding of SUT Find many bugs while exploring SUT Compare versions of the product

8. Thursday, November 1, 2001(c) 2001 Ibrahim K. El-Far. All rights reserved.8 Not So Obvious Advantages The model is an aid in regression Alternative to classic regression (variation) Cheap validation of regression test suites Aids in more rigorous verification Model checking Program proof Precondition analysis

9. Thursday, November 1, 2001(c) 2001 Ibrahim K. El-Far. All rights reserved.9 Choosing Models to Fit Needs Highly dependent on application type HTML processing component in a browser Calculator mathematical expressions State rich systems: GUI & Phone systems Parallel systems Different models for different system aspects Combinations of models are more useful Try to reuse models from design & requirements analysis

10. Thursday, November 1, 2001(c) 2001 Ibrahim K. El-Far. All rights reserved.10 Resources Time & Budget Computing resources for test automation Skilled and experienced personnel Specialized test artifacts like test generators, oracles, etc …

11. Thursday, November 1, 2001(c) 2001 Ibrahim K. El-Far. All rights reserved.11 Training Useful theoretical elements Automate theory and formal languages Graph theory and traversal algorithms Statistics and stochastic processes Useful skills Various methods to build models Writing test generators Tool-specific skills

12. Thursday, November 1, 2001(c) 2001 Ibrahim K. El-Far. All rights reserved.12 Tool Support Input & output enumeration Input simulation & interface wrappers Test evaluation mechanisms Environment monitoring & simulation Test generators & suite analyzers Model generators designers & managers Tools to identify model variables in SUT

13. Thursday, November 1, 2001(c) 2001 Ibrahim K. El-Far. All rights reserved.13 Tips on Model Building Model needs to be as brief as possible without sacrificing information Model should not contain information that are not pertinent for your objectives Model should exclude pertinent information that is deemed redundant or useless

14. Thursday, November 1, 2001(c) 2001 Ibrahim K. El-Far. All rights reserved.14 Model needs to be as readable as possible, even at the expense of brevity Model needs to be left open for changes and additions, if at all possible Model writing needs to consider implementation details even if the model does not Tips on Model Building

15. Thursday, November 1, 2001(c) 2001 Ibrahim K. El-Far. All rights reserved.15 General MBT Tips Start as early as possible Build models incrementally Use only models that fit your needs Invest in training Try to allocate as many resources Write simple tools or use off-the-shelf products and simple scripts to start Inescapable Evils of MBT

16. Thursday, November 1, 2001(c) 2001 Ibrahim K. El-Far. All rights reserved.16 Inescapable Evils of MBT State Explosion Use many small models Use other types of models Employ hierarchical views Employ partial views Abstract details Exclude redundant or useless details

17. Thursday, November 1, 2001(c) 2001 Ibrahim K. El-Far. All rights reserved.17 Inescapable Evils of MBT Test Evaluation Write direct evaluation routines Code a comparator to a competing system Use a different implementation or version Plausibility is more useful than correctness Use assertions to relate program execution to model

18. Thursday, November 1, 2001(c) 2001 Ibrahim K. El-Far. All rights reserved.18 Is MBT Worth It? The wise answer is, It depends. Do you have existing models? Do you have time for initial activities? How late are you starting? Does your modeling experience match the software you are trying to test? Do you have scripting/programming expertise on your team?

19. Thursday, November 1, 2001(c) 2001 Ibrahim K. El-Far. All rights reserved.19 Sargon: A Free Testing Tool Primarily a research tool Implements various testing techniques Open source testing utilities Implemented in Java More information on http://www.testingresearch.com/ http://www.testingresearch.com/

20. Thursday, November 1, 2001(c) 2001 Ibrahim K. El-Far. All rights reserved.20 Queries Contact Info Clarification Comments Concerns Questions Suggestions E-mail: ielfar@acm.orgielfar@acm.org Official URL: http://www.testingresearch.com http://www.testingresearch.com Personal URL: http://www.cs.fit.edu/~ielfar http://www.cs.fit.edu/~ielfar Snail mail: Computer Sciences Department, Software Engineering Program, Florida Institute of Technology, 150 West University Boulevard, Melbourne, Florida 32901, USA