PREDICT Model for Test Automation

Does it sound familiar to you? Organization has procured test automation tools Management expectations are high Multiple test automation project initiatives started Enthusiasm, commitment and motivation declined with time Script maintenance had become a significant challenge Test automation tools are back on shelf

Testing and test automation requires same skills Capture/Replay is a successful test automation approach Test automation is an integral part of the over all test process Test automation is effective in solving regression testing issues Every test and test type can be automated Test automation development is software development Test automation projects can be delivered by personnel with strong project management skills Spot Quiz

Essential building blocks for successful test automation solutions Test automation solution approach

PREDICT Model Rationalize Estimate Design Implement Consolidate Transition Plan

PREDICT Model Structured methodology Multi stage process consisting of seven components Plan Rationalize Estimate Design Implement Consolidate Transition Step-wise approach Easy to apply

PREDICT Model - Plan Rationalize Transition Plan Estimate Design Implement Consolidate

Need for test automation project What do you consider to start a project ? Test related checks Objective Scope Documentation Test ware Organization related checks Investments Expectations Resource availability Technology related checks Stability of SUT Test tool suitability

Need for test automation project Are you getting bang for your buck ? Costs Licensing Framework Implementation Maintenance factor Benefits Saved manual effort per test run Number of test runs Re-usability factor Amortizing benefits Life cycle of the application/product Opportunities for test automation

Plan of approach Introduction Objective Pre-requisites Set-up pilot project Activities Planning Deliverables Organization Resources

PREDICT Model - Rationalize Rationalize Transition Plan Estimate Design Implement Consolidate

How do we rationalize requirements ? 5 P factors –Project –Process –Product –People –Price Test suite attributes Building trade-offs based on project context

Project Objective and scope Clear objectives in terms of time, money and quality Strategy Test automation strategy fitment with manual testing strategy Build Build frequency Configuration management Bad experiences User interface Stability Cycle time Regression possible

Process Quality documentation Manual test scripts, documentation Development approach Phases clearly defined, how early can test automation start Test practice New builds tested before delivering to QA Regression moments Naming conventions User Interface conventions

Product Stability Base platform stability Testability Degree of interaction between the tool and product under test Test tool integration Integration between test management tool and configuration management Test oracles Interaction with non UI components,database Automation framework Availability of frame work to reuse

People Organization readiness Management expectations Resource availability Machine, Licenses and people with right skills Non- programming resources Framework inclined towards non-programming resources QA population Ratio of development to QA Collaboration Collaboration between QA and Dev Technical skill set Test engineers with technical skills

Price Return on Investment Opportunities of reusing test automation investment Reusability of Investment Are there any products which are tested in a similar way Business Plan Is there a business plan to realize benefits of Test Automation

Test suite attributes Maintainability Reusability Reliability Usability Portability

Which is best test automation regime ? IT DEPENDS ON THE PROJECT CONTEXT

PREDICT Model - Estimate Rationalize Transition Plan Estimate Design Implement Consolidate

Test automation estimation challenges Test organization maturity Scope of requirements Framework Skill level Test case complexity Domain complexity Test tool Testability

Estimation techniques Delphi Gather estimates from each team member Ask high and low estimator to explain estimates Repeat twice, then use average Three point method Have team estimate best, worst case and expected case Use expected case Variances between best and worst useful to gauge estimate accuracy

Work Break down Structure

Risk categories Technical Test tool Application/product Communication

Risk management

PREDICT Model - Design Rationalize Transition Plan Estimate Design Implement Consolidate

Test suite design issues Testability Test suite design patterns

Testability considerations Tools Design Naming Conventions Exception & Error Handling Timing issues Logging Visual Cue Processes & Documentation

Record/Play back Functional de-composition Key-word Driven User Interface-driven Model-based Test automation development approaches

Advantages Useful in determining how the tool interacts with the application under test Provides initial ideas on how to develop test scripts Useful while Playing around with the tool Disadvantages Test scripts contains hard-coded values Un-reliable scripts : Any small change in the application results in not being able to replay Maintenance costs are astronomical Not viable & cost-effective Record & Playback

Reduces test scripts into their fundamental tasks Driver Scripts : Initialization, Calling scripts in a desired order Generic scripts : General,Application specific,Navigation,Data verification,Reporting,logging etc… Business Function Scripts : Re-usable Business functionality,Makes calls to Generic Scripts Test Script : Test Logic, Makes calls to Business Function Scripts and Generic Scripts Test data and logic are separated Hierarchical structure is employed : Modular design Each test script unique data file Functional decomposition

Test Director Reports Test Scripts Generic Library Business Library Test Scripts Environment Configuration Stored ProcObject Repository Technical Architecture

Functional decomposition: Analysis Advantages Modular design : Reduces redundancy and duplication of efforts Test data input & output stored in a single data file : Easy maintenance Scripts can be developed whilst application development is in progress : Generic Scripts Single point maintenance is possible using Business Function scripts Scripts can return TRUE (or) FALSE to the calling scripts. This makes it easy for Error Recovery & un-attended test execution Assemble tests on demand Disadvantages Proficiency in the TSL Complex test data file management

The test script is represented as a set of key-words in a spread sheet Examples of Key-words are start_test, Initial window,Url,Input,Verify etc.. Driver Script Performs the initialization as required Driver Script Reads and processes the file name(s) of the test scripts Driver Script Matches the Key Words contained in the file Driver Script invokes Business Function & Generic scripts Report the return values back The entire process is data-driven Key-word driven

Advantages Preserves most of the advantages of the Functional decomposition method Very similar to documenting manual test script Minimal programming knowledge is required to write and execute test scripts Return on investment can be achieved quicker than other methods in most of the cases Re-usability significantly increases over a period of time Test script development is possible with out a functioning application Disadvantages Investment in initial frame work Training might become tedious Key-word driven - Analysis

Generic Scripts Keyword Driven Test Cluster Business Function Script4 Reports Driver Script Business Function Script1 Business Function Script2 Business Function Script3 GUI Map Technical Architecture

Abstraction of test structure: Test set Test case Test step Test action Key-words are identified based on the application under test Test steps are created using user interface Test set management (Creation/execution) Test set Test case Test step Test action User interface driven

Automation Framework Recording Engine Script Generation Module APPLICATION User Interface Test Test Script XML Object Repository Object Creation Module Execution Engine Win Runner/QT P/Silk Test/… Library Structure Technical Architecture

Advantages Preserves most of the advantages of the previous methods Insulates tests both from the changes of the application and failings of test automation tools Zero-programming required Return-on-investment is very high Re-usability significantly increases over a period of time Robustness and maintainability is high Partly application independent Minimal learning curve Challenges High initial investment in creating framework Functioning application needed User interface, Model driven - Analysis

Test Case Generator (In key words) Keyword Translator engine (Test tool specific) Automated Test Scripts Test driver generator Behavior modeler Requirements (UML) Win Runner Silk Test Other test tools Model Driven

PREDICT Model - Implement Rationalize Transition Plan Estimate Design Implement Consolidate

Test automation implementation issues Team structure Work allocation Coding guide-lines and check lists Peer reviews Test runs Integration Configuration management

Work allocation

Coding guidelines and checklist

Reviews

Building integrations Test Script1 Test Script2 Test Script3 Call Test Script1() Call Test Script2() Call Test Script3()

Configuration management : VSS

Test execution Test suite execution check-lists are used Test Execution A separate test environment is prepared Fresh test data in test DB Fresh test data in Application DB Tests are executed Test suite execution issues & root cause analysis is documented Report Test Results Update test scripts if needed

Test execution log

PREDICT Model - Consolidate Rationalize Transition Plan Estimate Design Implement Consolidate

Update scripts and test data Complete documentation (User Manual, installation Guide and maintenance guide) Freeze & archive test suite Professionally packaged shippable test suite Easy distribution mechanism Activities in consolidation

User manual

Professional packaging

PREDICT Model - Transition Rationalize Transition Plan Estimate Design Implement Consolidate

Acceptance test runs performed Knowledge transfer to the user groups Gain feedback from the field Prepare for maintenance and sustenance Up-gradation plans Discussion for next project Transition process

Successful test automation projects require mature test automation leadership skills Test automation projects should be treated like a development project Understanding PREDICT Model Challenges, issues, tips and tricks in applying PREDICT model Wrap up