Economic Perspectives in Test Automation: Balancing Automated and Manual Testing with Opportunity Cost Paper By – Rudolf Ramler and Klaus Wolfmaier Presented By – Pallavi Gulati

Topics Why Automated Testing? Simplistic Cost Model for deciding between Automated vs. Manual Testing Flaws in the Simplistic Cost Model New Model based on Opportunity Cost Benefits offered by the new model Future Plans

Why the need to Automate Tests? Testing accounts for at least 50% of total project cost Fast-paced iterative development Shrinking schedules and budgets Test Automation proposed to reduce testing costs

Advantages - Automated Tests Reduces overall cost of RUNNING tests Though some initial cost required for writing automated tests Increases the number of tests run Increases frequency at which tests are run Incremental cost of test running is less Major concern in deciding whether to fix a particular bug or not

Disadvantages - Automated Tests Can not automate every test case – timing-related, appearance, user-friendly, some system & integration test cases, involving different hardware setups May not catch new bugs, targeted towards regression bugs Higher latency as compared to manual testing which can start immediately

When to go for automation? How do managers generally decide when to automate? Decision is based on factors like: Efforts required to run manual test Frequency of testing Cost of third-party / in-house developed tool Efforts required to write automated tests

Simplistic Cost Model Vm = Estimated Cost of designing manual tests Am = Overall cost of manual testing for some # of times Va = Estimated Cost of designing automated tests (may also include cost of test tool) Aa = Overall cost of automated testing for some # of times

Flaws - Simplistic Cost Model Manual & Automated Testing are not comparable Exploring new functionality vs regression testing Only costs are analyzed not benefits Manual testing starts early in the process but some latency in automated testing All test cases & executions are considered equal Project Context (budget, resources) not considered Misses additional cost factors like training on the testing tool, new process adaptation

Why automated tests fail to provide expected benefits? Due to flaws in estimation using the Simplistic Cost Model and other factors like: Failure to understand that automated tests are targeted for regression bugs Gross underestimation of the effort required to develop automated tests Practically no allocation of efforts for maintaining automated tests

When to automate? vs. What to automate? It does not help to automate every test Need to decide trade-off between automated and manual testing Alternative - Opportunity Cost Model

Opportunity Cost Model Consider manual & automated test cost Decide how many manual test cases should be automated Pick those which will give maximum benefit / risk mitigation / coverage Project budget & deadlines considered Looks something like...

Opportunity Cost Model Va = 1 hour to automate 1 test 0 min to run 1 automated test 0 min to come up with 1 manual test Dm = 15min to execute 1 manual test

Restriction 1

Restrictions 2 and 3

Target Function Ra = Risk Mitigation for automated testing Rm = Risk Mitigation for manual testing This allows us to include the benefit obtained by automating a test

Scenarios for Analysis

Scenarios for Analysis

Model can be extended to include Additional Influencing Factors Tests that rely solely on automation e.g. 100 concurrent transactions Productivity changes over time Causes an outward curved graph Growing Test Effort in iterative development Improves risk mitigation of automated test Maintenance cost Early / Late return on investment Developing automated test early using data driven technique Defect detection capability random input invoking crash

Advantages– Opportunity Cost Model Draws attention to balancing automated and manual testing Benefits as well as costs Risk mitigation capability of tests Budget and deadlines More realistic comparison between automated and manual testing

Future Plans Reduce complexity & make it more practical to use Evaluate preciseness and completeness Identify & add more influencing factors Analyze need to add more restrictions

Summary Advantages of Automated Test Simplistic Cost Model Flaws Proposed Opportunity Cost Model Benefits offered Future Plans for expanding the model

Wake up !!!