1. Economic Perspectives in Test Automation: Balancing Automated and Manual Testing with Opportunity Cost
Paper By – Rudolf Ramler and Klaus Wolfmaier
Presented By – Pallavi Gulati

2. Topics Why Automated Testing?
Simplistic Cost Model for deciding between Automated vs. Manual Testing
Flaws in the Simplistic Cost Model
New Model based on Opportunity Cost
Benefits offered by the new model
Future Plans

3. Why the need to Automate Tests?
Testing accounts for at least 50% of total project cost
Fast-paced iterative development
Shrinking schedules and budgets
Test Automation proposed to reduce testing costs

4. Advantages - Automated Tests
Reduces overall cost of RUNNING tests
Though some initial cost required for writing automated tests
Increases the number of tests run
Increases frequency at which tests are run
Incremental cost of test running is less
Major concern in deciding whether to fix a particular bug or not

5. Disadvantages - Automated Tests
Can not automate every test case – timing-related, appearance, user-friendly, some system & integration test cases, involving different hardware setups
May not catch new bugs, targeted towards regression bugs
Higher latency as compared to manual testing which can start immediately

6. When to go for automation?
How do managers generally decide when to automate?
Decision is based on factors like:
Efforts required to run manual test
Frequency of testing
Cost of third-party / in-house developed tool
Efforts required to write automated tests

7. Simplistic Cost Model Vm = Estimated Cost of designing manual tests
Am = Overall cost of manual testing for some # of times
Va = Estimated Cost of designing automated tests (may also include cost of test tool)
Aa = Overall cost of automated testing for some # of times

8. Flaws - Simplistic Cost Model
Manual & Automated Testing are not comparable
Exploring new functionality vs regression testing
Only costs are analyzed not benefits
Manual testing starts early in the process but some latency in automated testing
All test cases & executions are considered equal
Project Context (budget, resources) not considered
Misses additional cost factors like training on the testing tool, new process adaptation

9. Why automated tests fail to provide expected benefits?
Due to flaws in estimation using the Simplistic Cost Model and other factors like:
Failure to understand that automated tests are targeted for regression bugs
Gross underestimation of the effort required to develop automated tests
Practically no allocation of efforts for maintaining automated tests

10. When to automate? vs. What to automate?
It does not help to automate every test
Need to decide trade-off between automated and manual testing
Alternative - Opportunity Cost Model

11. Opportunity Cost Model
Consider manual & automated test cost
Decide how many manual test cases should be automated
Pick those which will give maximum benefit / risk mitigation / coverage
Project budget & deadlines considered
Looks something like...

12. Opportunity Cost Model
Va = 1 hour to automate 1 test
0 min to run 1 automated test
0 min to come up with 1 manual test
Dm = 15min to execute 1 manual test

13. Restriction 1

14. Restrictions 2 and 3

15. Target Function Ra = Risk Mitigation for automated testing
Rm = Risk Mitigation for manual testing
This allows us to include the benefit obtained by automating a test

16. Scenarios for Analysis

17. Scenarios for Analysis

18. Model can be extended to include Additional Influencing Factors
Tests that rely solely on automation
e.g. 100 concurrent transactions
Productivity changes over time
Causes an outward curved graph
Growing Test Effort in iterative development
Improves risk mitigation of automated test
Maintenance cost
Early / Late return on investment
Developing automated test early using data driven technique
Defect detection capability
random input invoking crash

19. Advantages– Opportunity Cost Model
Draws attention to balancing automated and manual testing
Benefits as well as costs
Risk mitigation capability of tests
Budget and deadlines
More realistic comparison between automated and manual testing

20. Future Plans Reduce complexity & make it more practical to use
Evaluate preciseness and completeness
Identify & add more influencing factors
Analyze need to add more restrictions

21. Summary Advantages of Automated Test Simplistic Cost Model Flaws
Proposed Opportunity Cost Model
Benefits offered
Future Plans for expanding the model

22. Wake up !!!