1. Estimate Testing Size and Effort Using Test Case Point Analysis
Vistacon 2011, December 7th, 2011
Estimate Testing Size and Effort Using Test Case Point Analysis
Vu Nguyen
KMS-Technology and QASymphony

2. Agenda Objectives Background and Motivation
Existing Estimation Methods
Test Case Point Analysis
Effort Estimation Methods using TCP
Conclusion and Future Work

3. Objectives
Discuss importance of software estimation, especially for testing projects
Discuss existing methods for estimating testing activities
Introduce an approach to estimating testing projects
Test Case Point Analysis
Effort Estimation using TCP

4. Agenda Objectives Background and Motivation
Existing Estimation Methods
Test Case Point Analysis
Effort Estimation Methods using TCP
Conclusion and Future Work

5. Software Estimation and Its Importance
process of determining the cost, time, staff, and other related attributes of software projects, often before work is performed
Estimation is important for the success or failure of software projects
Provides inputs for
Making investment decisions
Budget and staff allocation
Tradeoff and risk analysis
Project planning
Stakeholder negotiation
etc.

6. Popular Software Estimation Methods
Sizing Methods
Source Lines of Code (SLOC)
Function Points Analysis
Use Case Points
Effort Estimation Methods
Expert judgment/experience
Productivity index
COCOMO, SEER-SIM, SLIM models
Software testing estimation methods
Using a test distribution percentage
Test Case Points Analysis (by Cognizant Technology Solutions) [2]
Mainly based on the number of steps

7. Motivation Testing accounts for up to 50% of project effort [1]
Current problems
estimates are done for the whole project rather than testing specific
lack of reliable methods designed for estimating size and effort of software testing
vague definitions of testing productivity
due to the lack of a size measure for software testing
There are needs of
accurately estimating effort of testing activities
measuring size and productivity of testing activities
measuring effectiveness and efficiency of software testing

8. Motivation (cont’d) Our aim to
Defining a method for estimating the size of testing activities
Defining methods to estimate testing effort, schedule, and staff accurately using this size measure
Using this size measure as a basis for measuring
productivity
effectiveness
other testing metrics

9. Agenda Objectives Background and Motivation
Existing Estimation Methods
Test Case Point Analysis
Effort Estimation Methods using TCP
Conclusion and Future Work

10. Test Case Point Analysis
Principles
Size must reflect the mass and complexity of the testing project
Size should correlate with testing effort
Test case point is measured using test cases as main input
Test case complexity is based on
Checkpoints
Precondition
Test Data
Type of test case

11. Test Case Point Analysis (cont’d)
Process overview
Estimate Testing Effort
Estimated Effort
Project Attributes
Test Case
Test Case Point Analysis
Test Case Points

12. Test Case Point Analysis (cont’d)
TCPA estimates the size of testing projects using test cases as input
Steps
Measure test case complexity
Count checkpoints
Measure the complexity of precondition
Measure the complexity of data
Adjust Test Case Point by type of test
Consider 11 types of test case
Each type has a weight
Adjust TCP using the weight
Use functional test cases as baseline with the weight of 1.0

13. Checkpoint Checkpoint Counting rule 1
Is the condition in which the tester verifies whether the result produced by the target function matches the expected criterion
One test case consists of one or many checkpoints
Counting rule 1
One checkpoint is counted as one Test Case Point

14. Test Case Precondition
specifies the condition to execute the test case
mainly affects the cost to execute the test case
may be related to data prepared for the test case
Four levels of complexity
None
Low
Medium
High

15. Test Case Precondition (cont’d)
None
The precondition is not applicable or important to execute the test case
Or, the precondition is just reused from the previous test case to continue the current test case
Low
The condition for executing the test case is available with some simple modifications required
Or, some simple set-up steps are needed
Medium
Some explicit preparation is needed to execute the test case
The condition for executing is available with some additional modifications required
Or, some additional set-up steps are needed
High
Heavy hardware and/or software configurations are needed to execute the test case

16. Test Case Precondition (cont’d)
Counting rule 2A: Counting Unadjusted Test Case Points for Precondition:
Each complexity level of precondition is assigned a number of Test Case Points
Based on our survey of 18 senior QA engineers

17. Test Data Test Data Four levels of complexity
used to execute the test case
can be generated at the test case execution time, sourced from previous tests, or generated by test scripts
Test data is test case specific, or general to a group of test cases, or for the whole system
Four levels of complexity
None
Low
Medium
High

18. Test Data (cont’d) None Low Medium High
No test data preparation is needed
Low
Simple test data is needed and can be created during the test case execution time
Or, the test case uses a slightly modified version of existing test data and requires little or no effort to modify the test data
Medium
Test data is deliberately prepared in advance with extra effort to ensure its completeness, comprehensiveness, and consistency
High
Test data is prepared in advance with considerable effort to ensure its completeness, comprehensiveness, and consistency
This could include using support tools to generate data and a database to store and manage test data
Scripts may be required to generate test data

19. Test Data (cont’d)
Counting rule 2B: Counting Unadjusted Test Case Points for Test Data:
Each complexity level of Test Data is assigned a number of Test Case Points
Based on our survey of 18 senior QA engineers

20. Adjust Test Case Point
Test Case Point counted till this point is considered Unadjusted Test Case Point (UTCP)
UTCP is adjusted by considering types of test case
Each type of test case is assigned a weight
Adjusted Test Case Point (ATCP):
n ATCP = ∑UTCPi * Wi
i=1
UTCPi - the number of UTCP counted for the test case ith.
Wi - the weight of the test case ith, taking into account its test type

21. Weight by Type of Test
Based on our survey of 18 senior QA engineers

22. Weight by Type of Test (cont’d)

23. Test Case Point Analysis - Summary
Process overview
Count Checkpoints
Determine Precondition Complexity
Adjust with Test Type
Test Case
UTCP
TCP
Determine Test Data Complexity

24. Agenda Objectives Background and Motivation
Existing Estimation Methods
Test Case Point Analysis
Effort Estimation Methods using TCP
Conclusion and Future Work

25. Estimate Testing Effort
Estimate testing effort using TCP
Test effort distribution, four phases
Test Planning
Test Analysis and Design
Test Execution
Test Tracking and Reporting
Effort estimation methods
Productivity index
Regression
Each of these phases may be performed multiple times

26. Estimate Testing Effort (cont’d)
When estimating TCP, assume that each phase is performed ONCE
Effort is estimated dependent on how many times a phase is performed

27. Productivity Index
Effort is computed using productivity index of completed project
Productivity index is measured as person-month per TCP
Effort = TCP * Productivity Index
Simple method

28. Regression
Estimate effort of new projects using size and effort of completed projects
Regression formula

29. Tools Support by QASymphony [3]
qTest – Quality Management Suite
Manage requirements and test cases
Plan testing activities and resources
Monitor and control testing activities
Perform measurement, analysis, and reporting
Test case point analysis
Effort estimation
qTrace – defect and scenario recording tool
Record defect
Record user scenarios

30. Conclusion and Future Work
It is important to estimate size and effort of testing projects
Test Case Point Analysis (TCPA) was proposed to estimate size. Elements include
Checkpoints
Precondition
Test Data
Test Type
Simple two methods for estimating effort were introduced
Productivity index
Regression

31. Conclusion and Future Work
Advantages of TCPA
Easy to implement
Reflecting real complexity of test cases
Independent with the number of steps
Limitations and future work
Need empirical validations
Recalibrate constants used in the method
Support only manual testing
Lack of support for performance testing

32. Thank You

33. References
[1] Y. Yang, Q. Li, M. Li, Q. Wang, An empirical analysis on distribution patterns of software maintenance effort, International Conference on Software Maintenance, 2008, pp
[2] N. Patel, M. Govindrajan, S. Maharana, S. Ramdas, “Test Case Point Analysis”, Cognizant Technology Solutions, White Paper, 2001
[3] QASymphony: