EVALUATION OF V&V TOOLS GOAL To assess feasibility of introducing automation into an existing process in a given organization which produces products for particular types of customers

EVALUATION OF V&V TOOLS (continued) METHODOLOGY Derive score card Identify quality factors Rank factors by priority weights (1-10) to indicate the importance in your organization Add blank area for scoring Add blank area for value = weight * score

EVALUATION OF V&V TOOLS METHODOLOGY (continued) Based on score card quality factors, select candidate V&V tools for comparison Based on scorecard and test tool selections, build or select a benchmark which is challenging to both tools, which can be served well by either tool in different scenarios or applications, and can be shown to be not biased toward either tool

EVALUATION OF V&V TOOLS (continued) METHODOLOGY (continued) Apply the tools to the benchmark independently, capturing context and metrics for later comparison. As far as possible limit the number of degrees of freedom, by utilizing the same or similar test strategies or test plans, or even test cases (in the case of code coverage tools, for example)

EVALUATION OF V&V TOOLS METHODOLGY (continued) Evaluate the completeness of the tool scoring and document with justification for your evaluation of completeness Analyse the results of the scoring for all factors and for preferred factors only Draw conclusions and make recommendations with rationale