1. ODC Level Performance Models

2. ODC PPM
Following ODCs are included:
SITATesting
SABRE
VOYA
SEI
TRIVENT
NITL C4
HORIZON-HRES
Hawaiian
BA Responsive NN

3. PPM Model - SITATesting
Model Parameter Description:
Metric Type
Sub-process
Metric Name
Metric Definition
Metric Attribute
QPPO
Test Execution
Test Execution Productivity
(Weighted test cases executed )/ (total execution effort)
Efficiency
Level 1:
Performance
Test Execution Rate
(Number of test cases executed )/ (total execution effort in person days)
% Defect Rejected
(Total number of defects rejected) / (total number of defects reported)
Effectiveness
Level 2:
Controllable
Test Design
% Test Data Creation Effort
(Total Test Data Creation Effort in Person days) / (Total planned Testing Effort in person days) * 100
Project Management
Team Competency
Planned teams domain competency %
Review
% Review Effort
(Total review effort in Person days) / (Total planned Testing Effort in Person days) * 100
% Effort in RTB Refinement
(Total RTB Refinement Effort in Person days) / (Total planned Testing Effort in person days) * 100

4. PPM Model - SITATesting
Model Equations:
Test Execution Productivity = ( ) * '% Defects Rejected'+ (1.769) * 'Test Execution Rate' +_5.317
% Defects Rejected = (-1.51) * '% Effort in RTB Refinement'+ (-0.698) * '% Review effort' +_0.013
Test Execution Rate = (96.809) * '% Test Data Creation effort '+ (14.227) * 'Team Competency' +_1.149
Y-variable
Test Execution Productivity
X-Variable
Test Execution Rate
% Defects Rejected
Confidence Level
95%
Adjusted R
P-Value
P-Value for Normality of Standard Residual
Y-variable
% Defects Rejected
X-Variable
% Review effort
% Effort in RTB Refinement
Confidence Level
95%
Adjusted R
P-Value
E-06
P-Value for Normality of standard Residual
Y-variable
Test Execution Rate
X-Variable
Team Competency
% Test Data Creation effort
Confidence Level
95%
Adjusted R
P-Value
P-Value for Normality of standard Residual

5. PPM Model - SITATesting
Baseline:
Metric Name
Average
St. Deviation
Test Execution Productivity
9.1522
% Defect Rejected
0.0323
0.0206
Test Execution Rate
2.4033
% Test Data Creation effort
0.0623
0.0126
Team Competency
0.5429
0.0916
% Effort in RTB Refinement
0.0181
0.0071
% Review effort
0.0902
0.0198

6. PPM Model - SITATesting
Justification:
Effectiveness parameters for Testing COE project are:
Defects Leaked to Production
% Defects rejected by Customer
Historical project data on defect leaked to production shows that it is under control and there is not much variation with respect to meeting customer and organization goals.
There is performance metrics level equation established for % Defect rejected by customer which is addressing the customer effectiveness goal.
Back

7. Quantitative Process Performance Objective
PPM Model - SEBRE
BO and QPPO Mapping
Business Objective
Quantitative Process Performance Objective
Improve CSS Rating to Desired Service Level
Improve cycle time by 10% by Jan 2017
Note: Based on historical data from Jan 2016 until Dec 2016 we will re-look at this data at the end of Jan 2017

8. PPM Model - SEBRE QPPO Mapping
Quantitative Goal for Process Performance Objective (QPPO)
Sub Process
Sub Process Performance Metrics
Sub process Controllable Metric
Improve the Activation Cycle time(in weeks)
Planned goal is 10% improvement of the cycle time
Current goal is 11 weeks
Planned goal is 10 weeks for Jan 2017(10% Improvement)
Activation implementation
Duration of Requirement analysis.
Duration of QA testing.
Learning weightage/ Complexity
Sabre experience

9. PPM Model - SEBRE PPM Regression Equation:
Actual Cycle time (weeks) = Sabre Experience Learning Weightage/ Complexity
Based on the R-Sq(adj) and P-value of the X factors considered , PPM is recommended for use
Predictor Coef SE Coef T P
Constant
Sabre experience
Learning weightage/ Complexity
S = R-Sq = 76.4% R-Sq(adj) = 73.9%
P value should be less than .05 and R square adjustment should be more than 70. This shows that our equation is healthy or appropriate

10. VOYA – Process Performance Model
Main Process Metrics (QPPO)
Performance Parameter
Mean Time to Resolve (Y)
Test coverage per MR
(X1)
Review Effectiveness per MR
(X2)
Regression Equation is
MTTR = Test Coverage per MR Review Effectiveness
R square is 95.4% and R-square adjusted is 93.1%
P- value of X parameters of equation is less than 0.05
#10 10 10

11. VOYA – Process Performance Model
Predictor Coef SE Coef T P VIF
Constant
Test Coverage per MR
Review Effectiveness
S = R-Sq = 95.4% R-Sq(adj) = 93.1%
Analysis of Variance
Source DF SS MS F P
Regression
Residual Error
Total
Source DF Seq SS
Test Coverage per MR
Review Effectiveness
#11 11 11

12. Baselines of the factors for process stability
MTTR
Test Coverage per MR
Review Effectiveness
Mean
2.13
8.98
0.73
STDEV
0.99
3.73
0.41
Count 7

13. PPM Model-SEI The regression equation is
Main Process Metric (QPPO)
Performance Parameter
LN((No of script passed/No. of script failed))
(Y)
Unit Testing effort rate
(X1)
Review Effectiveness (X2)
The regression equation is
Y= * Unit Testing effort rate(X2) * Rev Eff
Project Baselines
Output Parameter (Y)
Unit Testing effort rate (X1)
Rev Eff. (X2)
Mean
2.4
1.5
2.2
Std Dev
1.60
0.65
1.04
UCL
7.20
3.42
5.38
LCL
0.00

14. PPM Model-SEI Model Parameter Description: Metric Type Sub-process
Metric Name
Metric Definition
Metric Attribute
QPPO
COUT
LN-(First Pass Success)
No. of script passed in first review by sql SME/ Total No. of script submitted for review to sql SME
Effectiveness
Controllable
Unit Testing effort rate
Unit testing effort for sql script (Hrs) /No of sql script submitted for review
Review Effectiveness
Internal review wt defect for sql scripts / Internal Review effort for sql script (Hrs)

15. Regression Statistics
PPM Model-SEI – Web Factory
Regression Equation
SUMMARY OUTPUT
Regression Statistics
The regression equation is
Multiple R
Y = * Unit Testing effort rate(X2) * Rev Eff
R Square
Adjusted R Square
Standard Error
Observations 21
ANOVA df SS MS F
Significance F
Regression 2
2.52E-09
Residual 18
0.315
Total 20
Coefficients
t Stat
P-value
Lower 95%
Upper 95%
Lower 95.0%
Upper 95.0%
Intercept
Unit Testing effort rate
Review effectivness
P value is less than .05 for all the x’s and for intercept

16. Regression Statistics
PPM – SEI CAS
The Regression Equation is
Defect Density( QA) = *(UTC coverage)-0.07( % code review effort)
Based on the R-sq ( adj)and p value of x factors considered, PPM is recommended for use
Regression Statistics
Multiple R
R Square
Adjusted R Square
Standard Error
Observations 17
ANOVA df SS MS F
Significance F
Regression 2
1.29E-05
Residual 14
Total 16
Coefficients
Standard Error
t Stat
P-value
Lower 95%
Upper 95%
Lower 95.0%
Upper 95.0%
Intercept
3.05E-07
/User Story
% Code review effort/ total engineering effort

17. Baselines of the factors for process stability
QA Defect Density
Unit Test Coverage
% Code Review Effort
Mean
0.36
1.89
4.94
STDEV
0.15
0.72
0.98
Count 17
UCL
0.4
3.33
6.9
LCL
0.45
2.98
Operational Definition
Factor
Definition
Metric Formula
Unit
Unit Testing Coverage
Total no. of Unit test case executed per story
Total number of Unit Test cases executed / User Story
Number
% Code review effort
code review effort ( as a % of Total engineering effort)
% of code review effort (Code review effort/ Total engineering effort *100)
Percentage
QA Defect Density
No. of defects found in QA Phase per story point
Weighted defects found in QA testing/ Actual velocity(Story Points)

18. Thrivent – Process Performance Model
Main Process Metrics (QPPO)
Performance Parameter
Productivity (Y)
% COQ
(X1)
Review Effectiveness (X2)
% Resources Cross trained across applications
(X3)
Regression Equation is
Productivity = COQ % Review Effectiveness % Resources Trained Across Appl
R square is 91.7% and R-square adjusted is 87.5%
P- value of X parameters of equation is less than 0.05
#18 18 18

19. PPM – Thrivent The regression equation is
Productivity = COQ % Review Effectiveness
% Resources Trained Across Appl
Predictor Coef SE Coef T P VIF
Constant
COQ %
Review Effectiveness
% Resources Trained Across Appl
S = R-Sq = 91.7% R-Sq(adj) = 87.5%
Analysis of Variance
Source DF SS MS F P
Regression
Residual Error
Total
Source DF Seq SS
COQ %
Review Effectiveness
% Resources Trained Across Appl

20. Baselines of the factors for process stability
Productivity (hr/MR)
Cost of Quality (%)
Review Effectiveness (ratio)
% Resources trained across appl
Mean
27.28
28.73
1.65
41.38
STDEV
5.12
1.22
0.83
2.69
Count 10

21. PPM Model – NITL Model Parameter Description: Parameter Formula
Baseline
STDEV
Remarks
Productivity
12.35
1.72
Defect Density (/hrs)
0.32
0.13
Resource Utilization
98.58
0.83
Project overview (scope, team, start & end dates, lifecycle model) – Focus projects as well as non-focus projects;
#21 21 21

22. PPM Model – NITL
Productivity (Y) = * Actual capacity Utilisation * Defect Density
Project overview (scope, team, start & end dates, lifecycle model) – Focus projects as well as non-focus projects;

23. PPM Model – NITL Based on the R-Sq(adj) and P-value of the X factors
The regression equation is
Productivity = * Actual capacity utilization * Defect density
SUMMARY OUTPUT
Regression Statistics
Multiple R
R Square
Adjusted R Square
Standard Error
Observations 12
ANOVA df SS MS F
Significance F
Regression 2
1.65E-05
Residual 9
Total 11
Coefficients
t Stat
P-value
Lower 95%
Upper 95%
Lower 95.0%
Upper 95.0%
Intercept
Actual Capacity Utilisation
1.25
Defect Density
-5.71
RESIDUAL OUTPUT
Observation
Predicted Actual Productivity
Residuals 1 3 4 5 6 7 8 10
Based on the R-Sq(adj)
and P-value of
the X factors
considered , PPM
Is recommended for use.

24. TC Executed / TC prepared Internal weighted defects / SP
PPM Model – C4
Model Parameter Description:
Metric Type
Sub-process
Metric Name
Metric Definition
Metric Attribute
Improve Customer Satisfaction
Construction
ST Defects/ST
TC Executed / TC prepared
Effectiveness
Testing
Internal weighted defects / SP

25. PPM Model – C4 The regression equation is
ST Defect /SP (Y) = ( )*TC Create + ( )*Story Point Delivered +
( )*DB Components
SUMMARY OUTPUT
Regression Statistics
Multiple R
R Square
Adjusted R Square
Standard Error
Observations 9
ANOVA df SS MS F
Significance F
Regression 2
Residual 6
Total 8
Coefficients
t Stat
P-value
Lower 95%
Upper 95%
Lower 95.0%
Upper 95.0%
Intercept
TC execution coverage by Sprint team
Weighted Defects/SP by Sprint team
RESIDUAL OUTPUT
Observation
Predicted Weighted Defects/SP by ITG
Residuals 1 3 4 5 7

26. PPM Model – C4 Baseline: Metric Name Average St. Deviation
Productivity
8.97
1.94
Wt. defect / SP
0.185
0.07
Resource Capacity Utilization
0.9344
0.0258
% Burnout
0.9516
Average of Velocity (SP)
% of story changed

27. PPM Model - HORIZON-HRES
Model Parameter Description:
Sources
Plotting Freq.
Remarks
QPPO
Contractual Directive
Internal Directive
Actual PQ
20% improvement from the baselined till stage 11. baselined till stage 11 is 19.7 to bring it to 16. 16
Iteration end
Contractual requirement is to achieve 16 at the stage end . Variation from 17 to 12 is allowed and discussed with client. USL= 17 LSL= 12
Project overview (scope, team, start & end dates, lifecycle model) – Focus projects as well as non-focus projects;
Process composition done with the result of Causal, hence new PPB were release and PPM also recalibrated with the new data set and parameters. 27
#27 27

28. PPM Model - HORIZON-HRES
Main Process Metric (QPPO)
Performance Parameter
Controllable Parameter
Productivity (Y)
Rework Ratio (X1)
Partner Testing Defects Density (X2)
Backlog Index (X3)
Delivered FP/ Effort
Rework FP/ Fresh FP
Wtd Partner Testing defects/Fresh FP
Actual stories / Planned stories (in FP)
PQ Overall (Y) = (5.13) Rework Ratio + (4115.4)Defect density(in process/KLOC)+(-8.05) Backlog Index (Actual/Plan)
Project overview (scope, team, start & end dates, lifecycle model) – Focus projects as well as non-focus projects;
Voyager

29. PPM Model - HORIZON-HRES
Actual PQ= *Shelf *Review Effectiveness+8.295*TC/TFP
Based on the R-Sq(adj)
and P-value of
the X factors
considered , PPM
Is recommended for use.

30. QPPO Mapping - Hawaiian
Quantitative Goal for Process Performance Objective (QPPO)
Sub Process
Sub Process Performance Metrics
Sub process Controllable Metric
Test case execution productivity
Current goal is 17
Planned goal for Oct 2016 is 18 (5% Improvement)
Test Case Execution
(TC Executed/No. of Days)*100
Testing effectiveness
(Bugs raised/ Test cases executed for a month)
Team’s average domain knowledge
(for the quarter)

31. Productivity Execution
Steps for building PPM – Overall Productivity - Hawaiian
PPM is build using Multiple Linear Regression
Base Data Used is
Months
Productivity Execution
Bugs raised/
Test case
 Average domain
knowledge
July '15 18
22.82
Aug '15 16
21.8
Sept '15 17
21.82
Oct '15
21.52
Nov '15
Dec '15 19
22.32
Jan '16
21.93
Feb '16 21
23.93
Regression Equation:
Productivity Execution = * (Bugs raised/Test case) * (Average Domain Knowledge)

32. PPM - Hawaiian Regression Equation:
Productivity Execution = * (Bugs raised/Test case) * (Average Domain Knowledge)
Based on the R-Sq(adj) and P-value of the X factors considered , PPM is recommended for use
Predictor Coef SE Coef T P VIF
Constant
Bugs raised/Test case
Average Domain Knowledge
S = R-Sq = 93.8% R-Sq(adj) = 91.3%
Analysis of Variance
Source DF SS MS F P
Regression
Residual Error
Total
Source DF Seq SS
Bugs raised/Test case
Average Domain Knowledge

33. Average Domain knowledge
Baselines of the factors for process stability - Hawaiian
Productivity
Bugs raised/Test Case
Average Domain knowledge
Mean
17.4
0.036
22.6
Std Dev
1.85
0.01
1.06
UCL
22.92
0.07
25.76
LCL
11.83
0.00
19.39

34. PPM Model – BA Responsive
Model Parameter Description:
Parameter
Formula
Baseline
STDEV
Remarks
Productivity
Productive Capacity/ Sprint
% Change in Story
Story point add during sprint / Initial Story point
25.594
Defect Density (/hrs)
Total Weighted / Story Point
Resource Utilization
Project overview (scope, team, start & end dates, lifecycle model) – Focus projects as well as non-focus projects;
#34 34 34

35. PPM Model – BA Responsive
Main Process Metric (QPPO)
Performance Parameter
Controllable Parameter
Productivity (Y)
% Change in Story
Productivity (Y) = %change in story Defect Density Resource utilization
Project overview (scope, team, start & end dates, lifecycle model) – Focus projects as well as non-focus projects;

36. PPM Model – BA Responsive
Based on the R-Sq(adj)
and P-value of
the X factors
considered , PPM
Is recommended for use.

37. PPM Model – NN Model Parameter Description: Metric Type Sub-process
Metric Name
Metric Definition
Metric Attribute
QPPO
Project Management
Resource Capacity Utilization%
(Actual effort spent in a specific period)/(Planned effort during that period)
Efficiency
Performance
MR Resolution
MR Service Rate
(no. of MRs serviced during the specified period)/(no. of days in the specified period)
Controllable
Average Exp.
Average Exp of team in years
Training
Cross IBIS Training %
Percentage of modules known to different Team members

38. PPM Model – NN Model Equations:
Resource-Capacity-Utilization % = (2.024)*MR-Arrival-Rate + ( )*Average-Exp. + (5.65)*Cross IBIS Training % +_3.763
Y-variable
Resource-Capacity-Utilization %
X-Variable
MR-Arrival-Rate
Average-Exp.
Cross IBIS Training %
Confidence Level
95%
Adjusted R
P-Value
P-Value for Normality of Standard Residual

39. PPM Model – NN Baseline: Metric Name Average St. Deviation
Resource Capacity Utilization%
MR Service Rate
Average Years of Exp.
Cross IBIS Training %

40. PPM Model – NN Back Justification:
This project uses only efficiency parameters for generating PPM for predicting outcome of project. This is because the effectiveness parameter (Rejection Index) is well controlled and always meeting it’s goal except in one or two cases.
Back

41. Thank You