ODC Level Performance Models

ODC PPM Following ODCs are included: SITATesting SABRE VOYA SEI TRIVENT NITL C4 HORIZON-HRES Hawaiian BA Responsive NN

PPM Model - SITATesting Model Parameter Description: Metric Type Sub-process Metric Name Metric Definition Metric Attribute QPPO Test Execution Test Execution Productivity (Weighted test cases executed )/ (total execution effort) Efficiency Level 1: Performance Test Execution Rate (Number of test cases executed )/ (total execution effort in person days) % Defect Rejected (Total number of defects rejected) / (total number of defects reported) Effectiveness Level 2: Controllable Test Design % Test Data Creation Effort (Total Test Data Creation Effort in Person days) / (Total planned Testing Effort in person days) * 100 Project Management Team Competency Planned teams domain competency % Review % Review Effort (Total review effort in Person days) / (Total planned Testing Effort in Person days) * 100 % Effort in RTB Refinement (Total RTB Refinement Effort in Person days) / (Total planned Testing Effort in person days) * 100

PPM Model - SITATesting Model Equations: Test Execution Productivity = 43.274+ (-195.841) * '% Defects Rejected'+ (1.769) * 'Test Execution Rate' +_5.317 % Defects Rejected = 0.123+ (-1.51) * '% Effort in RTB Refinement'+ (-0.698) * '% Review effort' +_0.013 Test Execution Rate = -0.818+ (96.809) * '% Test Data Creation effort '+ (14.227) * 'Team Competency' +_1.149 Y-variable Test Execution Productivity X-Variable Test Execution Rate % Defects Rejected   Confidence Level 95% Adjusted R 0.662535157 P-Value 0.036309138 0.045838288 0.003054625 P-Value for Normality of Standard Residual 0.325313266 Y-variable % Defects Rejected X-Variable % Review effort % Effort in RTB Refinement   Confidence Level 95% Adjusted R 0.607252001 P-Value 0.000512091 0.003774327 3.83729E-06 P-Value for Normality of standard Residual 0.624184368 Y-variable Test Execution Rate X-Variable Team Competency % Test Data Creation effort   Confidence Level 95% Adjusted R 0.771534742 P-Value 0.000809932 0.00139909 0.659211668 P-Value for Normality of standard Residual 0.600941212

PPM Model - SITATesting Baseline: Metric Name Average St. Deviation Test Execution Productivity 59.8305 9.1522 % Defect Rejected 0.0323 0.0206 Test Execution Rate 12.9376 2.4033 % Test Data Creation effort 0.0623 0.0126 Team Competency 0.5429 0.0916 % Effort in RTB Refinement 0.0181 0.0071 % Review effort 0.0902 0.0198

PPM Model - SITATesting Justification: Effectiveness parameters for Testing COE project are: Defects Leaked to Production % Defects rejected by Customer Historical project data on defect leaked to production shows that it is under control and there is not much variation with respect to meeting customer and organization goals. There is performance metrics level equation established for % Defect rejected by customer which is addressing the customer effectiveness goal. Back

Quantitative Process Performance Objective PPM Model - SEBRE BO and QPPO Mapping Business Objective Quantitative Process Performance Objective Improve CSS Rating to Desired Service Level Improve cycle time by 10% by Jan 2017 Note: Based on historical data from Jan 2016 until Dec 2016 we will re-look at this data at the end of Jan 2017

PPM Model - SEBRE QPPO Mapping Quantitative Goal for Process Performance Objective (QPPO) Sub Process Sub Process Performance Metrics Sub process Controllable Metric Improve the Activation Cycle time(in weeks) Planned goal is 10% improvement of the cycle time Current goal is 11 weeks Planned goal is 10 weeks for Jan 2017(10% Improvement) Activation implementation Duration of Requirement analysis. Duration of QA testing. Learning weightage/ Complexity Sabre experience

PPM Model - SEBRE PPM Regression Equation: Actual Cycle time (weeks) = 22.6 - 1.39 Sabre Experience - 3.60 Learning Weightage/ Complexity Based on the R-Sq(adj) and P-value of the X factors considered , PPM is recommended for use Predictor Coef SE Coef T P Constant 22.598 1.538 14.69 0.000 Sabre experience -1.3858 0.3636 -3.81 0.001 Learning weightage/ Complexity -3.5985 0.7998 -4.50 0.000 S = 2.06022 R-Sq = 76.4% R-Sq(adj) = 73.9% P value should be less than .05 and R square adjustment should be more than 70. This shows that our equation is healthy or appropriate

VOYA – Process Performance Model Main Process Metrics (QPPO) Performance Parameter Mean Time to Resolve (Y) Test coverage per MR (X1) Review Effectiveness per MR (X2) Regression Equation is MTTR = 4.08 - 0.204 Test Coverage per MR - 0.327 Review Effectiveness R square is 95.4% and R-square adjusted is 93.1% P- value of X parameters of equation is less than 0.05 #10 10 10

VOYA – Process Performance Model Predictor Coef SE Coef T P VIF Constant 4.0842 0.2586 15.79 0.000 Test Coverage per MR -0.20399 0.02513 -8.12 0.001 1.120 Review Effectiveness -0.3267 0.2751 -1.19 0.301 1.120 S = 0.261336 R-Sq = 95.4% R-Sq(adj) = 93.1% Analysis of Variance Source DF SS MS F P Regression 2 5.6348 2.8174 41.25 0.002 Residual Error 4 0.2732 0.0683 Total 6 5.9080 Source DF Seq SS Test Coverage per MR 1 5.5385 Review Effectiveness 1 0.0963 #11 11 11

Baselines of the factors for process stability MTTR Test Coverage per MR Review Effectiveness Mean 2.13 8.98 0.73 STDEV 0.99 3.73 0.41 Count 7

PPM Model-SEI The regression equation is Main Process Metric (QPPO) Performance Parameter LN((No of script passed/No. of script failed)) (Y) Unit Testing effort rate (X1) Review Effectiveness (X2) The regression equation is Y= - 1.177 + 1.215 * Unit Testing effort rate(X2) + 0.793 * Rev Eff Project Baselines Output Parameter (Y) Unit Testing effort rate (X1) Rev Eff. (X2) Mean 2.4 1.5 2.2 Std Dev 1.60 0.65 1.04 UCL 7.20 3.42 5.38 LCL 0.00

PPM Model-SEI Model Parameter Description: Metric Type Sub-process Metric Name Metric Definition Metric Attribute QPPO COUT LN-(First Pass Success) No. of script passed in first review by sql SME/ Total No. of script submitted for review to sql SME Effectiveness Controllable Unit Testing effort rate Unit testing effort for sql script (Hrs) /No of sql script submitted for review Review Effectiveness Internal review wt defect for sql scripts / Internal Review effort for sql script (Hrs)

Regression Statistics PPM Model-SEI – Web Factory Regression Equation SUMMARY OUTPUT Regression Statistics The regression equation is   Multiple R 0.942968 Y = - 1.177 + 1.215* Unit Testing effort rate(X2) + 0.793* Rev Eff R Square 0.889188 Adjusted R Square 0.876876 Standard Error 0.561249 Observations 21 ANOVA df SS MS F Significance F Regression 2 45.49794 22.74897 72.21887 2.52E-09 Residual 18 5.670006 0.315 Total 20 51.16795 Coefficients t Stat P-value Lower 95% Upper 95% Lower 95.0% Upper 95.0% Intercept -1.17738 0.323438 -3.64022 0.001872 -1.8569 -0.49787 Unit Testing effort rate 1.21526 0.294939 4.120384 0.000642 0.595617 1.834903 Review effectivness 0.793038 0.182358 4.348801 0.000387 0.409918 1.176158 P value is less than .05 for all the x’s and for intercept

Regression Statistics PPM – SEI CAS The Regression Equation is Defect Density( QA) = 0.91-0.09*(UTC coverage)-0.07( % code review effort) Based on the R-sq ( adj)and p value of x factors considered, PPM is recommended for use Regression Statistics Multiple R 0.894318 R Square 0.799805 Adjusted R Square 0.771206 Standard Error 0.07281 Observations 17 ANOVA   df SS MS F Significance F Regression 2 0.296512 0.148256 27.9659 1.29E-05 Residual 14 0.074218 0.005301 Total 16 0.370731   Coefficients Standard Error t Stat P-value Lower 95% Upper 95% Lower 95.0% Upper 95.0% Intercept 0.916552 0.100951 9.07919 3.05E-07 0.700034 1.13307 /User Story -0.09678 0.040547 -2.38692 0.031651 -0.18375 -0.00982 % Code review effort/ total engineering effort -0.07639 0.029719 -2.57038 0.022221 -0.14013 -0.01265

Baselines of the factors for process stability QA Defect Density Unit Test Coverage % Code Review Effort Mean 0.36 1.89 4.94 STDEV 0.15 0.72 0.98 Count 17 UCL 0.4 3.33 6.9 LCL 0.45 2.98 Operational Definition Factor Definition Metric Formula Unit Unit Testing Coverage Total no. of Unit test case executed per story Total number of Unit Test cases executed / User Story Number % Code review effort code review effort ( as a % of Total engineering effort) % of code review effort (Code review effort/ Total engineering effort *100) Percentage QA Defect Density No. of defects found in QA Phase per story point Weighted defects found in QA testing/ Actual velocity(Story Points)

Thrivent – Process Performance Model Main Process Metrics (QPPO) Performance Parameter Productivity (Y) % COQ (X1) Review Effectiveness (X2) % Resources Cross trained across applications (X3) Regression Equation is Productivity = 55.5 + 0.769 COQ % - 2.78 Review Effectiveness - 1.09 % Resources Trained Across Appl R square is 91.7% and R-square adjusted is 87.5% P- value of X parameters of equation is less than 0.05 #18 18 18

PPM – Thrivent The regression equation is   Productivity = 55.5 + 0.769 COQ % - 2.78 Review Effectiveness - 1.09 % Resources Trained Across Appl Predictor Coef SE Coef T P VIF Constant 55.52 12.95 4.29 0.005 COQ % 0.7686 0.1925 3.99 0.007 1.154 Review Effectiveness -2.7815 0.6640 -4.19 0.006 1.034 % Resources Trained Across Appl -1.0881 0.2380 -4.57 0.004 1.182 S = 2.09125 R-Sq = 91.7% R-Sq(adj) = 87.5% Analysis of Variance Source DF SS MS F P Regression 3 288.022 96.007 21.95 0.001 Residual Error 6 26.240 4.373 Total 9 314.262 Source DF Seq SS COQ % 1 142.524 Review Effectiveness 1 54.097 % Resources Trained Across Appl 1 91.400

Baselines of the factors for process stability Productivity (hr/MR) Cost of Quality (%) Review Effectiveness (ratio) % Resources trained across appl Mean 27.28 28.73 1.65 41.38 STDEV 5.12 1.22 0.83 2.69 Count 10

PPM Model – NITL Model Parameter Description: Parameter Formula Baseline STDEV Remarks Productivity 12.35 1.72 Defect Density (/hrs) 0.32 0.13 Resource Utilization 98.58 0.83 Project overview (scope, team, start & end dates, lifecycle model) – Focus projects as well as non-focus projects; #21 21 21

PPM Model – NITL Productivity (Y) = -314.52 + 3.43 * Actual capacity Utilisation - 23.67 * Defect Density Project overview (scope, team, start & end dates, lifecycle model) – Focus projects as well as non-focus projects;

PPM Model – NITL Based on the R-Sq(adj) and P-value of the X factors The regression equation is   Productivity = -314.52 + 3.43 * Actual capacity utilization - 23.67 * Defect density SUMMARY OUTPUT Regression Statistics Multiple R 0.955730115 R Square 0.913420053 Adjusted R Square 0.894180065 Standard Error 0.558569826 Observations 12 ANOVA df SS MS F Significance F Regression 2 29.62447607 14.81223803 47.47508 1.65E-05 Residual 9 2.80800225 0.31200025 Total 11 32.43247832 Coefficients t Stat P-value Lower 95% Upper 95% Lower 95.0% Upper 95.0% Intercept -109.40 27.17096618 -4.026473888 0.002989 -170.868 -47.9382 Actual Capacity Utilisation 1.25 0.271942317 4.610776428 0.001271 0.638689 1.869041 Defect Density -5.71 1.682270808 -3.391871542 0.007975 -9.51161 -1.90049 RESIDUAL OUTPUT Observation Predicted Actual Productivity Residuals 1 9.083415263 0.333335627 10.44086832 0.361953083 3 10.90861764 -0.202590501 4 13.31706564 0.243370436 5 12.5174413 0.052780207 6 11.99485749 -0.311343562 7 12.26692178 0.05527523 8 14.72873877 0.721177287 11.92008703 -0.236870855 10 14.02308773 -0.897086938 12.7945878 -0.75809102 14.16528933 0.638091005 Based on the R-Sq(adj) and P-value of the X factors considered , PPM Is recommended for use.

TC Executed / TC prepared Internal weighted defects / SP PPM Model – C4 Model Parameter Description: Metric Type Sub-process Metric Name Metric Definition Metric Attribute Improve Customer Satisfaction Construction ST Defects/ST TC Executed / TC prepared Effectiveness Testing Internal weighted defects / SP

PPM Model – C4 The regression equation is ST Defect /SP (Y) =0.21021 + (0.00039)*TC Create + (-0.00044)*Story Point Delivered + (-0.00247)*DB Components SUMMARY OUTPUT Regression Statistics   Multiple R 0.974973454 R Square 0.950573237 Adjusted R Square 0.934097649 Standard Error 0.024390517 Observations 9 ANOVA df SS MS F Significance F Regression 2 0.068646 0.034323 57.69586 0.000121 Residual 6 0.003569 0.000595 Total 8 0.072216 Coefficients t Stat P-value Lower 95% Upper 95% Lower 95.0% Upper 95.0% Intercept 0.550690079 0.088515 6.221412 0.000797 0.334101 0.767279 TC execution coverage by Sprint team -0.32642889 0.127291 -2.56443 0.042653 -0.6379 -0.01496 Weighted Defects/SP by Sprint team -0.8895684 0.250362 -3.55312 0.012027 -1.50218 -0.27695 RESIDUAL OUTPUT Observation Predicted Weighted Defects/SP by ITG Residuals 1 0.29639999 0.013678 0.131230635 -0.00244 3 0.069524578 0.035739 4 0.112044905 -0.01204 5 0.108044561 -0.04091 0.046643252 -0.01253 7 0.01263348 0.007775 0.005648856 0.00705 -0.00068816 0.003682

PPM Model – C4 Baseline: Metric Name Average St. Deviation Productivity 8.97 1.94 Wt. defect / SP 0.185 0.07 Resource Capacity Utilization 0.9344 0.0258 % Burnout 0.9516 0.2036- Average of Velocity (SP) 48.64861 11.0327 % of story changed 0.198614 0.139058

PPM Model - HORIZON-HRES Model Parameter Description:   Sources Plotting Freq. Remarks QPPO Contractual Directive Internal Directive Actual PQ 20% improvement from the baselined till stage 11. baselined till stage 11 is 19.7 to bring it to 16. 16 Iteration end Contractual requirement is to achieve 16 at the stage end . Variation from 17 to 12 is allowed and discussed with client. USL= 17 LSL= 12 Project overview (scope, team, start & end dates, lifecycle model) – Focus projects as well as non-focus projects; Process composition done with the result of Causal, hence new PPB were release and PPM also recalibrated with the new data set and parameters. 27 #27 27

PPM Model - HORIZON-HRES Main Process Metric (QPPO) Performance Parameter Controllable Parameter Productivity (Y) Rework Ratio (X1) Partner Testing Defects Density (X2) Backlog Index (X3) Delivered FP/ Effort Rework FP/ Fresh FP Wtd Partner Testing defects/Fresh FP Actual stories / Planned stories (in FP) PQ Overall (Y) = 19.56 + (5.13) Rework Ratio + (4115.4)Defect density(in process/KLOC)+(-8.05) Backlog Index (Actual/Plan) Project overview (scope, team, start & end dates, lifecycle model) – Focus projects as well as non-focus projects; Voyager

PPM Model - HORIZON-HRES Actual PQ= 72.212-429.774*Shelf-17.165*Review Effectiveness+8.295*TC/TFP Based on the R-Sq(adj) and P-value of the X factors considered , PPM Is recommended for use.

QPPO Mapping - Hawaiian Quantitative Goal for Process Performance Objective (QPPO) Sub Process Sub Process Performance Metrics Sub process Controllable Metric Test case execution productivity Current goal is 17 Planned goal for Oct 2016 is 18 (5% Improvement) Test Case Execution (TC Executed/No. of Days)*100 Testing effectiveness (Bugs raised/ Test cases executed for a month) Team’s average domain knowledge (for the quarter)

Productivity Execution Steps for building PPM – Overall Productivity - Hawaiian PPM is build using Multiple Linear Regression Base Data Used is Months Productivity Execution Bugs raised/ Test case  Average domain knowledge July '15 18 0.037244898 22.82 Aug '15 16 0.049034908 21.8 Sept '15 17 0.030233449 21.82 Oct '15 0.04112746 21.52 Nov '15 0.048905449 Dec '15 19 0.021046512 22.32 Jan '16 0.036443662 21.93 Feb '16 21 0.024444444 23.93 Regression Equation: Productivity Execution = - 15.4 - 64.8 * (Bugs raised/Test case) + 1.58 * (Average Domain Knowledge)

PPM - Hawaiian Regression Equation: Productivity Execution = - 15.4 - 64.8 * (Bugs raised/Test case) + 1.58 * (Average Domain Knowledge) Based on the R-Sq(adj) and P-value of the X factors considered , PPM is recommended for use Predictor Coef SE Coef T P VIF Constant -15.422 7.636 -2.02 0.099 Bugs raised/Test case -64.85 25.02 -2.59 0.049 1.599 Average Domain Knowledge 1.5822 0.3174 4.99 0.004 1.599 S = 0.543776 R-Sq = 93.8% R-Sq(adj) = 91.3% Analysis of Variance Source DF SS MS F P Regression 2 22.397 11.198 37.87 0.001 Residual Error 5 1.478 0.296 Total 7 23.875 Source DF Seq SS Bugs raised/Test case 1 15.047 Average Domain Knowledge 1 7.349

Average Domain knowledge Baselines of the factors for process stability - Hawaiian Productivity Bugs raised/Test Case Average Domain knowledge Mean 17.4 0.036 22.6 Std Dev 1.85 0.01 1.06 UCL 22.92 0.07 25.76 LCL 11.83 0.00 19.39

PPM Model – BA Responsive Model Parameter Description: Parameter Formula Baseline STDEV Remarks Productivity Productive Capacity/ Sprint 16.6876872 10.1738773 % Change in Story Story point add during sprint / Initial Story point 25.594 29.9488 Defect Density (/hrs) Total Weighted / Story Point 3.355257 4.386381 Resource Utilization 1.008463 0.092138 Project overview (scope, team, start & end dates, lifecycle model) – Focus projects as well as non-focus projects; #34 34 34

PPM Model – BA Responsive Main Process Metric (QPPO) Performance Parameter Controllable Parameter Productivity (Y) % Change in Story Productivity (Y) = 4.0 + 0.09 %change in story -11.02 Defect Density + 10.60 Resource utilization Project overview (scope, team, start & end dates, lifecycle model) – Focus projects as well as non-focus projects;

PPM Model – BA Responsive Based on the R-Sq(adj) and P-value of the X factors considered , PPM Is recommended for use.

PPM Model – NN Model Parameter Description: Metric Type Sub-process Metric Name Metric Definition Metric Attribute QPPO Project Management Resource Capacity Utilization% (Actual effort spent in a specific period)/(Planned effort during that period) Efficiency Performance MR Resolution MR Service Rate (no. of MRs serviced during the specified period)/(no. of days in the specified period) Controllable Average Exp. Average Exp of team in years Training Cross IBIS Training % Percentage of modules known to different Team members

PPM Model – NN Model Equations: Resource-Capacity-Utilization % = -328.978 + (2.024)*MR-Arrival-Rate + (100.303)*Average-Exp. + (5.65)*Cross IBIS Training % +_3.763 Y-variable Resource-Capacity-Utilization % X-Variable MR-Arrival-Rate Average-Exp. Cross IBIS Training % Confidence Level 95% Adjusted R 0.893981 P-Value 0.048979 0.009184 0.006057 P-Value for Normality of Standard Residual 0.059265

PPM Model – NN Baseline: Metric Name Average St. Deviation Resource Capacity Utilization% 51.91083 11.55799 MR Service Rate 3.481631 1.461467 Average Years of Exp. 2.304167 0.049075 Cross IBIS Training % 25.26297 1.034765

PPM Model – NN Back Justification: This project uses only efficiency parameters for generating PPM for predicting outcome of project. This is because the effectiveness parameter (Rejection Index) is well controlled and always meeting it’s goal except in one or two cases. Back

Thank You