1 Predictors of customer perceived software quality Paul Luo Li (ISRI – CMU) Audris Mockus (Avaya Research) Ping Zhang (Avaya Research)

2 Need to View Quality from the Customer’s Perspective … We translate these advanced technologies into value for our customers … -IBM (#9 on the Fortune 500) … Our strategy is to offer products, services and solutions that are high tech, low cost and deliver the best customer experience. -HP (#11 on the Fortune 500) … We deliver unparalleled value to our customers. Only by serving our customers well do we justify our existence as a business -Avaya (#401 on the Fortune 500)

3 What Would be Ideal Predict customer perceived quality  Using customer characteristics  For each customer Key idea: Focus on the customer

4 Possible Applications of Predictions How do I plan deployment to meet the quality expectations of the customer? How do I target improvement efforts? How do I allocate the right resources to deal with customer problems Predict customer experience for each customer Identify possible causes of problems Predict customer interactions

5 Solutions for Software Producers How do I plan deployment to meet the quality expectations of the customer? How do I target improvement efforts? How do I allocate the right resources to deal with customer problems Predict customer experience for each customer Identify possible causes of problems Predict customer interactions

6 To Improve Customer Perceived Quality How do I plan deployment to meet the quality expectations of the customer? How do I target improvement efforts? How do I allocate the right resources to deal with customer problems Predict customer experience for each customer Identify possible causes of problems Predict customer interactions

7 Gaps in Current Research Prior work examined:  Software defect prediction for a single customer (Musa et al. 1987, Lyu et al. 1996)  Software defect prediction for modules or features (Jones et al. 1999, Khoshgoftaar et al. 1996) Is not scalable

8 Not Focused on Customers Prior work examined:  Software defect prediction for a single customer (Musa et al. 1987, Lyu et al. 1996)  Software defect prediction for modules or features (Jones et al. 1999, Khoshgoftaar et al. 1996) Tell us nothing about a specific customer

9 Does not Capture other Aspects of Customer Perceived Quality Prior work examined:  Software defect prediction for a single customer (Musa et al. 1987, Lyu et al. 1996)  Software defect prediction for modules or features (Jones et al. 1999, Khoshgoftaar et al. 1996) Does not predict other aspects of customer perceived quality that are not code related.

10 Research Contributions Predict software defects for each customer in a cost effective manner Predict other aspects of customer perceived quality for each customer Empirically validate deployment, usage, software, and hardware predictors

11 Rest of This Talk The setting Customer interactions (outputs) Customer characteristics (inputs) Results Conclusion

12 Empirical Results from a Real World Software System Avaya telephone call processing software system  7 million+ lines of C/C++  Fixed release schedule Process improvement efforts Tens of thousands of customers  90% of Fortune 500 companies use it Professional support organization

13 Data Used are Commonly Available Customer issue tracking system  Trouble ticket database The equipment database Change management  Sablime database Data collected as a part of everyday operations Data sources available at other organizations e.g. IBM and HP

14 Data collected as a part of everyday operations Data sources available at other organizations e.g. IBM and HP At Other Organizations Customer issue tracking system  Trouble ticket database The equipment database Change management  Sablime database

15 Customer Interactions (Outputs) We assume customer interaction == customer perceived quality Five customer interaction (Chulani et al. 2001, Buckley and Chillarege 1995) within 3 month of deployment  Software defects: high impact problem  System outages: high impact problem  Technician dispatches  Calls  Automated alarms Important for Avaya and likely for other organizations as well

16 Examine Customer Installations Months after general availability Number of deployments 1 5

17 Capture Characteristics of Each Installation Months after general availability Number of deployments 1 5 Customer 1: Deployed first month, a Large system, Linux… Customer 2: Deployed first month, a Small system, Windows… Customer 3: Deployed first month, a Large system, Proprietary Os… Customer 4: Deployed first month, a Small system, Linux… Customer 5: Deployed first month, a Large system, Linux…

18 Analyze Using Statistical Analysis Months after general availability Number of deployments 1 5 Customer 1: Deployed first month, a Large system, Linux… Customer 2: Deployed first month, a Small system, Windows… Customer 3: Deployed first month, a Large system, Proprietary Os… Customer 4: Deployed first month, a Small system, Linux… Customer 5: Deployed first month, a Large system, Linux… SimilaritiesDifferences

19 Category of Predictors (Kinds of Inputs) We examine:  Deployment issues  Usage patterns  Software platform  Hardware configurations Prior work examines:  Software product  Development process Common sense issues, but lack empirical validation

20 Category of Predictors (Kinds of Inputs) We examine:  Deployment issues  Usage patterns  Software platform  Hardware configurations Prior work examines:  Software product  Development process Key idea: From the customer’s perspective, they are not good predictors (i.e. do not vary for a single release)

21 Specific Predictors (Inputs) Total deployment time  deployment issues Operating system  software platform, hardware configurations System size  hardware configurations, software platform, usage patterns Ports  usage pattern, hardware configurations Software upgrades  deployment issue

22 Recap Predict for each customer (outputs):  Software defects  System outages  Technician dispatches  Calls  Automated alarms Using Logistic regression and Linear regression Using predictors (inputs):  Total deployment time  Operating system  System size  Ports  Software upgrades For a real world software system

23 Example: Field Defect Predictions

24 Predictors

25 Nuisance Variables

26 All Predictors are Important

27 The Most Important Predictor Total deployment time (deployment issue)  Systems deployed half way into our observational period are 13 to 25 times less likely to experience a software defect

28 May Enable Deployment Adjustments Total deployment time (deployment issue)  Systems deployed half way into our observational period are 13 to 25 times less likely to experience a software defect  May be due to software patching, better tools, more experienced technicians

29 Another Important Predictor Total deployment time (deployment issue) Operating system (software platform, hardware configurations)  Systems running on the proprietary OS are 3 times less likely to experience a software defect compared with systems on running the open OS (Linux)  Systems running on the commercial OS (Windows) are 3 times more likely to experience a software defect compared with systems running on the open OS (Linux)

30 May Allow for Targeted Improvement or Improved Testing Total deployment time (deployment issue) Operating system (software platform, hardware configurations)  May be due to familiarity with the operating system  May be due to operating system complexity

31 More Results in Paper The complete results and analyses for field defects Predictions for other customer interactions

32 Validation of Results and Method We accounted for data reporting differences  Included indicator variables in the models to identify populations (e.g. US or international customers) We independently validated the data collection process  Independently extracted data and performed analyses We interviewed personnel to validate findings  Programmers  Field technicians

33 Summary: Identified Predictors of Customer Perceived Quality We identified and quantified characteristics, like time of deployment, that can affect customer perceived quality by more than an order of magnitude

34 Summary: Modeled Customer Interactions We identified and quantified characteristics, like time of deployment, that can affect customer perceived quality by more than an order of magnitude We created models that can predict various customer interactions and found that predictors have consistent effect across interactions

35 Summary: Deployment is Important for High Reliability We identified and quantified characteristics, like time of deployment, that can affect customer perceived quality by more than an order of magnitude We created models that can predict various customer interactions and found that predictors have consistent effect across interactions We learned that controlled deployment may be the key for high reliability systems

36 Improve Customer’s Experiences You can target improvement efforts You can allocate the right resources to deal with customer reported problems You can adjust deployment to meet the quality expectations of your customers

37 Predictors of customer perceived software quality Paul Luo Li Audris Mockus (Avaya Research) Ping Zhang (Avaya Research)

38 Predicted Number of Calls Match Actual Number of Calls Calls for the next release Calls Time Predictions are made here