1. IV&V Facility PI: Katerina Goseva – Popstojanova Students: Sunil Kamavaram & Olaolu Adekunle Lane Department of Computer Science and Electrical Engineering West Virginia University, Morgantown, WV katerina@csee.wvu.edu Real-World Software Reliability Assessment (WVU UI#7: Sensitivity of Software Reliability to Operational Profile Errors: Architecture-Based Approach)

2. IV&V Facility 2 What we are doing? Anyone can see a fire What we need are smoke detectors But what about the sensitivity and accuracy of the alarms ?

3. IV&V Facility 3 Problem statement & Our goal  Traditional view: Point estimate of software reliability computed from the model using point estimates of input parameters  Problem: Estimation of a trustworthy operational profile is difficult  IV&V information on operational profiles - limited, may be inaccurate  Single operational profile could not be sufficient to describe the use by different users  Software systems evolve - operational profile may change  Our goal: Reliability “sensitometer” that enables us to answer the question “How parameters uncertainty propagates into overall application reliability?”  Develop an architecture-based methodology for uncertainty analysis of software reliability & apply it on case studies

4. IV&V Facility 4 What we can do?  Benefits to IV&V  Software reliability assessment throughout the life cycle (keeping track of the software evolution)  Allocation of testing efforts  Software certification Reliability frequency chart & distribution fitting Certainty bands (percentiles) Entropy as a measure of uncertainty Execution rates & uncertainty of components

5. IV&V Facility 5 Architecture - based methodology for uncertainty analysis Uninformed Approach (maximum entropy) Uninformed Approach (maximum entropy) Intended Approach (historical data, UML) Intended Approach (historical data, UML) Informed Approach (component traces) Informed Approach (component traces) 1-p 23 1-p 12 1 2 E 3 p 23 p 12 1 Fault injection Fault injection Non-failed executions Non-failed executions Growth models Growth models R1R1 R2R2 R3R3 Uncertainty analysis

6. IV&V Facility 6 Methods for uncertainty analysis Uncertainty analysis Sensitivity studies Entropy Confidence intervals Probability distributions Analytical Monte Carlo simulation Method of moments Perturbation analysis

7. IV&V Facility 7 Choice of the method  Choose the method using the following criteria  Data requirements & ability to collect data  Reliability measures  Accuracy of the solution  Scalability with respect to the number of components  Our goal: fill the table Method Data requirements Reliability measures Accuracy of the solution Scalability

8. IV&V Facility 8 Construction of the software architecture model 1 2 E 3 1-p 23 1-p 12 p 23 p 12 1  Structural phase – establishment of static software architecture  Software specifications  Architectural design  Parser-based or lexically based tools (SIAT tool - Titan Systems Corporation)  Statistical phase – estimation of the relative frequencies of component interactions, that is, transition probabilities  Uniform distribution – maximum entropy approach  Historical data  Software specification (e.g. UML use case & sequence diagrams)  Component traces from profiles or test coverage tools (Testing tool for JSC AERCam project - Dr.Yann-Hang Lee, ASU)

9. IV&V Facility 9 European Space Agency case study Informed Approach (component traces) Informed Approach (component traces) 1-p 23 1-p 12 1 2 E 3 p 23 p 12 1 Fault Injection (real faults) Fault Injection (real faults) R1R1 R2R2 R3R3 Two faulty versions were obtained reinserting the real faults discovered during the integration testing and operational usage Component traces obtained during testing were used for constructing software architecture & estimating transition probabilities  Almost 10.000 lines of C code  The program has been extensively used after the last fault removal without failures; this gold version is used as an oracle

10. IV&V Facility 10 Parameter estimation  Two versions  Version A: faulty components 1&2, fault-free component 3  Version B: faulty components 2, fault-free components 1&3  Transition probabilities where is the number of times control was transferred from component i to component j, and  Component reliability where is the number of failures and is the number of executions of component i in N randomly generated test cases 0.68660.7364B 0.77040.5933A p 23 p 12 Version R 1 R 2 R 3 A0.84280.83461 B1 1

11. IV&V Facility 11 Construction of the architecture – based software reliability modelF 1-R 1 1-R 2 1-R 3 E 3 (1-p 23 )R 2 1 2 p 23 R 2 (1-p 12 )R 1 p 12 R 1 R3 R3 C 1

12. IV&V Facility 12 Traditional View: Point estimates of software reliability  Actual reliability of the software where F is the number of system failures in N randomly generated test cases  Estimated reliability from the model  Results Version Actual reliability Estimated reliability Error A0.73930.76012.81% B0.8782 0%

13. IV&V Facility 13 Methods for uncertainty analysis Uncertainty analysis Sensitivity studies Entropy Confidence intervals Probability distributions Analytical Monte Carlo simulation Method of moments Perturbation analysis

14. IV&V Facility 14 Sensitivity of software reliability to variations in operational profile Version A reliabilityVersion B reliability R max = 0.8414 R min = 0.7048 R max = 0.9983 R min = 0.8363

15. IV&V Facility 15 Methods for uncertainty analysis Uncertainty analysis Sensitivity studies Entropy Confidence intervals Probability distributions Analytical Monte Carlo simulation Method of moments Perturbation analysis

16. IV&V Facility 16 Uncertainty study based on entropy  Entropy quantifies the uncertainty present in a stochastic source where represents the usage distribution and the transition probabilities  Higher entropy implies an exponentially greater number of statistically typical paths  Maximum entropy – all transitions that are exit arcs from each state are equiprobable

17. IV&V Facility 17 Uncertainty of the operational profile H max = 0.5514 H min = 0.0404  Operational profile A (H= 0.4707) is more uncertain than operational profile B (H= 0.4604)  Software systems that have uniform operational profile are more uncertain and thus would require more testing H max = 0.5514 H min = 0.0404

18. IV&V Facility 18 Uncertainty of software reliability Operational profile  Considering software failure behavior increases the uncertainty for both versions compared to the uncertainty due to operational profile  Version B, which is more reliable, is less uncertain than version A Version A uncertainty Version B uncertainty Version A reliabilityVersion B reliability

19. IV&V Facility 19 Uncertainty of components for the operational profile  Uncertainty of component i is estimated using the conditional entropy  Uncertainty of component i will be higher if it transfers the control to more components and the transition probabilities are equiprobable Version A Version B

20. IV&V Facility 20 Uncertainty of components for the software reliability model  Uncertainty of component 1 version B remains the same because  For all other components uncertainty increases due to  Components that have higher expected execution rate, higher component uncertainty, and moderate reliability should be allocated more testing effort Version B Version A

21. IV&V Facility 21 Methods for uncertainty analysis Uncertainty analysis Sensitivity studies Entropy Confidence intervals Probability distributions Analytical Monte Carlo simulation Method of moments Perturbation analysis

22. IV&V Facility 22 Uncertainty study based on the method of moments  Method of moments involves the following steps 1.Obtain the expression for the system reliability using the architecture-based software reliability model 2.Expand the expression for system reliability using Taylor series 3.Determine the moments of the components reliabilities 4.Estimate the mean and the variance of the system reliability using the parameter moments and Taylor series coefficients

23. IV&V Facility 23 First order Taylor series  First order Taylor series expansion where is the mean component reliability, and  Mean reliability is  Variance of the reliability is where is the variance of the component reliability

24. IV&V Facility 24 Second order Taylor series  Second order Taylor series expansion  Mean reliability is  Variance of the reliability is

25. IV&V Facility 25 Method of moments for the case study  Second order approximation does not improve accuracy significantly First order Taylor series Second order Taylor series Mean reliability0.7601 Version AStandard deviation0.0825 Variance0.0068 Mean reliability0.8782 Version BStandard deviation0.0589 Variance0.0035  Version B is more reliable with less variance of the reliability

26. IV&V Facility 26 Methods for uncertainty analysis Uncertainty analysis Sensitivity studies Entropy Confidence intervals Probability distributions Analytical Monte Carlo simulation Method of moments Perturbation analysis

27. IV&V Facility 27 Uncertainty study based on Monte Carlo simulation  Monte Carlo simulation involves the following steps 1.Obtain the expression for the system reliability using the architecture-based software reliability model 2.Assign probability distributions to the transition probabilities and components reliabilities 3.Sample the distributions 4.Compute the reliability of the system using the sampled values 5.Repeat steps 3&4 until the desired number of values of system reliability has been generated 6.Calculate the moments, frequency chart and percentiles for the system reliability, do the distribution fitting

28. IV&V Facility 28 Variation of the operational profile: Frequency chart and distribution fitting Mean0.7600 Standard deviation (Spread of the distribution)0.0210 Variance (Spread of the distribution)0.0004 Skewness (Lean of the distribution)0.2072 Kurtosis (Peakedness of the distribution)2.6047

29. IV&V Facility 29  95% certainty band shows the range of values in which reliability has 95% chance of falling Variation of the operational profile: Percentiles 75% 95%

30. IV&V Facility 30 Convergence of the mean The estimation of the mean reliability converges after around 3000 iterations Mean reliability =0.7600

31. IV&V Facility 31  Reliability is more sensitive to p 1E ; the variance is positive  Reliability is also sensitive to p 12 ; the variance is negative Variation of the operational profile: Sensitivity measured by contribution to variance

32. IV&V Facility 32 Version AVersion B Variation of the operational profile and component reliabilities: Frequency charts Version AVersion B Mean0.75890.8780 Standard deviation (Spread of the distribution)0.08600.0660 Variance (Spread of the distribution)0.00740.0044 Coefficient of variation (Relative measure of spread)0.14930.0752 Skewness (Lean of the distribution)-0.5190-0.9646 Kurtosis (Peakedness of the distribution)3.13674.2254

33. IV&V Facility 33 Version AVersion B Variation of the operational profile and component reliabilities: Distribution fitting & percentiles

34. IV&V Facility 34 Making a choice Method Data requirements Reliability measures Accuracy of the solution Scalability Sensitivity Point estimatesSensitivity of the point estimate Exact analytical solutionLarge systems Entropy Point estimatesNAExact analytical solutionLarge systems Method of moments Moments of the parameters Moments  Approximate solution: accuracy may be increased by higher order Taylor series Small to medium systems Monte Carlo simulation  Distribution functions of the parameters  Generation of random numbers  Distribution  Moments  Approximate solution: accuracy may be increased by increasing the sample size  Sampling errors may be involved in case of long tail distributions Large systems

35. IV&V Facility 35 Accomplishments  Architecture-based methodology for uncertainty analysis of software reliability was developed  Four different methods already developed  These methods were illustrated on the European Space Agency software

36. IV&V Facility 36 Future work  Develop other methods for uncertainty analysis  Complete “Make a choice” table  Apply & validate all methods using NASA case studies  SIAT tool - Titan Systems Corporation  Testing tool for JSC AERCam project - Dr.Yann-Hang Lee, ASU