1. Software Architecture based Performance and Reliability Evaluation Vibhu S. Sharma Ph.D. Scholar CSE, IITk

2. Cutting Edge 2005 IIT Kanpur Software architecture is the fundamental organization of a system, embodied in its components, their relationships to each other and the environment, and the principles governing its design and evolution. - IEEE 1471-2000. Software architecture is the fundamental organization of a system, embodied in its components, their relationships to each other and the environment, and the principles governing its design and evolution. - IEEE 1471-2000. Software architecture styles are like templates which are followed by commonly used systems: Software architecture styles are like templates which are followed by commonly used systems: Layered style : Web-based systemsLayered style : Web-based systems Pipe and Filter style : CompilersPipe and Filter style : Compilers...... Software Architecture Client 1 Web- server Application Server Client 2 Client nc A web-based software system DB

3. Cutting Edge 2005 IIT Kanpur Modeling the workflow of the Software Architecture We use finite absorbing Discrete Time Markov Chains (DTMC) for representing the software architecture workflow. We use finite absorbing Discrete Time Markov Chains (DTMC) for representing the software architecture workflow. Examples: Examples: These DTMCs can be used to find the average visit counts to various components for a typical request as well as the limiting probabilities. These DTMCs can be used to find the average visit counts to various components for a typical request as well as the limiting probabilities. Very useful for reliability as well as performance prediction. Very useful for reliability as well as performance prediction. 123C p 12 p 23 1 1-p 12 1-p 23 ESA Antenna array configuration tool architecture S0S0 S1S1 S2S2 S nl-1 S nl S 2nl+1 1 1 p 12 p 23 p (nl-2)(nl-1) p (nl-1)nl..... S nl+1 S nl+2 S 2nl-1 S 2nl 111 1 1..... 1- p (nl-1)nl 1-p 23 1-p 12 1 A typical layered architecture

4. Cutting Edge 2005 IIT Kanpur Architecture based Reliability prediction Issues/Questions: Issues/Questions: What is the reliability of a software system composed of unreliable components?What is the reliability of a software system composed of unreliable components? How does increasing the reliability of a particular component affects overall reliability ?How does increasing the reliability of a particular component affects overall reliability ? How does changing the architecture affect reliability ?How does changing the architecture affect reliability ? Reliability modeling using DTMCs Reliability modeling using DTMCs Model the component failure as a state in the DTMC and associate probabilities of transition (unreliabilities).Model the component failure as a state in the DTMC and associate probabilities of transition (unreliabilities). Overall reliability is the probability of ending in the correct state.Overall reliability is the probability of ending in the correct state. Another option is using a reward based methodAnother option is using a reward based method 1 2 3 4 5 S F p f1 p f2 p f3 p f4 p f5 (1-p f1 )p 12

5. Cutting Edge 2005 IIT Kanpur Architecture based performance prediction Issues/Questions: Issues/Questions: What is the maximum number of clients the system can able to handle before it saturates ? What effect does varying the number of clients have on the throughput and the average response time ? How should software components be allocated to the hardware nodes ? What changes should be made in the system to improve performance ? We take an example of layered software architecture. We take an example of layered software architecture. A Discrete Time Markov Chain (DTMC) is used to characterize the control flow in the layered software architecture. The DTMC is used to calculate the visit counts to various layers and then a closed queueing network (QN) model is constructed and solved. S0S0 S1S1 S2S2 S nl- 1 S nl S 2nl +1 1 1 p 12 p 23 p (nl-2)(nl- 1) p (nl-1)nl..... S nl +1 S nl +2 S 2nl- 1 S 2nl 111 1 1..... 1- p (nl-1)nl 1-p 23 1-p 12 1

6. Cutting Edge 2005 IIT Kanpur Some Analysis Results Average response time and throughput and bottleneck analysis Effect of change in individual reliabilities & workload on overall reliability

7. Cutting Edge 2005 IIT Kanpur Reliability and Performance Tradeoff Given a set of components and a set of machines to deploy these onto What are the performance & reliability characteristics of the setup under a given load ? Which architecture should be chosen for optimal performance and reliability ? What is the tradeoff ? Software failures of different types cause different effects on the performance of the system. Hardware availability also needs to be taken into account. Hardware availability also needs to be taken into account. Some configurations will result in high throughput with decreased reliability and vice-versa. Some configurations will result in high throughput with decreased reliability and vice-versa.

8. Combining Static Analysis and Testing Frameworks to find Software Defects Vipindeep V

9. Cutting Edge 2005 IIT Kanpur Defect Prevention during Software Development Write Code Compile/Build Customer Evaluate Program Specification Static analysis Test Suite Testing

10. Cutting Edge 2005 IIT Kanpur Testing + Static Analysis – How? Static Analysis Vs Testing Static Analysis Vs Testing Different areas altogether, but motive – identify defectsDifferent areas altogether, but motive – identify defects Combining the approaches… improved program checkingCombining the approaches… improved program checking Motivations for the Proposed Approach Motivations for the Proposed Approach Static analysis concernsStatic analysis concerns Testing is done.. Bugs are revealedTesting is done.. Bugs are revealed Ideal test suite => identify all bugsIdeal test suite => identify all bugs Non-ideal scenario of TestingNon-ideal scenario of Testing Identify defects, partially Identify defects, partially Some paths/blocks are “not (properly) tested”.. Why?? Some paths/blocks are “not (properly) tested”.. Why?? Static analysis effort on unexplored parts of the codeStatic analysis effort on unexplored parts of the code Our proposed approach Our proposed approach Focused Static Analysis with Path Pruning using Coverage DataFocused Static Analysis with Path Pruning using Coverage Data

11. Cutting Edge 2005 IIT Kanpur Overview of the approach

12. Cutting Edge 2005 IIT Kanpur Pros and Cons Advantages:Advantages: 1. New real errors can be identified which escaped in _all_paths analysis 2. Rigorous testing effort counted for better static analysis 3. Reduce execution time 4. Reduced warnings => Reduced Noise Disadvantages:Disadvantages: 1. Some errors escape in the pruned path analysis 2. Depends on testing framework

13. Cutting Edge 2005 IIT Kanpur Applied the approach with FindBugs, Jlint for JAVA Applied the approach with FindBugs, Jlint for JAVA Results from testing were encouraging Results from testing were encouraging Tested the applicability using Random Path Selection Tested the applicability using Random Path Selection Preliminary Results

14. Cutting Edge 2005 IIT Kanpur Conclusions Works well in practice Works well in practice Finds enough real defects to be usefulFinds enough real defects to be useful Noise is low enough that people can use itNoise is low enough that people can use it Scales well, so works on large very large code basesScales well, so works on large very large code bases More issues to explore.. More issues to explore.. Better algorithm to use testing dataBetter algorithm to use testing data Path profiles, Crash dumps – focused static analysisPath profiles, Crash dumps – focused static analysis Use static analysis results for focusing the testing effortUse static analysis results for focusing the testing effort Combining static and dynamic analysisCombining static and dynamic analysis Usability related issuesUsability related issues

15. State Based Testing Automation Atul Gupta Ph.D. Scholar

16. Cutting Edge 2005 IIT Kanpur State Based Testing Program behavior can be modeled as a state graph Program behavior can be modeled as a state graph A formal testing approach at all levels- unit, component or system level. A formal testing approach at all levels- unit, component or system level. Effective testing strategy to O-O software Effective testing strategy to O-O software Test Cases and Test Oracles can be automatically generated and evaluated Test Cases and Test Oracles can be automatically generated and evaluated Atul Gupta PhD Scholar

17. Cutting Edge 2005 IIT Kanpur Class CoinBox { unsigned totalQtrs; unsigned curQtrs; unsigned allowVend; public: CoinBox( ) { totalQtrs = 0; curQtrs = 0; allowVend = 0; } void retQtrs( ) { curQtrs = 0; } void addQtr( ) { curQtrs = curQtrs +1; if (curQtrs > 1) allowVend = 1; } void vend( ) { if (allowVend) { totalQtrs = totalQtrs + curQtrs; curQtrs = 0; allowVend = 0; } }; An Example : A CoinBox Class WAITQ1Q2 allowVend = 0 curQtrs = 0 allowVend = 0 curQtrs = 1 allowVend = 1 curQtrs >= 2 addQtr( ) retQtrs( ) vend( )

18. Cutting Edge 2005 IIT Kanpur State Based Testing Criteria All-Transition Coverage (AT) All-Transition Coverage (AT) All-Transition-pair Coverage (ATP) All-Transition-pair Coverage (ATP) Full Predicate coverage (FP) Full Predicate coverage (FP) Transition-Tree Coverage (TT) Transition-Tree Coverage (TT) All-Round-Trip path coverage (ART) All-Round-Trip path coverage (ART) Complete Sequence (CS ) Complete Sequence (CS ) WAITQ1Q2 allowVend = 0 curQtrs = 0 allowVend = 0 curQtrs = 1 allowVend = 1 curQtrs >= 2 addQtr( ) retQtrs( ) vend( )

19. Cutting Edge 2005 IIT Kanpur Generating Test Sequences CriterionPrefix(State) Test Sequence Test Oracle (State) All-Transition (AT): WAITQ1Q2 addQtr( )- retQtrs( ) addQtr( )-addQtr()- retQtrs( ) vend( ) WAITWAITWAIT All-Transition-Pair (ATP) At Q1 WAITWAIT addQtr( )- retQtrs( ) addQtr( )- addQtr( ) WAITQ2 At Q2 Q1Q1Q1Q2Q2Q2 addQtr( )- retQtrs( ) addQtr( )- addQtr( ) addQtr( )- vend( ) addQtr( )- retQtrs( ) addQtr( )- addQtr( ) addQtr( )- vend( ) WAITQ2WAITWAITQ2WAIT At WAIT Q2Q1Q2 retQtrs( )- addQtr( ) vend ( )- addQtr( ) Q1Q1Q1 Transition-Tree (TT) WAITWAITWAITWAIT addQtr( )- retQtrs( ) addQtr( )- addQtr( )-retQtrs( ) addQtr( )- addQtr( )-vend( ) addQtr( )- addQtr( )- addQtr( ) WAITWAITWAITQ2 WAITQ1Q2 allowVend = 0 curQtrs = 0 allowVend = 0 curQtrs = 1 allowVend = 1 curQtrs >= 2 addQtr( ) retQtrs( ) vend( ) WAIT Q1 Q2 addQtr( ) retQtrs( ) vend( ) WAIT Q2 WAIT

20. Cutting Edge 2005 IIT Kanpur Research Issues How to generate test cases automatically from a state model How to generate test cases automatically from a state model How to execute them in an automated manner How to execute them in an automated manner How to evaluate the results automatically How to evaluate the results automatically

21. Cutting Edge 2005 IIT Kanpur Our Approach Identify State Model of a Class Identify State Model of a Class Generate test sequences based on some Coverage Criteria Generate test sequences based on some Coverage Criteria Convert these sequences into JUnit format for Automatic Execution and Result evaluation Convert these sequences into JUnit format for Automatic Execution and Result evaluation AIM : Complete end-to-end Automation