IV&V Facility 1 FY2002 Initiative: Software Architecture Metrics Hany Ammar, Mark Shereshevsky, Nicholay Gradetsky, Diaa Eldin Nassar, Walid AbdelMoez,

Slides:



Advertisements
Similar presentations
Design of Experiments Lecture I
Advertisements

September 6, Achieving High Software Reliability Taghi M. Khoshgoftaar Empirical Software Engineering Laboratory Florida Atlantic.
1 Independent Verification and Validation Current Status, Challenges, and Research Opportunities Dan McCaugherty IV&V Program Manager Titan Systems Corporation.
(Some) Software Engineering Research at NJIT Sergio Bogazzi, Yao Fei Chen, GuanJie Jiang, BoYu, Ali Mili.
A GOAL-BASED FRAMEWORK FOR SOFTWARE MEASUREMENT
Software Metrics II Speaker: Jerry Gao Ph.D. San Jose State University URL: Sept., 2001.
An Experimental Evaluation on Reliability Features of N-Version Programming Xia Cai, Michael R. Lyu and Mladen A. Vouk ISSRE’2005.
CSE 300: Software Reliability Engineering Topics covered: Software metrics and software reliability.
1 Validation and Verification of Simulation Models.
1 Jianwei Dai, Lei Wang, and Faquir Jain Department of Electrical and Computer Engineering University of Connecticut Analysis of Defect Tolerance in Molecular.
Software Verification and Validation (V&V) By Roger U. Fujii Presented by Donovan Faustino.
Methodology for Architectural Level Reliability Risk Analysis Lalitha Krothapalli CSC 532.
What Exactly are the Techniques of Software Verification and Validation A Storehouse of Vast Knowledge on Software Testing.
Software Architecture Risk Assessment (SARA) Tool Khader Basha Shaik Problem Report Defense Master of Science in Computer Science Lane Department of Computer.
SAS_08_AADL_Exec_Gluch MAC-T IVV Model-Based Software Assurance with the SAE Architecture Analysis & Design Language (AADL) California Institute.
Research Heaven, West Virginia Verification and Validation of Adaptive Systems Online Failure Detection and Identification for IFCS through Statistical.
بسم الله الرحمن الرحيم الحمد لله ، والصلاة والسلام على رسول الله
University of Coimbra, DEI-CISUC
Introduction to Computers
Software Architecture Metrics Hany Ammar, Mark Shereshevsky, Ali Mili, Walid Rabie and Nicholay Gradetsky Lane Department of Computer Science & Electrical.
VTT-STUK assessment method for safety evaluation of safety-critical computer based systems - application in BE-SECBS project.
Research Heaven, West Virginia 1 FY 2004 Initiative: Risk Assessment of Software Architectures Hany Ammar, Katerina Goseva-Popstojanova, Ajith Guedem,
1 Software Quality CIS 375 Bruce R. Maxim UM-Dearborn.
1 Department of Electrical and Computer Engineering University of Virginia Software Quality & Safety Assessment Using Bayesian Belief Networks Joanne Bechta.
IV&V Facility PI: Katerina Goseva – Popstojanova Students: Sunil Kamavaram & Olaolu Adekunle Lane Department of Computer Science and Electrical Engineering.
Research Heaven, West Virginia A Compositional Approach for Validation of Formal Models Bojan Cukic, Dejan Desovski West Virginia University NASA OSMA.
INFORMATION THEORETIC METRICS FOR SOFTWARE ARCHITECTURES Monday, March 30, 2001 Ali Mili West Virginia University.
UML Diagrams: Sequence Diagrams The Requirements Model, and The Dynamic Analysis Model Instructor: Dr. Hany H. Ammar Dept. of Computer Science and Electrical.
Research Heaven, West Virginia 1 FY 2003 Initiative: IV&V of UML Hany Ammar, Katerina Goseva-Popstojanova, V. Cortelessa, Ajith Guedem, Kalaivani Appukutty,
1 Software Reliability Assurance for Real-time Systems Joel Henry, Ph.D. University of Montana NASA Software Assurance Symposium September 4, 2002.
West Virginia University Towards Practical Software Reliability Assessment for IV&V Projects B. Cukic, E. Gunel, H. Singh, V. Cortellessa Department of.
Software Engineering 2 Software Testing Claire Lohr pp 413 Presented By: Feras Batarseh.
IV&V Facility 1 FY 2002 Initiative IV&V of UML Hany Ammar, Katerina Goseva-Popstojanova, V. Cortelessa, Ajith Guedem, Diaa Eldin Nassar, Walid AbdelMoez,
Software Engineering Laboratory, Department of Computer Science, Graduate School of Information Technology and Science, Osaka University Dependence-Cache.
1 11 Simple Linear Regression and Correlation 11-1 Empirical Models 11-2 Simple Linear Regression 11-3 Properties of the Least Squares Estimators 11-4.
September Bound Computation for Adaptive Systems V&V Giampiero Campa September 2008 West Virginia University.
1 Program Testing (Lecture 14) Prof. R. Mall Dept. of CSE, IIT, Kharagpur.
Estimating Component Availability by Dempster-Shafer Belief Networks Estimating Component Availability by Dempster-Shafer Belief Networks Lan Guo Lane.
Chap. 5 Building Valid, Credible, and Appropriately Detailed Simulation Models.
Research Heaven, West Virginia 1 FY 2004 Initiative: Risk Assessment of Software Architectures Hany Ammar, Katerina Goseva-Popstojanova, Ajith Guedem,
THE IRISH SOFTWARE ENGINEERING RESEARCH CENTRELERO© What we currently know about software fault prediction: A systematic review of the fault prediction.
Research Heaven, West Virginia FY2003 Initiative: Hany Ammar, Mark Shereshevsky, Walid AbdelMoez, Rajesh Gunnalan, and Ahmad Hassan LANE Department of.
CS532 TERM PAPER MEASUREMENT IN SOFTWARE ENGINEERING NAVEEN KUMAR SOMA.
Software Architecture Risk Assessment (SARA) Tool Khader Shaik, Wallid Abdelmoez, Dr. Hanny Ammar Lane Department of Computer Science and Electrical Engineering,
1 Software Engineering: A Practitioner’s Approach, 6/e Chapter 15a: Product Metrics for Software Software Engineering: A Practitioner’s Approach, 6/e Chapter.
West Virginia University Sherif Yacoub, Hany H. Ammar, and Ali Mili A UML Model for Analyzing Software Quality Sherif Yacoub, Hany H. Ammar, and Ali Mili.
Prof. Hany H. Ammar, CSEE, WVU, and
Research Heaven, West Virginia PI: Katerina Goseva – Popstojanova Students: Ajay Deep Singh & Sunil Mazimdar Lane Dept. Computer Science and Electrical.
Deploying Analytical Redundancy for System Fault Tolerance V. Cortellessa, D. Del Gobbo, A. Mili, M. Shereshevsky, and Z. Zhuang CSEE Dept. West Virginia.
Building Valid, Credible & Appropriately Detailed Simulation Models
CORRELATION-REGULATION ANALYSIS Томский политехнический университет.
CS223: Software Engineering Lecture 25: Software Testing.
Model-Based Risk Assessment
Software Testing.
Achieving High Software Reliability
بسم الله الرحمن الرحيم الحمد لله ، والصلاة والسلام على رسول الله
Software Risk Assessment based on UML models
Strengthening Key Performance Indicators and Quality Assurance in Research in Ugandan Universities: A Case Study of Islamic University in Uganda. DR. MATOVU.
Fault Injection: A Method for Validating Fault-tolerant System
Chapter 13 Quality Management
Eastern Mediterranean University Department of Mechanical Engineering
Software Architecture Risk Assessment (SARA) Tool
Software Verification and Validation
Software Verification and Validation
Automated Analysis and Code Generation for Domain-Specific Models
Building Valid, Credible, and Appropriately Detailed Simulation Models
Methodology for Architectural Level Reliability Risk Analysis
Software Verification and Validation
Introduction to Computers
Presentation transcript:

IV&V Facility 1 FY2002 Initiative: Software Architecture Metrics Hany Ammar, Mark Shereshevsky, Nicholay Gradetsky, Diaa Eldin Nassar, Walid AbdelMoez, J. Shriver, A. Jalan LANE Department of Computer Science and Electrical Engineering West Virginia University Ali Mili, Bo Yu, Yan Wang, Krupa Doshi College of Computing Science New Jersey Institute of Technology The OSMA Software Assurance Symposium Berkley Springs, WV, September, 2002 The more information I give the more errors I may pass– Couched by Tim WVU UI: Software Architecture Metrics

IV&V Facility 2 Outline Core idea The amount of information Transferred What we can do Benefits Project Overview Research Objectives Quantitative Factors Error Propagation Change Propagation Future Work

IV&V Facility 3 Core idea The amount of information Transferred If component A talks to components B –It is important how much information can be transferred –In Information theoretic terms this is measured by the entropy of data and control information Standard view –Call graphs – control-flow and data-flow information between components –“This talks to that” What is the amount of information transferred? –example1: Binary information- entropy is 1-bit –example2: 1000 possible commands – entropy is 10-bits Intuition: –Error propagation in example2 > Error propagation in example1

IV&V Facility 4 What We Can Do Develop error propagation matrix (details follow) Component B (listens) alarm!!! Component A (talks)

IV&V Facility 5 Benefits Use metrics to identify and focus on problem areas in the architecture There is a need to analyze the quality of reference architectures in product-line architectural development (NASA Reference Architecture – the SEEDS project at GFSC, Workshop was held in June , 2002 in San Diego, CA) The developments of techniques for measuring Error/Change Propagation help analysts identify trouble spots in the architecture (advances the state of the art) The development of automated tools help the analyst apply these techniques on large architectures (advances the state of the practice)

IV&V Facility 6 Project Overview FY01 Defined quantitative factors that are relevant to qualitative attributes of the architecture –Error Propagation –Change Propagation –Requirements Propagation Defined information theoretic (Entropy-based) architecture metrics

IV&V Facility 7 Project Overview (cont.) FY02 Establish analytical/and empirical relationships between the Entropy-based metrics and Error/Change propagation factors Automate computation of the metrics, and apply to a NASA case study, –Architecture Metrics Tool to be presented in the tools session,

IV&V Facility 8 Project Overview (cont.) FY03 Apply the process and metrics to several NASA case studies - Use sanitized Architecture information to estimate the quantitative factors, (links to SIAT tool project with Mark Suder) - Compare Entropy-based metrics to metrics produced from existing metrics tools such as McCabe, (links to Tim Menzies machine learning tool) Conduct empirical Studies to validate the analytical and empirical relationships

IV&V Facility 9 Research Objectives Define information metrics for software architectures based on information theoretic measures Automate the process of computing these metrics. Apply the process and metrics to NASA projects Validate the correctness and usefulness of these metrics.

IV&V Facility 10 Quantitative Factors Selected and defined the following factors Error Propagation, Significance for Reliability Change Propagation, Significance for Maintainability Requirements Propagation, Significance for Evolvability

IV&V Facility 11 Definition of Quantitative Factors: Error Propagation  The error propagation (EP) is a measure of the probability that an error in the message sent by A will cause an error in B. S X AB x x` x OR Error

IV&V Facility 12 Error Propagation AB VABVAB v1v1 v3v3 v2v2 S1S1 S2S2 S3S3 EP(A  B) = Observed Message Redundancy (OMR) TR(V A  B, S B ) measures the average amount of redundancy in messages that B receives from A as seen by component B Intrinsic Message Redundancy (IMR) R(V A  B ) measures the intrinsic amount of redundancy in the messages that B receives from A

IV&V Facility 13 Static Error Propagation Matrix Error Propagation Coefficient (static) =

IV&V Facility 14 Dynamic Error Propagation Matrix Error Propagation Coefficient (dynamic) =

IV&V Facility 15 Controlled Experiments Empirical Validation of Error Propagation Analysis Extract Model Info..rtmdl Design Experiments Corrupt Model Simulate and Log Normal Log Error Logs Analyze Logs Empirical Error Propagation Components, Connectors Connector, Message MATLAB Summit BASIC Fault Model

IV&V Facility 16 Error Propagation Controlled Experiments Results The following correlation results are based on 3562 experiments The correlation goes down as the number of injected faults in the experimental side goes down The over all correlation coefficient between the two matrices =

IV&V Facility 17 Change Propagation We define Change Propagation from component A to component B as the probability that a change in A due to corrective/ perfective maintenance requires a change in B to maintain the overall function of the system. We propose an analytical formula that approximate change propagation, and present conducted empirical study to measure change propagation.

IV&V Facility 18 Change Propagation Analytical study Where |I X | and |O X | stand for the number of inputs and outputs, respectively of component X. DIR(Deterministic injectivity rate) of X is defined as An upper bound on Change Propagation Factor CP between two components A and B

IV&V Facility 19 Change Propagation Empirical Study In our experiment, we randomly select a large amount of changes within each component, then go into the source code level to see, if the change will propagate to other components. By computing for each pair of components (A,B), the number for which change in A cause a change in B, we are able to derive the change propagation matrix.

IV&V Facility 20 Change Propagation Matrix (Empirical study)

IV&V Facility 21 Future Work Investigate correlation between Error Propagation estimates and data measured in several NASA software (SIAT using X34, ITOS case studies) Investigate correlation between Change Propagation estimates and data measured from several applications, for more examples of changes. Investigate the correlation between static and dynamic metrics Develop a methodology for estimating Requirements Propagation