1. SAK5102 Software Evaluation Measuring External Attributes

2. External Attributes “ those whose measurement must take account both its behaviour and the environment it exists in” ● Internal attributes can be measured objectively, and usually fairly easily ● External attributes can involve aspects that are difficult to quantify, and often introduce subjectivity ● External attributes can usually only be measured toward the end of the development ● We usually measure internal attributes to predict measures of external attributes ● We validate our models (relationships between internal and external attributes) by measuring external attributes

3. Fitness for purpose ● Does it actually work ● Does it contain any “problems” ● De facto standard – defect density

4. The problem with problems ● A human error creates a fault that causes a failure – which to measure? - Process failures lead to errors - Developers fix faults - testing tries to cause failures - users report failures

5. Examples ● Spelling mistake in document ● Incorrect information in documentation that cause valuable data to be lost ● Incorrect information in documentation that causes data to have be re- entered ● One part of source code does not follow coding standard ● Screen background is wrong colour ● Organisation of controls leads to operator error causing death ● Code does not compile due to spelling error of variable name ● Code does not compile due to type-mismatch ● Application unexpectedly terminates on rare occasions with little data loss and is quick to restore ● Application unexpectedly terminates on rare occasions with significant data loss and takes significant amount of time to restore ● An uncommon key sequence causes the application to terminate in some circumstances ● Code contains function that is never called

6. Attributes of problems ● Location – where did the problem occur? ● Timing – when did the problem occur? ● Symptom – what was observed? ● End result – which consequences resulted? ● Mechanism – how did it occur? ● Cause – why did it occur? ● Severity – how much was the user affected? ● Cost – how much did it cost?

7. Example: cause of code fault (IEEE draft standard P1044) ● Logic problem - forgotten cases or steps - duplicate logic - extreme condition conditions neglected - unnecessary function - misinterpretation - missing condition test - checking wrong variable - iterating loop incorrectly ● Computational problem ● Data problem ● Enhancement ● ….. (table 5.2, pp166)

8. Defect Density ● Quality as Lack of defects ● Defects to be of two types: - known defects : discovered through testing, inspection, and other techniques - latent defects : may be present but we are as yet unaware ● Defect density = number of known defects product size - what is a defect? (error, fault, or failure) - what is product size? - what is product? - what is the process?

9. Evaluation ● Issues with definition ● Is a low defect density good or bad? - is it low because defects aren’t being reported? - does low defect density mean low number of unknown defects? ● Modules with higher than average defect densities are likely to be risky

10. Other defect-based metrics ● System spoilage = time to fix post-release defects total system development time ● Time has to be measured carefully and consistently ● Otherwise is reasonable metric

11. Other defect-based metrics ● AT&T Bell Laboratories ● Cumulative fault density – faults found internally ● Cumulative fault density – faults found by customers ● Total serious fault found ● Mean time to close serious faults ● High-level design review errors per KNCSL ● Low-level design errors per KNCSL ● Code inspection errors per inspected KNCSL ● …..

12. Defining Quality for Software ● External attribute, does depends on the environment ● E.g. “Modifiability” – the ease with which modifications can be made to the system - how big is the system being changed - how big is the modification to be made - how experienced are the staff in the (language, company, system,…)

13. Defining Quality of Software ● But specific notions of quality often also have many characteristics ● E.g. “modifiability” - how well documented is the code – “understandability” - how complex is the structure – “modularity” - the degree to which a change in one place will require changes in another – “viscosity” ● So different people have different ideas of what a given notion of quality may be

14. quality criteria Software Quality Models ● Meant to capture the composite nature quality ● For the use of the system being considered, consider the relevant quality factors, which are decomposed into quality criteria, which are measured by quality metrics

15. Example: McCall Software Quality Model

16. McCall model ● The McCall quality model is organized around three types of Quality Characteristics:  Factors (To specify): They describe the external view of the software, as viewed by the users.  Criteria (To build): They describe the internal view of the software, as seen by the developer.  Metrics (To control): They are defined and used to provide a scale and method for measurement. ● Since then, various quality models have been defined, adopted and enhanced over the years for example proposed by Boehm ● The need for one recognized standard quality model became more and more urgent. The ISO/IEC 9126 standard is the result of a consensus for a software quality model.

17. Example ● Consider correctness, which consists of completeness, traceability, consistency ➔ Apply checklist for each criteria for each of Requirement (R), Design (D), Implementation (I). E.g. Completeness ➢ Unambiguous references [R,D,I] ➢ All defined functions are used [R,D,I] ➢ All defined and referenced calling sequence parameter agree [D,I] ➢ Code agree with design [I] ➔ For each checklist item i assign 1 to “yes”, 0 to “no”, for each of the artifacts (R i, D i, I i ) and then compute, S com as ➔ Now compute correctness as

18. ISO 9126 standard quality model ● Called as Software Product Evaluation: Quality Characteristics and Guidelines for their Use ● As with McCall's this is also based on three levels:  Characteristics (Functionality, Reliability, Usability, Efficiency, Maintainability, Portability);  Sub-characteristics;  Metrics. ● Each characteristic is refined to a set of sub-characteristics and each sub-characteristic is evaluated by a set of metrics. Some metrics are common to several sub-characteristics.

19. ISO9126 - characteristics ● Each type of software has its own quality requirements. For example, if we consider:

20. ISO9126 : example Factor criteria metrics maintainability correctability Testability Expandability Fault counts Degree of testing effort Change count Closing time isolate/fix time fault rate Statement coverage branch coverage test plan completeness Resource prediction effort expendature Change effort change size change rate

21. usability ● The extent to which the product is convenient and practical to use ● The probability that the operator of a system will not experience a user interface problem during a given period of operation under a given operational profile ● How to measure this? ● How to predict this?

22. Measuring usability ● Evidence of good usability: well-structured manuals, good use of menus and graphics, informative error messages, help functions, consistent interfaces ● Decomposition ➔ Entry level ➔ Learnability ➔ Handling ability

23. Maintainability ● The ease with which maintenance activities can be performed: corrective, adaptive, preventive, perfective (including new enhancements) ● Decomposition ➔ correctability ➔ testability ➔ expandability

24. Mean Time To repair (MTTR) ● Problem recognition time ● Administrative delay time ● Maintenance tools collection time ● Problem analysis time ● Change specification time ● Change time (including testing and review)

25. Prediction from internal attributes ● “complexity” ● Number of changes made (or changes per LOC) ● Readability – Fog Index

26. Reliability ● Probability of failure-free operation ● Depends on hardware/software environment and user ● Common metrics include defect density and mean time to failure

27. Mean time to failure ● What is failure ● Are all failures equal ● What is time ● How do we predict time

28. Reliability prediction ● Reliability theory – comes from the hardware world where components eventually wear out ● Create models that describe probability of failure ● Software is not hardware ● Software does not wear out ● As each defect is fixed, reliability should improve (in long term)