1. SIM5102 Software Evaluation
Data Collection

2. What is good data?
Correctness - data collected according to the exact rules of definition of the metric
Accuracy - difference between the data and the actual value
Precision - number of decimal places needed to express the data
Consistent - no much difference in value when measured using different device or by different person, or repeatedly

3. What is good data?
Time-stamp - associate with a particular activity or time period to know when it was collected.
Replication - replicate for difference circumstances.

4. How to define the data?
Metric terminology must be clear and detailed, understood by all involved.
Two kinds of data
Raw data
Refined data
Direct and indirect measurement
Most organizations are interested in software quality, cost and schedule.

5. The problem with problems
Fig 5.2 (Software Quality Terminology)
Fault
occurs when a human error results in a mistake in some software product
Eg: developer misunderstand the user-interface requirement, and create a design with the wrong understanding, the design fault results in incorrect code.

6. The problem with problems
Failure
Departure of a system from its required behavior
Can be discovered during testing and operation
Can think of fault and failure as inside and outside views of the system.
Faults - problems that developers see
Failures - problems that the users see
Not every fault corresponds to a

7. The problem with problems
Reliability of software is defined in terms of failures observed during operation not faults.
Terminology is very important here. Refer to page 157 (error, anomalies, defects, bugs, crashes)
If investigation of failure reveal the fault, change is made to the product to remove it.

8. The problem with problems
Key elements of a problem:
Location - where did it occur
Timing - when did it occur
Symptom - what was observed
End result - which consequences resulted
Mechanism - how did it occur
Cause - why did it occur
Severity - how much was the user affected
Cost - how much did it cost

9. The problem with problems
These 8 attributes are mutually independent (applies only to initial measurement).
Is known as orthogonality
Orthogonality also refers to a classification scheme within a particular category.
Non-orthogonality can lead to data loss or corruption, Eg 5.1

10. The problem with problems
The 8 attributes should suffice for all types of problems, but are answered differently based on whether it is faults, failures or changes.

11. Failures
Focus on external problems of the system: installation, chain of events, effect on user or other system, cost.
Fig 5.3 (failure report), page 160
Location - code that uniquely identifies the installation and platform on which the failure was observed.
Timing - real time of occurrence, execution time up to the occurrence of failure.

12. Failures Cause Severity type of trigger, type of source
Often cross-referenced to fault and change reports.
Severity
how serious the failure’s end result
Classification for safety-critical system:
Catastrophic
Critical
Significant
Minor
Can also be measured in terms of cost

13. Failures
Cost - how much effort and other resources to diagnose and response to the failure
Ninth category; Count - number of failures in a stated time interval.
Mechanism, cause and cost can only be completed after diagnosis.
Data collection form for failure should include at least five categories.

14. Failure Report
Location: such as installation where failure was observed
Timing: CPU time, clock time or some temporal measure
Symptom: type of error message or indication of failure
End result: description of failure, such as “operating system crash”, “ services degraded”, “loss of data”, “wrong output”, “no output”
Mechanism: chain of events, including keyboard commands and state data, leading to failure
Cause: reference to possible fault(s) leading to failure
Severity: reference to a well-defined scale, such as “critical”, “major”, “minor”
Cost: cost to fix plus cost of lost potential business

15. Faults Focus on the internals of system Only the developer can see it
Location
which product(identifier and version) or part of the product contains the fault.
Spec, code, database, manuals, plan and procedures, report, standard/policy
Requirements, functional, preliminary design, detailed design, product design, interface, database, implementation.

16. Faults Timing - when the fault is created, detected, corrected.
Symptom - what is observed during diagnosis (Table 5.2, page 166)
End result - actual failure caused by fault, should be cross-referenced to failure report.
Cause
human error that led to the fault
Communication, conceptual, clerical

17. Faults Severity - impact on the user
Cost - total cost to system provider
Count - number of faults found in a product or subsystem or during a given period of operation.

18. Fault Report Location: module or document name
Timing: phases of development during which fault was created, detected, and corrected
Symptom: type of error message reported or activity which revealed fault
End result: failure cause by the fault
Mechanism: how cause was created, detected, corrected
Cause: type of human error that led to fault
Severity: refer to severity of resulting or potential failure
Cost: time or effort to locate and correct

19. Changes
Once a failure is experienced and its cause determined, the problem is fixed through one or more changes
Problem is fixed through 1 or more change to any or all of the development product
Changes report (Fig 5.5) - report the changes and track the most affected products

20. Changes Cause of change may be
Corrective - correcting a fault
Adaptive - system changes in some way so the product have to be upgraded
Preventive - find faults before they become failure
Perfective - redo something to clarify the system structure
Count - number of changes made in a given time or given system component.

21. Change Report Location: identifier of document or module changed
Timing: when change was made
Symptom: type of change
End result: success of change, as evidence by regression or other testing
Mechanism: how and whom change was performed
Cause: corrective, adaptive, preventive or perfective
Severity: impact on rest of the system
Cost: time and effort for change implementation and test

22. How to collect data Requires human observation and reporting
Manual data collection – bias, error, omission and delay -> uniform data collection form
Automatic data capture – desirable, essential – recording the execution time
To ensure data are accurate and complete, planning is essential.

23. How to collect data Planning involves:
Decide what to measure based on GQM analysis.
Determine the level of granularity (individual modules, subsystem, function etc)
Ensure that product is under configuration control (which version)
Form design

24. How to collect data Planning involves (continue)
Establish procedures for handling the forms, analyzing the data and reporting the results, setting a central collection point

25. How to collect data Keep procedure simple Avoid unnecessary recording
Train staff in recording data and procedure
Provide results to original providers promptly
Validate all data collected at a central collection point

26. Data collection forms Encourages collecting good, useful data
Self-explanatory
Should allow fixed-format data and free-format comments
Table 5.3 (Data collection forms for software reliability evaluation)

27. When to collect data
Data collection planning when project planning begins
Actual data collection takes place during many phases of development.
Can be collected at the beginning of the project to establish initial values and collected again later to reflect activities and resources being studied.

28. When to collect data
Data-collection activities should become part of the regular development process.
Should be mapped to the process model. (eg: Fig 5.9)

29. When to collect data Eg:
Data relating to project personnel (qualifications or experience) can be collected at the start of the project.
While other data collection, such as effort, begins at project start and continues through operation and maintenance
Count of the number of specification and design faults can be collected as inspections are performed
Data about changes made to enhance the product can be collected as enhancements are performed

30. How to store and extract data
Raw data should be stored in database set up using DBMS.
Can define data structure, insert, modify, delete and extract refined data.
Formats, ranges, valid values etc can be check automatically as they are input.

31. How to store and extract data
Raw database structure
Eg: Figure 5.10 (data structure that supports reliability measurement)
box - table, arrow denotes a many-to-one mapping, double arrow