27th International Symposium on Software Reliability Engineering ISSRE’2016 Error Abstraction Accuracy and Fixation during Error-based Requirements Inspections Supported by National Science Foundation Awards 1423279 and 1421006 Authors: Vaibhav Anu, Gursimran Walia (North Dakota State University) Wenhua Hu, Jeffrey C. Carver (University of Alabama) Gary Bradshaw (Mississippi State University)

Human Errors Human Errors Slips Lapses Mistakes Failure in the execution of plan Failure in the plan of action EXECUTION EXECUTION PLANNING e.g., lack of attention. e.g., inadequate planning, mostly from a lack of knowledge. e.g., forgetting one or more steps in a process. 2

Proposed APPROACH: Error ABSTRACTION AND INSPECTION (EAI) Error Abstraction (EA) Process What caused that fault? List of Human Errors List of New Faults (found during human error based reinspection) Reinspection (human error based) Requirements Document (SRS) List of Faults (found during FC Inspection) Inspection (FC based)

THE Human Error taxonomy (HET) Requirements Phase Human Errors STEP 1: Identifying Requirements Phase Human Errors from Software Engineering Literature STEP 2: Human Error Classification System Identified from Cognitive Psychology Literature Clerical Errors SLIPS LAPSES MISTAKES Lack of consistency in Requirement Specifications Loss of information from stakeholders Accidentally overlooking requirements Application errors Environment errors Information Management errors Wrong assumptions Low understanding of each other’s roles Mistaken belief that it is impossible to specify non-functional requirements in a verifiable form Not having a clear demarcation between client and users Lack of awareness of sources of requirements Problem-Solution errors Inadequate Requirements Process Syntactic errors

Purpose of current RESEARCH The current study examines the impact of Error Abstraction (EA) accuracy on inspectors’ fault detection effectiveness (during human error based reinspection). Another aspect of this research evaluates the “fixation tendencies” of inspectors during EA.

Empirical study design Training: Inspecting SRS using HET PRE-TEST Error Report Form Re-inspect SRS for remaining faults New Fault List 3 1 Reflection 2 Abstract and Classify Errors Inspectors POST -TEST 3 1 2 New Fault Form Fault Form Error Report Form Dr. Walia, I do not have the original experiment design figure. The one I have is monochromatic. Can you please replace this with the original. Inspect SRS using Errors Fault Checklist Error Abstraction Inspectors Post Study Questionnaire and Discussion with Subjects

Results of data analysis RQ1: Can an inspector’s EA accuracy predict their fault detection effectiveness during an error-based requirements inspection? Result: For both pre-test and post-test, it was found that EA accuracy was strongly and significantly correlated with the number of additional faults found using abstracted human errors. Implications for project managers employing error based inspections: it is important to ensure that inspectors are able to correctly perform the EA step, which in turn would help them detect more faults and ensure software quality. Pre-test (PGCS): EA Accuracy vs Fault Detection Effectiveness (r = 0.79, p = 0.0003) Post-test (RIM): EA Accuracy vs Fault Detection Effectiveness (r = 0.61, p = 0.0121)

Results of data analysis RQ2: Is there a potential for fixation bias during EA? Results: Inspectors tend to fixate on certain human error classes. More formal training on human error identification and improved error abstraction guidelines are required to help inspectors avoid fixation. Implications: Avoiding fixation is likely to improve EA accuracy, which consequently will improve fault detection effectiveness during error based inspections (as EA accuracy is strongly correlated with fault detection effectiveness). Effect of EA Fixation Clerical Errors Application Errors Lack of Consistency Errors Reported number of errors in RIM SRS (by all 16 subjects) 174 58 52 # of errors that were correctly abstracted (out of the reported number) 44 4 2 Top 3 most reported human errors classes (of HET) by subjects: Clerical errors (Slip) Application errors (Mistake) lack of consistency (Slip) Severe negative impact on EA accuracy

conclusion Results showed that EAI and HET can help provide significant gains in fault detection effectiveness Subjects provided suggestions for improving both, the error abstraction process, and the re-inspection (using abstracted errors) process More empirical studies under academic and industrial settings are required to generalize the results (particularly the results on human error insights).

Ongoing - Human ERROR ABSTRACTION ASSIST (HEAA)

27th International Symposium on Software Reliability Engineering ISSRE’2016 Error Abstraction Accuracy and Fixation during Error-based Requirements Inspections Supported by National Science Foundation Awards 1423279 and 1421006 Authors: Vaibhav Anu, Gursimran Walia (North Dakota State University) Wenhua Hu, Jeffrey C. Carver (University of Alabama) Gary Bradshaw (Mississippi State University)