*Graduate School of Engineering Science, Osaka University An Experimental Comparison of Checklist-Based Reading and Perspective-Based Reading for UML Design Document Inspection Giedre Sabaliauskaite* Fumikazu Matsukawa** Shinji Kusumoto** Katsuro Inoue** *Graduate School of Engineering Science, Osaka University **Graduate School of Information Science and Technology, Osaka University

Contents Background Research Goals Experiment description Reading techniques Experimental design Data analysis Results Conclusions and Further Work

Reading techniques are used to guide inspectors Software Inspection Software inspection is an effective and efficient method to detect defects in early stages of software development life-cycle Software inspection usually consists of several activities [1] Planning Key activity – defects are detected Reading techniques are used to guide inspectors Defect detection Defect collection Defect correction [1] O. Laitenberger, J.M. DeBaud, “An encompassing life cycle centric survey of software inspection”, The Journal of Systems and Software, vol. 50, no. 1, 2000, pp. 5-31.

Reading techniques Ad hoc and Checklist-based Reading are the most popular [1] Ad hoc: no guidance for inspectors Checklist-based Reading (CBR): Checklist Scenario-based Reading is a more recent approach: Scenario Perspective-based Reading (PBR) [2] is one of Scenario-based approaches Software product is inspected from different perspectives (designer, tester, etc.) Provides inspectors with different Scenario for each perspective We analyzed CBR and PBR inspection techniques during experiment [1] O. Laitenberger, J.M. DeBaud, “An encompassing life cycle centric survey of software inspection”, The Journal of Systems and Software, vol. 50, no. 1, 2000, pp. 5-31. [2] V.R. Basili, S. Green, O. Laitenberger, F. Lanubile, F. Shull, S. Sorumgard, M.V. Zelkowitz, “The Empirical Investigation of Perspective-Based Reading”, Empirical Software Engineering: An International Journal, vol.1 , no. 2, 1996, pp. 133-164.

Related research Several works are done in the area of Unified Modelling Language (UML) diagram inspection One of the works is described in [1] Experimental comparison of CBR and PBR 18 subjects (practitioners) and 2 software systems UML diagrams inspected – Class, State, Sequence, Collaboration Results - PBR 3-person teams exhibited higher effectiveness and lower cost per defect than CBR teams Authors of different studies came to the conclusion that UML inspections need to be further investigated [1] O. Laitenberger, C. Atkinson, M. Schlich, K. El Emam. An experimental comparison of reading techniques for defect detection in UML design documents. The Journal of Systems and Software, 53: 183-204, 2000.

Research Goals Conduct UML diagram inspection experiment in Osaka university Language used during experiment - Japanese Compare Reading techniques CBR and PBR with respect to Defect detection effectiveness Cost per defect Time spent on inspection Compare both individual inspector results and team results Acquire experience for future inspection The goals of the research are: To conduct an experiment in Osaka university to inspect design documents written in the notation of Unified Modelling Language. To inspect the following UML diagrams: Use-case Class Activity Sequence To compare To analyze To acquire

Structure of CBR and PBR techniques, used during experiment Diagram-specialized Checklist developed Included 20 questions PBR Three perspectives: User, Designer, Implementer Three scenarios (one for each perspective) developed Scenarios consisted of several steps, each of them included Explanations Tasks to perform Questions to answer

Assignment of UML diagrams All inspectors were given system requirement description and Use-Case diagrams Defects were injected into Activity, Class, Sequence and Component diagrams Document Seminar system Hospital system CBR PBR perspective User Designer Implementer System requirement description 1  Use-Case diagrams Activity diagrams 8 7 Class diagrams Sequence diagrams 12 Component diagrams

Experiment design Subjects Objects 59 third year students of the Software Development course, Osaka University Objects Seminar information system ( 24 pages ) Hospital information system ( 18 pages ) PBR CBR User Designer Implementer Seminar system 7 6 11 Hospital system 10

Defects Three types of defects inserted into UML diagrams Syntactic – missing or unnecessary information Semantic – misrepresentation of a concept, or unclear representation Consistency – inconsistency between UML diagrams In total, 15 defects inserted into the each software system 4 into Activity diagrams 3 into Class diagrams 5 into Sequence diagrams 3 into Component diagrams

Experimental Hypotheses H01: subjects who use PBR technique should spend less time on inspection H02: subjects who use PBR should have higher cost per defect H03: individual defect detection effectiveness should be different between CBR and PBR inspectors H04: team defect detection effectiveness should be different between CBR and PBR inspector teams

Data analysis Two types of analysis Individual data analysis Defect detection effectiveness Time spent on inspection Cost per defect Team data analysis Subjects assigned into 3-person virtual teams Effectiveness of CBR and PBR teams compared

Percentage of inspectors, who have detected defects relevant to different perspectives Both CBR and PBR exhibited similar results

Defect detection effectiveness (according to defect types) Syntactic defects: CBR more effective Semantic and Consistency defects: PBR more effective

Defect detection effectiveness (according to UML diagram types) Class Diagrams: CBR more effective Activity and Component Diagrams : PBR more effective Sequence Diagrams: CBR and PBR exhibited similar effectiveness

Time spent on inspection Time spent on inspection, Cost per defect and Defect detection effectiveness Variables Reading technique Mean SD Time spent on inspection CBR 62.9 11.7 PBR 51.3 15.1 Cost per defect 6.2 1.6 10.2 3.5 Effectiveness 70.2 11.5 69.1 15.3 PBR: 18% (11 min) less time spent on inspection CBR: 39% (4 min / defect) lower cost per defect CBR Effectiveness ≈PBR Effectiveness Independent samples t-test Mann-Whitney test Significant difference in Inspection Time and Cost No significant difference in Effectiveness

Virtual teams Subject assignment into 3-person virtual teams Any three CBR reviewers are included into team One reviewer using each of the three PBR perspectives included into team C9 C8 C7 C6 C5 C4 C3 C2 C1 CBR group I5 D5 U5 I4 D4 U4 I3 D3 U3 I6 D6 U6 I2 D2 U2 I1 D1 U1 PBR group Grouping virtual teams into statistically independent groups CBR groups: 3 teams included (11 inspectors for Seminar system, 10 subjects for Hospital system) PBR groups: 6 teams included (7 Users, 6 Designers, 6 Implementers for each systems)

Comparisons between CBR and PBR teams Hospital system Seminar system Number of comparisons: Comparisons C1 C2 C3 C4 C5 C6 C7 C8 C9 U1 D1 I1 U2 D2 I2 … U6 D6 I6 CBR group 1 PBR group 1 C1 C2 C3 C4 C5 C6 C7 C8 C10 U1 D1 I6 U2 D2 I1 … U6 D6 I5 CBR group 2 … … PBR group 2 …

Team defects detection effectiveness Team data analysis (1/2) CBR and PBR teams were compared with respect to Team defect detection effectiveness Number or comparisons, in which either CBR or PBR was more effective, or both techniques showed similar effectiveness Software system Team defects detection effectiveness CBR>PBR PBR>CBR CBR=PBR Seminar 54 753 326 688 585 743 712 544 449 600 Hospital 10 160 482 176 8 064 149 760 CBR teams more effective than PBR teams

Defect detection effectiveness Team data analysis (2/2) T-test with significance level of 2.5% was used to evaluate in which comparisons there was a significant difference between CBR and PBR team effectiveness Two types of comparisons made All defects included Syntactic defects omitted Comparison type Software system Defect detection effectiveness CBR > PBR PBR > CBR CBR = PBR All defects included Seminar 7631557968 Hospital 7233873848 Syntactic defect omitted 6968160000 85122000 CBR teams more effective than PBR teams

Experiment results Individual inspector results Inspectors who use PBR spend on average 18% less time on inspection Cost per defect of Inspectors who use PBR is on average 39% higher There is no statistically significant difference in defect detection effectiveness between individual CBR and PBR inspectors, however on average PBR is more effective for Semantic and Consistency defects PBR is more effective for Activity and Component diagrams CBR is more effective for Syntactic defects CBR is more effective for Class diagrams Three-person virtual team results CBR teams exhibit higher defect detection effectiveness than PBR teams

Threats to validity Internal validity External validity Selection: experiment was a mandatory part of the course Instrumentation: objects which we used could have influence External validity Students were used as subjects Design documents were similar to those which are used in practice, but the size of systems in industry is usually larger There were threats to internal and external validity, but they were not considered large in this experiment

Conclusions and Further research UML diagram inspection experiment was conducted The results of the experiment indicate that There is no statistically significant difference in effectiveness of CBR and PBR individual inspectors CBR virtual teams are more effective than PBR virtual teams Future research will be directed to further investigation of UML diagram inspection A replication of experiment was conducted in July 2002, during which real team meeting were performed More detail analysis of both individual and team data Research of Fault Content Estimation methods

Software Engineering Laboratory http://iip-lab.ics.es.osaka-u.ac.jp/

ADDITIONAL SLIDES: 26-32

Experiment Variables and Hypotheses Independent variables Reading technique (CBR and PBR) Dependent variables Number of defects found Time spent on inspection Defect detection effectiveness Cost per defect Null Hypotheses For individual inspectors H01: PBR Time > CBR Time H02: PBR Cost < CBR Cost H03: PBR Effectiveness = CBR Effectiveness For 3-person virtual teams H04: PBR Effectiveness = CBR Effectiveness

Information systems inspected Seminar information system System supports the process of a company which organizes seminars Includes activities from seminar planning to issuing the qualification certificate Designs relationships among personnel of seminar company, participants, lecturers, etc. Hospital information system System supports the process of a hospital Includes activities such as patient registration, medical examination, treatment, prescription of the medicines, etc. Designs the relationships among the personnel of a hospital, patients, pharmacists, etc.

Collected data Two types of data collected during experiment Defect data Time data Number of inspectors Defects Inspection time (min) Number of detectable defects Average number of defects detected max/min Average Seminar information system PBR User 7 4.43 6 / 3 60.43 90 / 46 PBR Designer 6 5.00 6 / 4 65.50 80 / 51 PBR Implementer 9 6.50 9 / 5 76.67 95 / 40 CBR 11 15 10.55 13 / 8 74.64 90 / 62 Hospital information system 48.29 70 / 25 3.83 5 / 3 59.17 73 / 30 6.33 7 / 5 63.30 77 / 44 10 10.50 12 / 8 70.10 94 / 60

CBR Checklist CHECKLIST Locate the following diagrams: Class Diagrams, Activity Diagrams, Sequence Diagrams and Component Diagrams. Answer the questions related to corresponding diagrams. When you detect a defect, mark it on the diagram and fill the necessary information in the Defect registration form. If you have any comments write them in Comment form. … SEQUENCE DIAGRAMS 10. Aren’t there any inconsistencies among Sequence Diagram, Class Diagram and Requirements Specification?  yes  no 11. Are all the necessary Objects and Messages defined? 12. Is every Class from Class Diagram included in any of Sequence Diagrams and vice versa? 13. Is every Use Case from Use-Case Diagram implemented in at least one of Sequence Diagrams? 14. Are there no redundant elements (Objects of Messages) in Sequence Diagram?

PBR Scenario DESIGNER SCENARIO Assume that you are the Designer of the system. The concern of the designer is to ensure that the designer’s needs are completely satisfied in the following documents: Requirement specification, Class Diagram and Sequence Diagram. During the inspection process you will need to inspect those documents in order to detect defects in Class and Sequence Diagrams from designer’s point of view. Please perform tasks following Step1 through Step3. For each step you must locate corresponding documents, follow the instructions and answer the given questions. When you detect a defect, mark it on the diagram and fill the necessary information in the Defect registration form. If you have some comments write them in Comment form. Step 3 Locate the Class Diagram and Sequence Diagrams Make a list of all Objects included in Sequence Diagrams and answer the following questions. 3.1. Are all the Objects of Sequence Diagrams defined in Class Diagrams? Are all Classes of Class Diagrams defined in Sequence Diagrams? 3.2. Are all the Messages between objects Sequence Diagram defined as Methods of the corresponding Class in Class Diagram? 3.3. Are there no redundant elements in Sequence diagrams?

Defect registration form

Software development process The main steps of software development process Development of Use-case diagrams Describing system activities in Activity diagrams Defining static structure of the system in Class diagrams Modelling dynamic aspects of the system in Sequence diagrams Detailed description of object states in Statechart diagrams Development of the Component diagrams User’s perspective in our experiment covered the second, and partially the third and the fourth steps Designer’s perspective covered the third and the fourth steps Implementer’s perspective covered the sixth step, and partially the third and the fourth steps