1. Slide 1 Qualitative observations on NTNU’s OORT experiment, ISERN, Hawaii, 8-10 Oct. 2000 NTNU OORT Experiment, March 2000 Some qualitative observations Reidar Conradi Software Engineering Group Dept. of Computer and Information Science (IDI) NTNU

2. Slide 2 Qualitative observations on NTNU’s OORT experiment, ISERN, Hawaii, 8-10 Oct. 2000 Background Repeated OORT experiment taken from CS735 course at UMD, Fall 1999. All artifacts and instructions in English. Part of 4th year QA/SPI course, taught by local responsibles; material adapted by R. Conradi in the USA. Students get pass/no-pass on the assignment. Main change: operationalized OORT instructions as Qij.x questions.

3. Slide 3 Qualitative observations on NTNU’s OORT experiment, ISERN, Hawaii, 8-10 Oct. 2000 Overall impressions Big variation in effort, dedication and results: –E.g. some teams did not report effort data, even did the wrong OORTs. Big variation in UML expertise. Students felt frustrated by the extent of the assignment, and that indicated efforts were too low -- felt cheated. Lengthy and tedious pre-annotation of artifacts, before real defect detection could start. Discovered many defects already during annotation, even defects that remained unreported. OORTs too ”heavy” for the given (small) artifacts? Some confusion about the assigments: what to be done and how, on what artifacts,...?

4. Slide 4 Qualitative observations on NTNU’s OORT experiment, ISERN, Hawaii, 8-10 Oct. 2000 OORT results Found many defects, not previously reported: –Loan Arranger: 30 (13+17) seeded defects & 23 more + 26 comments. –Parking Garage: 32 (21+11) seeded defects & 14 more + 30 comments. Defects actually reported, 4 groups for LA and 5 for PG, average and variance: –LA: 11 (7..14) seeded & 13 (3..27) more + 9 (6..16) comments. –PG: 7 (4..10) seeded & 4 (0..9) more +10 (0..21) comments. Effort spent: –LA: 5-6 hours. –PG: 10-13 hours. Lacking access to background/questionnaire data (delayed). In general: more data analysis to come.

5. Slide 5 Qualitative observations on NTNU’s OORT experiment, ISERN, Hawaii, 8-10 Oct. 2000 OORT comments Some unclear instructions: Executor/Observer role, Norwegian file names, file access, some typos. First read RD? Some unclear concepts: service, constraint, condition, … UML: not familiar by some groups. Technical comments on artifacts and OORTs: –Add comments/rationale to diagrams: UC and CDia are too brief. –CDe hard to navigate in -- add separators. –SqD had method parameters, but CDia not -- how to check? –Need several artifacts (also RD) to understand some OORT questions. –Many trivial checks could have been done by an automatic UML tool. –Many trivial typos and naming defects in the artifacts: Parking Garage artifacts need more work LA vs. Loan Arranger vs. LoanArranger, gate vs. Gate, CardReaders vs. Card_Readers. Fanny May = Loan Arranger? Lot = Parking Garage?