ISERN-Meeting, Honolulu, Hawaii 09 October 2000 Slide 0 Using Experiments to Teach Software Engineering Using Experiments to Teach Software Engineering Dieter Rombach Fraunhofer Institute for Experimental Software Engineering (IESE) Kaiserslautern, Germany ( University of Kaiserslautern Dept. of Computer Science Software Engineering Group Kaiserslautern, Germany wwwagse.informatik.uni-kl.de (wwwagse.informatik.uni-kl.de)

ISERN-Meeting, Honolulu, Hawaii 09 October 2000 Slide 1 Contents Challenges (teaching software engineering) Idea(s) Software Engineering Course Design Example Experiments Benefits Lessons Learned

ISERN-Meeting, Honolulu, Hawaii 09 October 2000 Slide 2 Challenges (teaching software engineering) Software development is an engineering activity; benefits of engineering principles need to be experienced actively! Most students in the past did not see relevance of engineering principles to their practical work; after course exam they went back to development business as usual! Students with prior work experience appreciated software engineering principles better

ISERN-Meeting, Honolulu, Hawaii 09 October 2000 Slide 3 Idea(s) Include empirical studies in regular software engineering courses (since 1997) - partial replications of research exp’s! Focus on basic principles such as – information hiding (to improve modifiability) – tractable documentation (to improve modifiability) – reading of programs (to improve effectiveness of defect finding as compared to testing) – reading of hl documents (to improve early defect detection) perform empirical (self experience) studies after initial teaching of techniques/methods - max. 3 x 3h sessions per exp! self-evaluate data and derive ‘good practice’ guidelines

ISERN-Meeting, Honolulu, Hawaii 09 October 2000 Slide 4 Software Engineering Course Design Two software engineering courses – Course SE 1: Systematic software development (& project) – Course SE 2: Software project planning & management (&..) SE 1 – overview – foundations of se – project planning & management – software application development – software system development – software component development – software maintenance & evolution Experiment 1 Experiment 2 Experiment 3

ISERN-Meeting, Honolulu, Hawaii 09 October 2000 Slide 5 Example Experiments Experiment-1: Effectiveness/Efficiency of different requirements reading techniques – goal: to understand that defects can be detected effectively/efficiently, even in informal requirements documents – techniques: ad-hoc, check-lists, pbr – design: each student applies ad-hoc & (check-list or pbr) on 2 different requirements documents (from building automation domain) - about 50 pages each – data collected: number/type of defects found, effort spent – results: more (at least equal # of) defects found (effectiveness increases; efficiency increases) - cap of 3h???

ISERN-Meeting, Honolulu, Hawaii 09 October 2000 Slide 6 Example Experiments Experiment-2: Understandability/Modifiability of differently structured/documented OO-Systems (guidelines!) – goal: to understand the impact of ‘good’ documentation and structuring guidelines on understandability/modification – techniques: vertically/horizontally consistent documentation vs … & high degree of information hiding/low coupling/… vs... – design: half the students use ‘good’, half ‘bad’ system (= requirements, design, code)  step 1: answer questions about system  step 2: perform changes – data collected: # correctly answered questions; # correctly made changes; effort involved – results: more questions answered/changes made correctly (better understandability/modifiability, efficiency?)

ISERN-Meeting, Honolulu, Hawaii 09 October 2000 Slide 7 Example Experiments Experiment-3: Effectiveness/efficiency of different code defect detection techniques – goal: to understand that defects can be found more effectively/efficiently via code reading than testing – techniques: PBR code reading versus structural/functional testing – design: fractional factorial design (see: Selby); each student uses PBR and one testing technique – data collected: number/types of defects found, effort spent – results: more (at least equal # of) defects found (code reading is always more (or at least as) ‘effective’ than (as) other techniques; same is true for ‘efficiency’)

ISERN-Meeting, Honolulu, Hawaii 09 October 2000 Slide 8 Benefits Students UNDERSTAND software engineering principles (‘they derived them themselves!’) Students (in their majority) own principles and continue to apply them naturally Industry appreciates knowledge about key se principles Students appreciate empirical studies as a means of learning Students live ‘empirically-based’ improvement in subsequent project courses (Optimization of PBR across courses leads to high defect detection effectiveness - >80%) It is fun for the teacher (No more ‘trust me, this is important’)

ISERN-Meeting, Honolulu, Hawaii 09 October 2000 Slide 9 Lessons Learned Participation should be voluntary (typically > 80%, i.e graduate students) Certificate of participation appreciated by students (& industry) Students expect experiments (!) Personal data must be owned by individuals Interpretation should be done (on aggregate level) in class feedback sessions (later complemented by other experimental results!) Materials should be packaged in reusable form (lab packages require lots of effort!) Learning effect must over-write research interests (or we pay!) Participation in publication of results could be offered ISERN could host database of pre-packaged experiments