Effective Testing and Debugging Methods and Its Supporting System with Program Deltas Makoto Matsushita, Masayoshi Teraguchi, and Katsuro Inoue Osaka University

ISFST2000 at Kanazawa Nov/02/2000 Page 2 Software Maintenance Software maintenance procedures –Code modification to apply changes of specs –Regression testing and debugging if any bugs Supporting tools for software maintenance Version management system (VMS) Regression testing tool

ISFST2000 at Kanazawa Nov/02/2000 Page 3 Debugging with Program Deltas It is hard to debug features which are not changed during maintenance Debugging with a code differences (deltas) Using test results of each version The deltas may contain some bugs

ISFST2000 at Kanazawa Nov/02/2000 Page 4 Ness and Ngo: Regression Containment Using VMS and testing tool Correct output (○) and wrong output (×) doesn’t appear by turns Only a single “wrong” case Only removes codes; no debugging

ISFST2000 at Kanazawa Nov/02/2000 Page 5 Zeller: Delta Debugging Using test tool (no VMS is required) Assumes “core” ( △ ) output Can’t consider development sequences Only two “wrong” cases ( △ and ×) Only analysis; no debugging

ISFST2000 at Kanazawa Nov/02/2000 Page 6 Our Research Objectives An improved method with program deltas –Consider more “wrong” cases –Support actual debugging activities Debugging methods “DMET”, and its supporting environment “DSUS”

ISFST2000 at Kanazawa Nov/02/2000 Page 7 Today’s topic Our debugging method “DMET” “DSUS” supporting environment DMET evaluation with DSUS prototype Conclusion

ISFST2000 at Kanazawa Nov/02/2000 Page 8 An approach of DMET DMET is composed of three phases: –Detection: a program delta (may contains some bugs) to be detected –Indication: show a delta with latest code –Reflection: apply modification to old versions

ISFST2000 at Kanazawa Nov/02/2000 Page 9 DMET: Detection Phase Using VMS and testing tools Assumes wrong output ( － ) which is different from the one of latest version Detect the last correct output and the oldest wrong output

ISFST2000 at Kanazawa Nov/02/2000 Page 10 DMET: Indication Phase Program deltas are arranged as “removed” and “added” program codes Detected deltas (maybe in older version) are mapped to the latest version

ISFST2000 at Kanazawa Nov/02/2000 Page 11 DMET: Reflection Phase Modification to the latest version are also applied to older versions –if application causes some errors, remove error version from test sequence –new sequence are used for further testing

ISFST2000 at Kanazawa Nov/02/2000 Page 12 Supporting System DSUS A GUI-based SDE for debugging –Program editors –Compile, Execute a program Cooperates other tools –RCS for version management system –DejaGnu for testing tools Auto-execution support –check-in/check-out, regression testing –three DMET procedures

ISFST2000 at Kanazawa Nov/02/2000 Page 13 System Configuration GUI: user interaction DSUS main : integration of other components RCS: wrapper for RCS commands DejaGnu: wrapper for DejaGnu commands

ISFST2000 at Kanazawa Nov/02/2000 Page 14 DSUS Screenshot

ISFST2000 at Kanazawa Nov/02/2000 Page 15 DSUS Evaluation Two experimentations –Experimentation A: correcting programs which contain bugs in unchanged potion of program –Experimentation B: debug programs corrected by experimentation A Preparation –Program specification and its program –Requirements for the new program –Test data

ISFST2000 at Kanazawa Nov/02/2000 Page 16 Experimentation Procedures Experimentation A –Apply new requirement to a (prepared) program –Correct programs which has a bug in unchanged portion of the program Experimentation B –Debug program collected in exp. A –Subjects are divided into two groups debugging without DMET debugging with DMET –Compare Elapsed time of debugging

ISFST2000 at Kanazawa Nov/02/2000 Page 17 Results of experiment A Three programs are collected DMET detects bugs in two programs; however, detected deltas of program Z doesn’t contain a bug Program# of vers# of testdeltasdetected ? X yes Y yes Z no

ISFST2000 at Kanazawa Nov/02/2000 Page 18 Results of experiment B Debugging time is reduced in X and Y, which bug is detected correctly. XYZ w/o DMET w/ DMET w/ DMET w/o DMET Program elapsed time (min)

ISFST2000 at Kanazawa Nov/02/2000 Page 19 Results of experiment B (cont.) According to Welch’s test (5 percents differences), a significant difference of X+Y between both groups is detected elapsed timeX+Y+ZX+Y w/o DMET w/ DMET DMET support effective debugging if the detection is correct

ISFST2000 at Kanazawa Nov/02/2000 Page 20 Discussion In program Z, elapsed time of w/ DMET takes longer time than w/o DMET –A bug in program Z is the case that DMET doesn’t consider; it is caused since some modification should be needed –DMET generates wrong deltas –Testees are confused with deltas

ISFST2000 at Kanazawa Nov/02/2000 Page 21 Discussion (cont.) However, we may estimate that the deltas truly contains a bug since it’s doubtful that: –the distance of deltas is longer, compared with total numbers of version –the delta contains the “base version (1.1)” program# of vers# of testdeltasdetected ? Z no

ISFST2000 at Kanazawa Nov/02/2000 Page 22 Conclusion DMET contributes to reduce debugging costs of the engineers, if the detected program delta contains bugs DMET doesn’t consider all cases, it may generates wrong delta –If wrong delta, debugging takes longer time –However, it can be estimated

ISFST2000 at Kanazawa Nov/02/2000 Page 23 Further Research Topics Some estimation methods for checking DMET deltas If the delta seems wrong, DSUS shows to the engineers that it maybe wrong. DMET extension for new programs Current DMET assumes that “base version” exists and it is correct. New program have no “base version”.