What is Computer Science? Three paradigms (CACM 1/89) Theory (math): definitions, theorems, proofs, interpretations Abstraction (science): hypothesize, model, experiment, analyze data, results Design (engineering): state requirements, design, implement, test
How to choose a method, tool, technique? Common sense Consultants Theoretical proof Benchmark Case study Survey Experimentation in the Small Experimentation in the Large?
Experiment Design – Define/Describe Subjects Independent Variables and Levels Dependent Variables Confounding Variables (noise) and how controlled Group Selection Treatment vs Measurement
Analyzing Results Descriptive statistics averages ranges standard deviations correlations Inferential statistics t-test ANOVA
Correlations from -1 to +1 “explains” variance should include statistical significance NOT causality see examples …
Useful Experimentation in the Small The Magic Number Seven, Plus or Minus Two; Miller 1956 Exploratory Experimental Studies Comparing Online and Offline Programming Performance; Sackman 1968 A Controlled Experiment in Program Testing and Code Walkthroughs / Inspections; Myers 1978 Reliability of Function Points Measurement – A Field Experiment; Kemerer 1993
To read a paper with empirical contributions ask the following: What was the author’s primary interest? How rigorous is the work? Internal validity External validity What was the STRENGTH of the statistical results reported? What serendipitous results were reported?
Lederer and Prasad Article Discuss methodology questionnaire sent to 400 computing managers and professionals 115 responded responses analyzed for correlations between approaches used and success Careful of cause and effect see footnote 2, page 53
Some interesting guidelines: assign initial estimating task to the developers delay finalizing initial estimate until after thorough study monitor progress (and use external auditors) use the estimate to evaluate personnel don’t rely on cost estimating software!