1. Empirical Results Copyright © Software Carpentry 2011 This work is licensed under the Creative Commons Attribution License See http://software-carpentry.org/license.html for more information. Software Engineering

2. Empirical Results Seven Years War

3. Software EngineeringEmpirical Results Seven Years War Actually nine years long (1754-63)

4. Software EngineeringEmpirical Results Seven Years War Actually nine years long (1754-63) Britain lost 1,512 sailors to enemy action

5. Software EngineeringEmpirical Results Seven Years War Actually nine years long (1754-63) Britain lost 1,512 sailors to enemy action And almost 100,000 to scurvy

6. Software EngineeringEmpirical Results Seven Years War Actually nine years long (1754-63) Britain lost 1,512 sailors to enemy action And almost 100,000 to scurvy Unnecessarily

7. Software EngineeringEmpirical Results James Lind (1716-94) 1747: first controlled medical experiment in history

8. Software EngineeringEmpirical Results James Lind (1716-94) 1747: first controlled medical experiment in history cider sea water vitriol oranges vinegar barley water

9. Software EngineeringEmpirical Results James Lind (1716-94) 1747: first controlled medical experiment in history cider sea water vitriol oranges vinegar barley water

10. Software EngineeringEmpirical Results James Lind (1716-94) 1747: first controlled medical experiment in history cider sea water vitriol oranges vinegar barley water Allowed British ships to be effective on long patrols during the Napoleonic Wars

11. Software EngineeringEmpirical Results 1950: Hill & Doll study comparing smokers and non-smokers

12. Software EngineeringEmpirical Results 1950: Hill & Doll study comparing smokers and non-smokers 1.Smoking causes lung cancer

13. Software EngineeringEmpirical Results 1950: Hill & Doll study comparing smokers and non-smokers 1.Smoking causes lung cancer 2. Many people would rather fail than change

14. Software EngineeringEmpirical Results 1950: Hill & Doll study comparing smokers and non-smokers 1.Smoking causes lung cancer 2. Many people would rather fail than change +

15. Software EngineeringEmpirical Results 1950: Hill & Doll study comparing smokers and non-smokers 1.Smoking causes lung cancer 2. Many people would rather fail than change +

16. Software EngineeringEmpirical Results A rarity in software engineering until the mid 1990s

17. Software EngineeringEmpirical Results But today, papers describing new tools or working practices routinely include results from empirical studies A rarity in software engineering until the mid 1990s

18. Software EngineeringEmpirical Results But today, papers describing new tools or working practices routinely include results from empirical studies Particularly ones by young researchers A rarity in software engineering until the mid 1990s

19. Software EngineeringEmpirical Results But today, papers describing new tools or working practices routinely include results from empirical studies Particularly ones by young researchers A rarity in software engineering until the mid 1990s Many are flawed or incomplete, but standards are constantly improving

20. Software EngineeringEmpirical Results

21. Software EngineeringEmpirical Results

22. Software EngineeringEmpirical Results

23. Software EngineeringEmpirical Results

24. Software EngineeringEmpirical Results

25. Software EngineeringEmpirical Results

26. Software EngineeringEmpirical Results Unsurprising in retrospect

27. Software EngineeringEmpirical Results Unsurprising in retrospect Actionable

28. Software EngineeringEmpirical Results What goes wrong in developers' first job?

29. Software EngineeringEmpirical Results What goes wrong in developers' first job?

30. Software EngineeringEmpirical Results What goes wrong in developers' first job?

31. Software EngineeringEmpirical Results Also unsurprising in retrospect, and actionable What goes wrong in developers' first job?

32. Software EngineeringEmpirical Results Statistics is just one path

33. Software EngineeringEmpirical Results Statistics is just one path

34. Software EngineeringEmpirical Results Statistics is just one path

35. Software EngineeringEmpirical Results Statistics is just one path Controlled experiments are expensive…

36. Software EngineeringEmpirical Results Statistics is just one path Controlled experiments are expensive… …and often eliminate exactly what we want to study

37. Software EngineeringEmpirical Results Statistics is just one path Controlled experiments are expensive… …and often eliminate exactly what we want to study Biggest hurdle is re-education

38. Software EngineeringEmpirical Results Test-Driven Development

39. Software EngineeringEmpirical Results Test-Driven Development An article of faith among many programmers

40. Software EngineeringEmpirical Results Test-Driven Development An article of faith among many programmers Meta-analysis of over 30 studies No consistent effect

41. Software EngineeringEmpirical Results Test-Driven Development An article of faith among many programmers Meta-analysis of over 30 studies No consistent effect Some positive

42. Software EngineeringEmpirical Results Test-Driven Development An article of faith among many programmers Meta-analysis of over 30 studies No consistent effect Some positive Some negative

43. Software EngineeringEmpirical Results Test-Driven Development An article of faith among many programmers Meta-analysis of over 30 studies No consistent effect Some positive Some negative Some inconclusive

44. Software EngineeringEmpirical Results Test-Driven Development An article of faith among many programmers Meta-analysis of over 30 studies No consistent effect Some positive Some negative Some inconclusive The better the study, the weaker the signal

45. Software EngineeringEmpirical Results Boehm et al (1975): "Some Experience with Automated Aids to the Design of Large-Scale Reliable Software"

46. Software EngineeringEmpirical Results Boehm et al (1975): "Some Experience with Automated Aids to the Design of Large-Scale Reliable Software" …and many more since

47. Software EngineeringEmpirical Results Boehm et al (1975): "Some Experience with Automated Aids to the Design of Large-Scale Reliable Software" …and many more since time number / cost Most errors introduced during requirements analysis and design

48. Software EngineeringEmpirical Results Boehm et al (1975): "Some Experience with Automated Aids to the Design of Large-Scale Reliable Software" …and many more since time number / cost Most errors introduced during requirements analysis and design The later they are removed, the more expensive they are

49. Software EngineeringEmpirical Results Pessimists Optimists

50. Software EngineeringEmpirical Results If we tackle the hump in the error injection curve, fewer bugs will get to the expensive part of the fixing curve. Pessimists Optimists

51. Software EngineeringEmpirical Results If we tackle the hump in the error injection curve, fewer bugs will get to the expensive part of the fixing curve. If we do shorter iterations, the total cost of fixing bugs will go down. Pessimists Optimists

52. Software EngineeringEmpirical Results If we tackle the hump in the error injection curve, fewer bugs will get to the expensive part of the fixing curve. If we do shorter iterations, the total cost of fixing bugs will go down. Pessimists Optimists

53. Software EngineeringEmpirical Results Fagan 1975: "Design and Code Inspections to Reduce Errors in Program Development"

54. Software EngineeringEmpirical Results Fagan 1975: "Design and Code Inspections to Reduce Errors in Program Development" Reading code carefully is the most cost effective way to find bugs

55. Software EngineeringEmpirical Results Fagan 1975: "Design and Code Inspections to Reduce Errors in Program Development" Reading code carefully is the most cost effective way to find bugs Cohen 2006: most of the value comes from the first hour and the first pair of eyes

56. Software EngineeringEmpirical Results Fagan 1975: "Design and Code Inspections to Reduce Errors in Program Development" Reading code carefully is the most cost effective way to find bugs Cohen 2006: most of the value comes from the first hour and the first pair of eyes Code review now normal in open source

57. Software EngineeringEmpirical Results

58. Software EngineeringEmpirical Results Do programming languages affect productivity?

59. Software EngineeringEmpirical Results Do programming languages affect productivity? Does using design patterns make for better code?

60. Software EngineeringEmpirical Results Do programming languages affect productivity? Does using design patterns make for better code? Can we predict software faults statistically?

61. Software EngineeringEmpirical Results Is up-front design cost-effective? Do programming languages affect productivity? Does using design patterns make for better code? Can we predict software faults statistically?

62. Software EngineeringEmpirical Results Is up-front design cost-effective? Why is it hard to learn how to program? Do programming languages affect productivity? Does using design patterns make for better code? Can we predict software faults statistically?

63. Software EngineeringEmpirical Results Is up-front architecture cost-effective? Why is it hard to learn how to program? Is open source software actually better? Do programming languages affect productivity? Does using design patterns make for better code? Can we predict software faults statistically?

64. Software EngineeringEmpirical Results Is up-front architecture cost-effective? Why is it hard to learn how to program? Is open source software actually better? Are some programmers 10X better? Do programming languages affect productivity? Does using design patterns make for better code? Can we predict software faults statistically?

65. February 2011 Copyright © Software Carpentry 2011 This work is licensed under the Creative Commons Attribution License See http://software-carpentry.org/license.html for more information. narrated by Greg Wilson