What We Think We Know About Software Development - and Why We Believe it’s True Steve Crouch, SSI with thanks to Greg Wilson Software Carpentry Workshop, Newcastle 14/05/2012

Why? Researchers are expected to just know/find out about programming on demand American car manufacturers – Compete on e.g. cost – But not quality – Enter Volkswagen, BMW, Mercedes! Building things right allows you to build things faster Condition of publication – Data archival of results – What about software that generates the data Image courtesy of M 93

Once Upon a Time… 7 Years War ( ) Britain loses 1512 sailors to enemy action …and almost 100,000 to scurvy

A Twist of Irony… True irony, not Alanis Morrisette irony James Lind ( ) 1747: (possibly) first ever controlled medical experiment CiderSea water Sulphuric acidOranges VinegarBarley water Ignored – not a proper English gentleman! Listened – repeated in 1794 by a… Courtesy of the National Library of Medicine

A Twist of Irony… True irony, not Alanis Morrisette irony James Lind ( ) 1747: (possibly) first ever controlled medical experiment CiderSea water Sulphuric acidOranges VinegarBarley water Ignored – not a proper English gentleman! Listened – repeated in 1794 by a… Courtesy of the National Library of Medicine

Towards Modern Medicine Medical profession eventually realised controlled studies were the right way to go David Sackett, 1992: ‘evidence- based medicine’ – Randomised double-blind test is gold standard Cochrane Collaboration – Archives results from hundreds of medical studies – Image courtesy of Lab Science Career

On to Software Engineering… Martin Fowler (IEEE Software, July/Aug 2009) – Wrote the book on refactoring “[Using domain specific language] leads to two primary benefits. The first, and simplest, is improved programmer productivity... The second... is... communication with domain experts.” Interesting! Image courtesy of adewale_oshineye

‘Just One More Thing…’ Perhaps seems like a good idea Look at the paper more closely… – Two substantive claims – …without a single citation! Could be correct, but where’s the data? A Scottish verdict! Even the best of us aren’t doing this correctly “Many software researchers advocate rather than investigate” – Robert L. Glass (2002) Image courtesy of whatleydude Image courtesy of futureatlas.com

Take Note… Things are Starting to Change Growing emphasis on empirical studies in s/w research since mid-1990s – Papers describing new tools or practices routinely include results from some kind of field study ICSE 2009 Almost always must include some field testing results in top- tier journal/conference papers now – Progress! Image courtesy of Hacklock

An Interesting Result… Aranda & Easterbrook (2005): “Anchoring and Adjustment in Software Estimation” “How long do you think it will take to make a change to this program?” 3 groups given two page spec, one variation per group embedded in text… Control Group: “I’d like to give an estimate for this project myself, but I admit I have no experience estimating. We’ll wait for your calculations for an estimate.” Group A: “I admit I have no experience with software projects, but I guess this will take about 2 months to finish.” Group B: “… I guess this will take about 20 months to finish.” Groups don’t know about this variation…

What Happened? This is an anchoring effect, well known in psychology The anchor mattered more than anything else: – Experience in domain, software engineering – Tools used – Formality of estimation Q: Are agile projects similarly afflicted, just on a shorter and more rapid cycle? Group A (lowball)5.1 months Control Group7.8 months Group B (highball)15.4 months

Frequently Misquoted Sackman, Erikson and Grant (1968): “Exploratory experimental studies comparing online and offline programming performance” “The best programmers are up to 28 times more productive than the worst.” Dozens of books quote this – ‘all you need is a few really, really good people’ – More productive than… others! – We put ourselves at ‘good’ end of bell curve!

Pick it Apart Done in 1968!! Study designed to compare batch vs interactive, not measure productivity How was productivity measured? No metric Best vs worst? Twelve programmers for an afternoon – Next “major” study was 54 programmers… – … for up to an hour Very bad use of statistics! Image courtesy of zolierdos

So What Do We Know? Lutz Prechelt This guy knows stuff – via studies – Productivity variations between programmers – Effects of language – Effects of web programming frameworks Productivity and reliability depend on length of program’s text, independent of language abstraction level e.g. Perl, Ruby, Java, Assembler - ~the same LOC/hour Therefore: go high-level (trade-off: performance!)

A Classic Result… Boehm et al (1975): “Some Experience with Automated Aids to the Design of Large-Scale Reliable Software” – Plus many, many more since 1.Most errors are introduced during requirements analysis and design 2.The later they are removed, the more expensive it is to take them out (1/10/100?) Operational: ,000 lines of code 122 software errors 79% because of poor understanding of requirements

…Which Explains a Lot Pessimists (traditional): “If we tackle the jump in the error injection curve, fewer bugs will get to the expensive part of the fixing curve” Optimists (agile): “If we do lots of short iterations, the total cost of fixing bugs will go down”

Greatest Hits Volume 1 For every 20% increase in problem complexity, there is a 100% increase in solution complexity (Woodfield, 1979) The two biggest causes of project failure are poor estimation and unstable requirements (van Genuchten, 1991 – and many others) Development teams do not benefit from existing experience, and they repeat mistakes over and over again (Brossler, 1999)

Greatest Hits Volume 2 Rigorous inspections can remove 60-90% of errors before first test is run (Fagan, 1975) The first review & hour matter most (Cohen, 2006) Physical distance doesn’t affect post-release fault rates. Distance in organisational chart does Nagappan et al (2007) & Bird et al (2009) Shouldn’t our development practices be built around these facts? Image courtesy of schoschie

Here’s Another Errors tend to cluster – Half the errors are found in 15% of the modules (Davis, 1995 quoting Endres 1975) – About 80% of the defects come from 20% of the modules, and about half the modules are error free (Boehm and Basili 2001) Known since 1975! When you identify more errors than expected in some program module, keep looking!

Learn to Read How did you learn to program in any language? But… – You don’t just go out and write War and Peace, you’d read other skilled writers first – You learn to read a foreign language before you learn to write it Err… what about programming languages? Teach maintenance first (Harlan Mills, 1990) The best way to prepare [to be a programmer] is to write programs and to study great programs that other people have written – Susan Lammers, 1986, quoting a younger Bill Gates Image courtesy of dwyman

And Finally… Get into good habits early It’s easy to cling to bad ones habits habits