T esting daily builds in an iterative development process Presentation at EuroSTAR 2003 Bjarne Månsson, KK Data Danmark

Slide 2 Testing daily builds Bjarne Månsson, August 2003 KK Data Danmark - the company Founded 1994 by Klaus Karkov Sub-supplier of embedded software solutions for instruments and industrial applications Consultancy on software testing, project management and software process improvement Applications and consultancy include bus ticket system (CTS) audio codecs (Scientific-Atlanta) hearing aids (Oticon) flight maintenance equipment (Terma) mobile telephones (Nokia) medical devices (Novo Nordic) Supplier of the RDS532 RDS Encoder See more at

Slide 3 Testing daily builds Bjarne Månsson, August 2003 KK Data Danmark - Bjarne Månsson M.Sc. (EE) from Denmark’s Technical University M.Phil. (EE) from University of Leeds, UK Internal Auditor (Bywater plc) Certified Software Tester (ISEB) 30 years’ experience in electronics and software development hereof 12 years’ experience in software process improvement and 7 years’ experience in software testing Fields of work telecommunications data acquisition (production, laboratory) CNC machines medical devices CRM software

Slide 4 Testing daily builds Bjarne Månsson, August 2003 Testing daily builds - the scenario The product web based client/server software for CRM (Customer Relationship Management) consisting of 7 individual software components Source code is version controlled in Microsoft Visual Source Safe (VSS) source code is checked into VSS every day Components are build every day Components are integrated into a complete system every day including an automatic regression test Source code of a daily build is labeled with a unique label we can always rebuild a particular version

Slide 5 Testing daily builds Bjarne Månsson, August 2003 The daily build - the process Source code is checked into VSS at the end of the day developers are requested to check in stable (tested) code otherwise developers shall leave out the code till the next day The source code is automatically built during the night source code is taken from the VSS all components are compiled components are integrated into an installation file the build is regression tested

Slide 6 Testing daily builds Bjarne Månsson, August 2003 The daily build - the problem The 7 components build was not stable only 20% (19 of 87) daily builds were successful an internal release point was on 18-Dec; only here, some improving is spotted Figure: Each dot above the x-axis is a failed component build

Slide 7 Testing daily builds Bjarne Månsson, August 2003 The daily build - the explanation and the impact Source code is checked into VSS at the end of the day often in the middle of some code implementation difficult to reach a stable point at the end of the day no time for proper module testing The derived impact using (losing) time on debugging why the build failed only making work-arounds – ”aarh, if I just had time to test this yesterday” frustration between component teams – ”it’s your fault that my code is not tested today” no regulary regression test - component interfaces are drifting apart delayed start on system testing degraded product quality (stability)

Slide 8 Testing daily builds Bjarne Månsson, August 2003 The daily build - the next step Source code for the build is taken from a promotion branch of the VSS Promotion branch shall only hold stable (tested) code Source code is transferred to the promotion branch by a qualifying process developer gets other components’ source code from the promotion branch developer compiles the candidate code against the other components developer integration tests the component against the other components only if test is successful, the candidate code is checked into the promotion branch developer only promotes code when it is ready (but at least once a week)

Slide 9 Testing daily builds Bjarne Månsson, August 2003 The daily build - the result of promoting code The 7 components build process is becoming stable 75% (84 of 127) daily builds were successful failed builds are evenly distributed in time (catching drift in component interfaces) Figure: A dot above the x-axis is a failed build

Slide 10 Testing daily builds Bjarne Månsson, August 2003 The daily build - the regression test Automated regression test: On a successful build, a regression test is automatically run Component test coverage varies from 20% to 90% With a stable build, we are now able to do regression tests 60% (50 of 84) regression tests were successful failed tests come in bursts in time (it takes time to correct derived errors) Figure: A dot above the x-axis is a failed test

Slide 11 Testing daily builds Bjarne Månsson, August 2003 The daily build - and the quality of the system Release criterias System tests completed No defects of severity 1 and severity 2 Defect trend falling (defects found per test hour) … and a lot more Internal releases 18-Dec-2000: internal “synchronisation” 24-Aug-2001: 1 st  (for internal use) 30-Nov-2001: 2 nd  (for  -test customers) Release 28-Feb-2002: Version 1.0

Slide 12 Testing daily builds Bjarne Månsson, August 2003 The daily build - and the quality of the system Defect trend Defects severity 1 and 2 FoundOpenTest hoursTrend -> 18-Dec > 24-Aug > 26-Oct > 2-Nov > 9-Nov > 16-Nov > 23-Nov > 30-Nov Aug -> 30-Nov

Slide 13 Testing daily builds Bjarne Månsson, August 2003 The daily build - and the quality of the system Defect trend variation? Paradox that defect trend goes > > 0.39 Feasible explanation internal synchronisation: system test specification is not complete and we use a lot of monkey test hours 1 st  : system test specification is complete but now we have not enough test hours 2 nd  system test specification is complete and we get sufficient qualified test hours Sufficient quality? System tests completed - OK No defects of severity 1 and severity 2 – 18 of severity 2 Defect trend falling - falling up to the 2 nd  release - OK

Slide 14 Testing daily builds Bjarne Månsson, August 2003 The daily build - and the quality of the system Any improvements up to the release? Defects severity 1 and 2 FoundOpenTest hoursTrend -> 10-Feb > 17-Feb > 24-Feb > 28-Feb Feb -> 28-Feb

Slide 15 Testing daily builds Bjarne Månsson, August 2003 The daily build - lessons learned The good... The code for the build is stable up to the release Component code is tested against other promoted components The developer only promotes code when the code is brought to a stable point Components are always synchronised System testing can start early Release criteria is seriously handled Tests are properly specified Time to market Succeeding releases tend to be more stable

Slide 16 Testing daily builds Bjarne Månsson, August 2003 The daily build - lessons learned the bad... The build code may be a week old Promotion process needs coaching Difficult to gain enough time for system testing (classical) and error correction Poor quality Component -, integration – and system test tend to merge into one test -> the big bang test

Slide 17 Testing daily builds Bjarne Månsson, August 2003 The daily build - lessons learned and the ugly … Time to market and poor quality cause that development resources are used for service packs and hotfixes Next release is delayed Development cost / support cost

Slide 18 Testing daily builds Bjarne Månsson, August 2003 The daily build - lessons learned but we know what to do