“The Role of Experience in Software Testing Practice” A Review of the Article by Armin Beer and Rudolf Ramler By Jason Gero COMP 587 Prof. Lingard Spring of 21

Introduction: What Is The Paper About? The authors set out to find out how experience is used in software testing in the real world, and set out to answer five specific questions. They did this by doing a study of three projects overseen by Siemens Austria. The authors defined “experience” as “practical knowledge that has been developed by direct observation of or participation in testing activities.” Their goal was to “better understand the role of experience for effective testing in order to develop successful testing strategies and tool support.” 2 of 21

Presentation Overview Part I: Why and how was their study done? Part II: A brief overview of the 3 Siemens projects: –A Telecommunications Project –A Banking and Insurance Project –A Railway Systems Project Part III: Summary of the results the authors obtained (What were the answers to the authors' five research questions?) Part IV: Authors' conclusions 3 of 21

Part I: Why and how was their study done? There haven't been a lot of studies which describe how software testing is actually done in the real world. Good information on how testing is actually conducted can be used to direct further research about testing. It is important to distinguish between ideas in academia about how testing ought to be conducted and how testing is actually conducted in the real world. 4 of 21

Part I: Why and how was their study done? The authors set out to answer these five research questions in their study: 1)“What testing activities are based on experience?” 2)“What is the perceived value of experience for testing?” 3)“What defined experience-based testing techniques are used?” 4)“How has the experience relevant for testing been established?” 5)“What measures are taken for managing and evolving experience in testing?” 5 of 21

Part I: Why and how was their study done? How did the paper's authors collect the data? Testers, developers, test managers and project managers were interviewed. Questions based on the research questions were asked during interviews. Project documents were also analyzed. The participants of the study reviewed the data that was collected to verify that it was accurate. 6 of 21

Part II: A brief overview of the 3 Siemens projects This source for this table was the article. 7 of 21

Part II: A brief overview of the 3 Siemens projects (Telecommunications Example – Slide 1) Project Mission: “Develop a high-reliable and high- performance software platform including IP-based services for telecommunication networks.” 35 People worked on this project, 12 in testing. Development and testing were done in Austria, Germany, other areas of Europe. 8 of 21

Part II: A brief overview of the 3 Siemens projects (Telecommunications Example – Slide 2) A majority of the tests were for non-functional requirements including: availability, performance, high load and long duration. The software would need to be used on two platforms, each with different hardware, and each of those could be configured in many ways. Component testing, integration testing and system testing were done. 9 of 21

Part II: A brief overview of the 3 Siemens projects (Telecommunications Example – Slide 3) Component Tests: test cases were derived from interface specifications and run with hardware device simulators. Personal experience dictated what additional tools to use. Integration Tests: mainly concerned with performance issues and done at the same time as development. Some tests were devised from the developers' experience due to the lack of detail in the requirements. System Tests: Test cases were derived from the requirements. Regression Testing: The selection of tests for regression testing was made based on the developers' experience. Developers advised the testers on what parts of the software were changed. 10 of 21

Part II: A brief overview of the 3 Siemens projects (Banking and Insurance Example – Slide 1) Replace an existing banking and insurance application with a Web application written in Java. Composition of the project team: employees of the banking and insurance company, Siemens personnel and people from other software engineering companies. Project team size: people. 11 of 21

Part II: A brief overview of the 3 Siemens projects (Banking and Insurance Example – Slide 2) The development process was incremental and consisted of several releases. The requirements consisted of use cases. Testing consisted of unit, integration, system and acceptance testing. 12 of 21

Part II: A brief overview of the 3 Siemens projects (Banking and Insurance Example – Slide 3) Unit and integration testing were done under the guidance of senior developers with experience in the technology. System tests: a team of 5 was drawn from important users and domain experts (again, experience is key here). The incremental nature of this project meant that, with each release, the testers knew likely places to find bugs based on their experience with previous releases. Because many functional and non-functional requirements were incomplete or ambiguous, some tests had to be made based on developers experience and the advice of domain experts. 13 of 21

Part II: A brief overview of the 3 Siemens projects (Railway System Example – Slide 1) This project involved development of a component of a railway control system. This was a safety- critical system, meaning that failures could not be allowed. About 100 people were involved in this project. There had to be a test case which corresponded with every single requirement. 14 of 21

Part II: A brief overview of the 3 Siemens projects (Railway System Example – Slide 2) Test cases for lower-level parts of the system were developed by testers familiar with railway control systems. Also, the requirements were not all precise, so exploratory testing, which was experience-based, had to be done. 15 of 21

Part III: Summary of the results the authors obtained (Q1: “What activities of testing are based on experience?”) The authors found that these activities relied on experience: –Test case development –Regression testing –Test automation 16 of 21

Part III: Summary of the results the authors obtained (Q2: “What is the perceived value of experience for testing?”)  More defects are found  Missing parts of the requirements specifications can still be tested for.  However, experience-based testing is expensive 17 of 21

Part III: Summary of the results the authors obtained (Q3: “What defined experience-based testing techniques are used?”) In Case #1: Telecommunications, the test cases had to be specified up front and had to be automated, which made experience-based testing difficult. Instead, more traditional testing techniques were used. In Case #2: Banking and Insurance, testing prior to the 2nd release relied on experience-based test cases based on experiences from the 1st release. Error guessing was used in this case to discover defects, and these test cases were based on where failures had occurred before. In Case #3: Railway Systems, error guessing had to be used to meet the IEC standard. 18 of 21

Part III: Summary of the results the authors obtained (Q4: “How has the experience relevant for testing been established?”) In other words, where do testers get their experience from? The authors found these sources cited most: –Previous testing of the same systems –Code analysis and defect removal –Participation in development and maintenance –Being a user of similar systems 19 of 21

Part III: Summary of the results the authors obtained (Q5: “What measures are taken for managing and evolving experience in testing?”) Experience was gained from just participating in testing itself. Making sure that testers share knowledge is important in managing and evolving experience. Iterative development also gave testers the opportunity to learn what to test for in each version of the software based on past experience. 20 of 21

Part IV: Authors' conclusions They note that one area for future research is to determine what the best combination of testing knowledge and domain knowledge is. The authors discovered that testing difficulties often arose due to imperfections in the requirement specifications. The authors say that, if perfect specifications cannot be created, employ experience-based testing. They noted that existing testing tools do not take into account the testers' experience, nor do they assist with gaining new experience, so perhaps these tools could be designed to fix these shortcomings. 21 of 21