Software testing research Ossi Taipale November 2011 Lappeenranta University of Tech.

Contents  Background  Research process  Research methods  Research projects, research objectives and results:  Basic research of software testing, ANTI-project  Reference model of software testing, MASTO- project  Software testing in the cloud, STaaS-project  Intended quality, STX-project  Research co-operation

Background, why testing  During the last decades productivity of Finland has been higher than in other EU countries, but now the productivity in the product and service sectors is slowing down meaning lower growth rate of GDP.  The key to increase productivity is in the hands of industry and service sectors that produce or apply ICT.  Productivity can be increased by creating tangible products or service products.  Testing enables product creation.

Background, software testing research Top-down approach:  1. Years 2004–2007 ANTI-project: Basic research of software testing, affecting factors.  2. Years 2008–2011 MASTO-project: Software testing strategy and management. Theses: Software Testing Strategy and Management and Software Testing in the Cloud. The reference model of software testing ISO/IEC Software Testing Standard. 1. Vocabulary, 2. Testing Processes, 3. Reporting. Part 4. Methods is under development. Ossi Taipale is a member of the WG26.  3. Years , new STX-project: Testing level and ”Testing Spice”.

Software testing research projects  Top down approach in the research projects : ANTI-project: Basic research of SW testing, MASTO-project: Reference model of SW testing, STX-project: Intended quality, ISO/IEC 29119: SW testing standard ISO/IEC and IEEE : and Testing Spice, 15504, IEEE1012, SW testing in the cloud

Background, software testing definition  We use the definition: Software testing consists of verification and validation.  By testing, we try to answer two questions: are we building the product right and are we building the right product?  Verification answers the question: are we building the product right? Basic verification methods include inspections, walkthroughs, and technical reviews. Checklists are used as the tools of verification.  Validation answers the question: are we building the right product? Validation activities can be divided into unit testing, integration testing, usability testing, function testing, system testing, and acceptance testing.

Research process OntologyBeliefs of the essence of phenomena under investigation Selection of standards and constructs: e.g. ISO/IEC 12207, and Research philosophy Epistemology Beliefs of the suitable method Concrete selections of the study: Delphi, survey, and grounded theory methods Empirical study Background assumptions

Research methods  Mixed methods approach consisting of quantitative and qualitative analyses (Hirschheim 1985).  The results of the quantitative and qualitative analyses are triangulated to increase the validity of the study. The principle of triangulation means that more than one method, observer or data set is used in a study to complement each other and to verify the findings (Denzin 1978).

Research methods  The objective of the Delphi method is to achieve the most reliable consensus of opinion of a group of experts. Delphi method can be used in finding good arguments about an ongoing process (Directing the study).  The survey method is used to gather information about feelings, motivations, plans, and beliefs. Tools for gathering such information are usually questionnaires or interviews (Fink & Kosecoff 1985).  Grounded theory (Strauss & Corbin 1990). The objective of a qualitative study is rather to find or reveal facts than to prove existing theorems. Strauss and Corbin (1990) define qualitative research as any kind of research that produces findings not arrived at by means of statistical procedures or other means of quantification.

Some results of the ANTI-project ( )  Hypotheses for improvement, process view: Testing ought to be adjusted according to the business orientation of the OU. Product oriented organizations should adopt a formal planned testing process. Service oriented organizations should adopt a flexible testing process that allows appropriate coordination with their clients. Enhanced testability of software components. Consider testability as an important factor in the selection of components. Review the testing process of your suppliers. Efficient communication and interaction between development and testing seems to reduce costs and improve quality. Product oriented OUs could develop straightforward processes because knowledge transfer is predictable. Service oriented OUs could develop innovative solutions because knowledge transfer varies according to the customer. The early involvement of testing seems to reduce costs and improve quality. The planning of testing is more straightforward in product oriented than in service oriented OUs. The risk-based testing strategy helps in avoiding ad hoc decisions on testing, because the decision on what to test is based on a predefined testing strategy.

Some results of the ANTI-project ( )  Hypotheses for improvement, knowledge management view: Business orientation affects the testing organization. In product oriented OUs, the organization model ought to support repetitive testing tasks that enable development of a deeper expertise. In service oriented OUs, the organization model ought to support adaptation to the processes of the customers that demand broader expertise. Business orientation affects the knowledge management strategy. OUs ought to adjust their knowledge management strategy according to the business orientation. Business orientation and the knowledge management strategy affect outsourcing. Codification of knowledge enables testing outsourcing. Identifying and avoiding barriers and using enablers improve knowledge transfer.

Some results of the ANTI-project ( )  Associations between testing outsourcing and knowledge management: Outsourcing seems to be more effective when independent testing agencies have enough domain knowledge. Outsourcing verification tasks is more difficult than outsourcing validation tasks.

Publications of the ANTI-project ( ) 1. Finding and Ranking Research Directions for Software Testing, EuroSPI, Budapest, Taipale, O., K. Smolander, H. Kälviäinen. 2. Cost Reduction and Quality Improvement in Software Testing, SQM, Southampton, UK, Taipale, O., K. Smolander, H. Kälviäinen. 3. A Survey on Software Testing, SPICE Conference, Luxembourg, SPICE,Taipale, O., K. Smolander, H. Kälviäinen (2006). 4. Factors Affecting Software Testing Time Schedule, ASWEC, Sydney, Australia, Taipale, O., K. Smolander, H. Kälviäinen (2006). 5. Improving Software Testing by Observing Practice, ISESE, Rio de Janeiro, Brazil, Taipale, O., K. Smolander. 6. Observing Software Testing Practice from the Viewpoint of Organizations and Knowledge Management, ESEM, Madrid, Spain, Taipale, O., K. Karhu, K. Smolander. 7. Triangulating Testing Schedule Over-runs from Knowledge Transfer Viewpoint, Lappeenranta University of Technology, Research Report 104, Taipale, O., K. Karhu, K. Smolander. 8. Outsourcing and Knowledge Management in Software Testing, EASE, Staffordshire, UK, K. Karhu, O. Taipale, K. Smolander.

Jussi Kasurinen Overview on MASTO-project ( )

MASTO  The project was a 3-year academic joint project conducted by LUT and Aalto University, funded by Tekes (Technology and Innovation Agency of Finland) and a group of companies and organizations.  The focus on the project was on software testing in different organizational levels.  MASTO at LUT focused on test process and organizational aspects.  Aalto focused on the testing work and project-level aspects.  Researcher exchange, Jussi Kasurinen – Lund, Sweden, Leah Riungu-Kalliosaari – Limerick, Ireland. Software-producing organization Projecese Pof these Organizational level, one company has one set of these Project-level work, one company may have several of these

Collection of qualitative and quantitative data, 65 interviews Round: Type of interview Interviewee role in the organization Interview themes 1st round: Semi- structured interview, 12 OUs Software designer or people responsible for software design and architecture. Design and development methods, Testing strategy and -methods, Agile methods, Standards, Outsourcing, Perceived quality 2nd round: Structured survey, additional semi- structured questions, 31 OUs, including 1st. Manager, test manager or project leader responsible for development project or test process of a product. Test processes and tools, Customer participation, Quality and Customer, Software Quality, Testing methods and -resources 3rd round: Semi- structured interview, 12 OUs, same as 1st. Tester or people responsible for doing testing in the development project. Testing methods, Testing strategy and – resources, Agile methods, Standards, Outsourcing, Test automation and –services, Test tools, Perceived quality Customer in testing 4th round: Semi- structured interview, 10 OUs, some new. Manager, test manager or project leader responsible for development project or test process of a product. Test policies, Test strategies, Test plans, Testing work, Software architecture, Delivery models, New software development concepts

ISO/IEC Test process

Test strategy constructs (and layers) Journal article “Software Test Automation in Practice: Empirical Observations” Conference paper: “Exploring Quality Concepts in Software Organizations” Conference paper: “Test Case Selection and Prioritization: Risk or Design-based?” Conference paper “Analysis of Problems in Testing Practices” Conference paper: “A Study on Agility and Testing Processes in Software Organizations” Covered in article “Software Test Automation in Practice: Empirical Observations” Conference paper “How Test Organizations Adopt New Testing Practices and Methods?” Conference paper “A Self-Assessment Framework for Finding Improvement Objectives with ISO/IEC Test Standard” Testing standard and publications

 According to ISO/IEC (Software product quality), the quality in the software product is a composition of several quality attributes  These quality attributes define the quality objectives for software Quality and quality objectives

All at least somewhat important Only in 9 out of 248 (3,6%) assessments “not important” Perceived quality, not absolute quality! Quality characteristics

Trust between the customer and developer. – Specifications are distributed freely, things are discussed and more information is available if needed. Existing conformance with the ISO/IEC model concepts. – Workflow has a feedback system which is used, plans and made decisions are assessed and changes can be made. Elaboration of quality – Everyone in the organization knows what kind of quality is preferred and tries to achieve that. Important things that affect perceived quality (Survey 2009)

End-product criticality – Obviously a rocket control system is tested more thoroughly than a mobile game, but preferred quality comes from the domain, not criticality. Development method – An agile product can be as high quality as a “traditionally” made product. – Open source is not necessarily better (or worse) than closed software. Outsourcing – In large organizations, outsourcing does not seem to affect perceived quality. – In smaller organizations it may, but not always. Things that do not affect the perceived quality that much even if they seem to have a big influence

Organizations do not tend to try out new ideas. Sporadic development is done when the inconveniences overcome acceptable losses. Even if the test process feedback is collected, it is often neglected if the process is “good enough”. How do the organizations develop their test process

 Combination of Test Improvement model (TIM) maturity levels and ISO/IEC processes.  Two results:  General maturity and conformance with the standard model.  Process improvement objectives to develop test process. Processes from ISO/IEC Maturity levels from TIM Individual assessment of each process area, development ideas General maturity/conformance estimation Self assessment framework

Publications from the MASTO project ( ) 1. Test Case Selection and Prioritization: Risk-Based or Design-Based? Jussi Kasurinen, Ossi Taipale and Kari Smolander, ESEM 2. A Self-Assessment Framework for Finding Improvement Objectives with ISO/IEC Test Standard, Jussi Kasurinen, Per Runeson, Leah Riungu and Kari Smolander, EuroSPI 3. Software Test Automation in Practice: Empirical Observations, Jussi Kasurinen, Ossi Taipale, Kari Smolander, AiSE 4. Exploring Perceived Quality in Software Organizations, Jussi Kasurinen, Ossi Taipale, Jari Vanhanen and Kari Smolander, IEEE 5. Analysis of Problems in Testing Practices, Jussi Kasurinen, Ossi Taipale and Kari Smolander, APSEC 6. A Study on Agility and Testing Processes in Software Organizations, Vesa Kettunen, Jussi Kasurinen, Ossi Taipale, and Kari Smolander, ISSTA 7. How Test Organizations Adopt New Testing Practices and Methods? Jussi Kasurinen, Ossi Taipale and Kari Smolander, TAICPART 8. Exploring the Perceived End-Product Quality in Software-Developing Organizations, accepted for publication in International Journal of Information System Modeling and Design, IGI Global, Jussi Kasurinen, Ossi Taipale, Jari Vanhanen and Kari Smolander.

Software Testing in the Cloud ( ) Leah Riungu-Kalliosaari

Objectives and background  Empirical study aimed at understanding how organizations can successfully use the cloud for testing  Cloud computing: A model for enabling convenient, on-demand network access to a shared pool of configurable computing resources (e.g. networks, servers, storage, application and services) that can be rapidly provisioned and released with minimal management effort or service provider interaction (NIST definition).  STaaS: A model for software testing whereby testing of an application is provided as a service across the internet.  Cloud-based testing is provided on-demand and billed on pay- per-use basis, so that the user pays only for the resources they have used.

Initial observations  The trend towards cloud services is growing  Traditional desktop applications  online software services e.g. SaaS, StaaS  According to a 2009 Gartner report, cloud computing changes the way software is delivered and used. This change means that traditional licensing of software seems to loose ground where as hiring of software services seems to be on the rise.  Mixed reactions  Generally positive  Positive – due to e.g. cost reduction, flexibility, performance, access to global markets  Negative – due to concerns regarding e.g. security, domain knowledge, test data management  Neutral – mainly explorative approach

Implications on Testing  With the cloud becoming more common, the quality expectations are growing.  New issues and complexities surrounding these services will need to be addressed.  Testing will become more important  Need for testing increases  Testing is seen as arena of cloud computing where the barriers to entry are relatively low. Testing in the cloud simply entails the use of computing resources and models in the cloud to perform testing. Cloud-based testing is provided on-demand and billed on pay-per-use basis, so that the user pays only for the resources they have used.

1a. SaaS software 3. Testing the cloud 2. Testing environments in the cloud 1b. Non-SaaS software 1.The system or application under test is available online 2.Testing infrastructure and platforms are hosted in the cloud (Including crowdsourcing/Human as a Service-(Haas)) 3.Testing of the cloud itself Facets of testing in the cloud Testing in the Cloud

A roadmap towards testing in the cloud  Develop understanding of cloud computing Cloud computing is increasingly becoming a feasible choice for testing.  Carry out pilot projects We recommend that organizations carry out pilot projects that enable them to fully explore the potential benefits.  Come up with elaborate strategies Cloud testing vendors as well as testing and quality assurance consulting firms will be called upon to offer advice and direction.  Enhance team interaction and prepare for complexities Organizations need to be prepared for additional testing brought about by the complexities and new requirements for cloud-based applications and systems.  Enhance co-operation between research and industry These include, among others, application issues e.g. the types of applications best suited for testing in the cloud; management issues like how to organize the human resources for cloud-based testing (e.g. crowdsourcing); and legal and financial issues such as the management of test data across different global jurisdictions and how to device appropriate pricing models.

Current research problem  To better understand how cloud based services can be achieved with intended quality  Identify the most important quality requirements for cloud-based services  Provide recommendations for achieving overall intended quality of cloud-based services  Difference between intended and achieved quality  What are the tradeoffs, and what is the optimal balance.

Publications from the Testing in Cloud – project ( ) 1. Research Issues for Software Testing in the Cloud, Leah Muthoni Riungu, Ossi Taipale, Kari Smolander, 2nd IEEE International Conference on Cloud Computing Technology and Science. 2. Software Testing as an Online Service: Observations from Practice, Leah Muthoni Riungu, Ossi Taipale, Kari Smolander, Third International Conference on Software Testing, Verification, and Validation Workshops. 3. Testing in the Cloud: Exploring the Practice, IEEE Software, Leah Riungu-Kalliosaari, Ossi Taipale, Kari Smolander.

Software testing and development for intended quality, STX ( ) Tero Pesonen

Objective  To show how software development, software testing and intended quality depend on one another.  Traditional software development and service models  Emerging XaaS (Everything as a Service) architectures, technologies and service models.  The project results help the participating companies in improving the efficiency of their quality management and software testing and hence the efficiency of their software development as a whole.  Testing techniques  Testing as a service

Research Problem OU’s are evaluated through an assessment framework Intended Software Quality ISO/IEC 25010, Software Product Quality Software Testing ISO/IEC 29119, Testing Spice, IEEE Std 1012 StaaS Software Development Software Products ISO/IEC ISO/IEC New Services SaaS

Research Problem  The framework used as a ”grid” through which each OU is assessed.  Intended quality vs. achieved quality.  Top-down and bottom-up analysis (testing work  project management  organizational level).  The assessment framework is defined through international software engineering standards.  We need to anchor the assessment to a specific, concrete view on what quality, software testing and software development are.  Based on standards, the model remains coherent throughout the assessment.  Project results and usability; contribution to standardization work.  The results include improvement proposals for the OU’s.  Data collection methods include interviews and surveys.

Work division, entire project  3 year project  12 months / work package; WP 1 starts 8/2011  WP1: Requirements for intended quality  Identifies the quality requirements and attributes that the OU's perceive as key for achieving the intended quality.  Defining intended quality, the importance of different quality characteristics in achieving it, traditional vs. SaaS  WP2: Building the intended quality  Achieved vs. Intended quality vs. the software development process  Action research  WP3: Assessment  How the OU’s can improve their quality management and software testing to achieve the intended quality more efficiently?

Contents of work package (WP) 1  Study units are OU's.  What are the quality requirements, and hence the quality attributes, that the OU perceives as key for achieving the intended quality (level & characteristics)?  Are some quality attributes more important than others when determining whether the targeted quality has been achieved? Do OU's or projects differ here?  Does SaaS development emphasize different kind of targeted quality (level, characteristics) to traditional software development?  Data collection for the first qualitative study September – November /  Additional data from other projects also incorporated.  Data analysis may produce first leads.  Second data collection round will constitute quantitative studies based on round 1 observations.

Publications from the STX project ( ) 1. Observations on eBusiness implementation capabilities in heterogeneous business networks", 11th IFIP International Conference on e-Business, 2011, Kaunas, Lithuania Pesonen T., Smolander K.