Towards the Integration of Quality Attributes into a Software Product Line Cost Model Andy Nolan1, Silvia Abrahão2 Paul Clements3, John McGregor4, Sholom Cohen3 1Rolls-Royce, UK 2Universidad Politécnica de Valencia, Spain 3Software Engineering Institute, USA 4Clemson University, USA

2 Rolls-Royce

The need for Software Product Lines 4 The Control Systems department is responsible for the Engine Electronic Controllers (EECs) for a range of small and large gas turbine engines for the aerospace industry. The software is developed to DO-178B Level-A standards The company has been developing high integrity software for over 20 years and has extensive data on its processes and productivity. We have the largest order book in history, new engine development places greater demand on the software team (shorter time scales and lower costs) Since 2008, Rolls-Royce has developed a PL which has potential for both the software and hardware aspects of engine design.

The need for Cost Models 5 Since 2004, we have invested in cost models to predict software cost based on COCOMO II. The cost models have led to an improvement in stability and productivity, with an average of around 11% productivity gain This benefit has occurred because a project has a better understanding of the Quality Attributes that affect cost and risk. When the SPL initiative was launched in 2008, the development of a reliable and comprehensive estimation model was seen as critical to making the right decisions to develop core assets that would bring the greatest business value.

6 Recap of SPLC 2009

Overview of the Model 7 This picture shows the overview of the model. Step 1: provided the size of each individual asset based on a library of features from past projects. Step 2: determine the asset value. The assets has been “weighted” to reflect their value rather than size. This was done because it was observed that not all assets are equal --- 30% of the assets account for 70% of the costs so, SIZE is not the KEY factor---. A new step “P-Process Model” was added to model the safety-critical development process and to understand which processes are required to develop an asset and which processes are required to deploy an asset. We also recognized that the process used to develop and deploy an asset varies depending on the asset variation mechanism. Step 3: was used to map the life of features ACROSS THE FUTURE ENGINE PROGRAMS. Step 4: was introduced so that the SPL team could understand the COSTS to develop a project using the TRADITIONAL METHODS and then COMPARE these costs with the SPL development costs (see step 10). Step 5 was introduced to model the additional costs to develop a SPL asset. This information was taken from CO-PL-MO – the RUSE (developing for reuse) and DOCU (additional documentation) factors. Step 6 was used to map the deployment of SPL assets into projects to understand if there was a net benefit, per asset, when considering the development, deployment and maintenance of assets. The step also revealed those features that would be bespoke to each project. Step 7 was a mechanical process and generated all the information together to understand the new costs for developing a project based on SPL assets. Step 8 was added to model the organization costs. This was derived from the SIMPLE model and consists of costs above-and-beyond traditional project development. Step 9 was added to model the effect of introducing the SPL initiative into the business. With any change comes an initial “shock” followed by a settling in period. COCOMO II was again used to model the business environment. Step 10 performed the final analysis and compared and contrasted the benefits of the SPL with a more traditional project centric organization. --------------------------------------------------------------------------------------------- ------------------------------------------------------------------------------ Rather then write more, this was taken from the paper. I quickly described the 10 steps and showed how we added drag factors to create a “realistic” business case. This may not be relevant to this paper. I suggest, when presenting, ask the audience to read the paper for more detail. OK! ------------------------------------------- The model can be used for a variety of purposes such as to communicate with business leaders, model decisions at the business, perform risk analysis, assess the return on investment of improvements, etc.

8 Recap of SPLC 2010

Cost = Size * Environment Size isn't everything! Environment Team capability Team experience Team complexity and location Process capability Management capability Tool & IT capability Project complexity Integration complexity Development approach Development standard Development domain Size Lines of Code Complexity

cost cost SPL COCOMO II Size & Complexity Quality Attribute 10 cost COCOMO cost COCOMO II Size & Complexity Quality Attribute Size, Complexity, Testability Maintainability Maturity, Reusability, Portability, Variability...... SPL The SPLC 2010 paper proposed that the existing cost model had to be extended with other Quality Attributes to get a better understanding of costs & benefits

The evolution of cost models 11 [2004] First generation of the Rolls-Royce cost model was based on COCOMO II The software product is represented as a single size measure (SLOC). Limited use of the architecture or characteristics of the product being developed [2008] Second generation of the model allowed us to estimate the cost to develop and deploy individual core assets The model estimates the cost-benefit of each core asset based on size, complexity, volatility, and difficulty to develop and deploy [2011] To understand the relevant Quality Attributes of the core assets and the relationship between them To understand and integrate Quality Attributes into the cost model

Selecting Quality Attributes 12 Selecting Quality Attributes SPLC 2011

Objectives 13 Look at quality attributes and their impact on the cost- benefit of Software Product Lines Every factor (question) a cost model considers is another opportunity to: Refine your understanding, make trade decisions and optimise the project Incorporate quality attributes into the Rolls-Royce Cost Model to analyze: The cost/benefit of having core assets with certain qualities The impact on quality attributes of design decisions Use the cost model to derive design principles for the core asset team For example how much variability should there be and when ius there too much or too little?

Maturity 14 Maturity of a core asset has been defined here as the “degree to which an asset is free from further modification”. A low maturity asset is likely to be exposed to changes and depending on when they manifest, this can lead to high levels of effort to fix defects Maturity becomes a sensitive issue for SPLs especially if products are using low maturity assets. Maturity can be estimated from process exposure and project exposure

Testability 15 It is an important quality attribute for any type of product, and in particular for safety-critical products. A non-SPL safety-critical product invests around 52% of its total development effort on some form of verification and validation. In a SPL at Rolls-Royce, data shows that RELATIVELY up to 72% of a product’s overall effort will be spent in some form of V&V Testability can be estimated from the number of test cases (decision points) required to exercise the complete asset

Need estimation tool based on both the number AND type of variations Variability 16 The selection of a specific variation mechanism for a core asset can have an impact on the final product development & deployment cost. The cost of variability in a core asset is calculated by multiplying the cost of deploying the asset (in a specific process) by the cost of using the different variation mechanisms Future Work Need estimation tool based on both the number AND type of variations

Need to develop a pre-process measure of reusability 17 The reusability of core assets largely determines the success of product line projects. Higher reusability of core assets indicates the potential for a higher return-on-investment. This can be measured retrospectively. Future Work Need to develop a pre-process measure of reusability

Need to develop a pre-process measure of maintainability 18 Maintainability of core assets is of great importance. It is also important to understand the degree of maintainability that a core asset should have to be cost-effective to the business. Measured by the effort to maintain an asset (normalised by size). Future Work Need to develop a pre-process measure of maintainability

19 Trade off analysis SPLC 2011

Quality Attribute Trade-offs 20 Firstly, we created a hypotheses diagram to show the relationship between the Quality Attributes

Quality Attribute Trade-offs 21 Then we measured the relationship between the attributes (strength and direction)

Quality Attribute Trade-offs 22 Now we can derrive information from the model ? is it better to have 2 simpler assets or 1 larger asset, to have selectable assets rather than configurable, constrain the requirements or build in variability?

23 SPLC 2012?

Next Steps 24 The Rolls-Royce estimation model will need to be updated to take into account the other quality attributes selected (i.e., maturity, variability and testability) Other quality attributes will be evaluated and the model developed further Model v.2. Without derived relationships Cost Business Sat. Project Sat. Variability Volatility Testability ROI Benefit Maturity Reusability Customer Sat. Maintainability -L -H -M H L M

25 1st International Workshop on Quantitative Methods in Software Product Line Engineering (QMSPLE 2011) QMSPLE 2011