© G. Ruhe Günther Ruhe University of Calgary & Expert Decisions Software Engineering Institute Pittsburgh, January 15, 2008 The Art and Science of Software Release Planning

© G. Ruhe Agenda  Decision Support in Requirements Engineering  Software Release Planning  EVOLVE*  Decision Support System ReleasePlanner®  Case Studies  Future Research

© G. Ruhe Laboratory for Software Engineering Decision Support  Created in July 2001 at the University of Calgary  Research team of 12 researchers (2 undergraduate, 3 graduate, 5 PhD students, and 2 profs)  Research topics: Decision support (systems) for  Software release planning  Project portfolio planning  Software resource management  COTS selection  Verification and validation  Research approach:  Interdisciplinary  Both fundamental and applied research  Empirical validation of results  University spin-off company: Expert Decisions

© G. Ruhe Problem description User requirements Developer’s requirements System design Component requirements Component-level design Unit code Executable components Executable system Usable system Used system is verified against is developed into is integrated into is validated against How to allocate test effort? When to terminate testing ? Inspection? Re-inspection? Reuse of components? Classes? Objects? Methods? Which technique? Which artifacts? Decisions, Decisions, Decisions …. Which requirements?

© G. Ruhe Description of a Decision Problem  Set A = {a1, a2, …} of alternatives (these alternatives are not necessarily described explicitly);  Set G = {g1, g2, …,gn} of criteria to evaluate each alternative a  A from different perspectives, and  Preference structure (partial order defined on A) Main categories of decision problems:  Selection (P  ): Select one alternative a*  A or a subset A*  A;  Triage (P  ): Assign each alternative a  A to one of the classes C1, C2, …, Ck.  Ranking (P  ): Arrange all alternatives in A according to an order a1 ≥ a2 ≥ …( a ≥ b means “alternative a is at least as good as b”). Other classification:  Structured versus unstructured decisions  Strategic, tactical, and operational decisions

© G. Ruhe Decisions in Requirements Engineering

© G. Ruhe Executable program Change requirement Software Evolution Program variants Planning Project goals Project plan Software Development Specification Information is  uncertain  inconsistent  incomplete  fuzzy Decision space  of large size  of high complexity  is dynamically changing Multiple objectives  Usability  Security  Reliability  Maintainability  Portability Hard and soft constraints on  Time  Effort  Quality  Resources Why are RE Decisions so Hard?

© G. Ruhe Expected Benefits from Decision Support  Basic assumption The more qualified computer-based/intelligent decision support is provided based on well-defined processes and roles, the more likely it is to find an appropriate decision.  Decision support systems combine the intellectual resources of individuals and organizations with the capabilities of the computer –to generate and evaluate (new) solution alternatives; –to allow more transparent decisions (to be better understood by involved people); –to analyse decisions pro-actively; –to better react on changes in the problem parameters (re- planning); –to make more effective (trade-off) decisions (improved quality); –to make more efficient decisions (improved cost-benefit).

© G. Ruhe Agenda  Decision Support in Requirements Engineering  Software Release Planning  EVOLVE*  ReleasePlanner®  Case Studies  Future Research

© G. Ruhe Planning for Software Releases  Incremental (not monolithic) software development  Set of competing (to be implemented) features  Stakeholder prioritize objects in terms of urgency, business value, risk, …  Release cycle time can be predefined or can be open  Limited resources and budgets need to considered  Release: Set of features constituting the next version of the product  Release can be refined into increments (strategic versus operational planning)  Hypothesis: Comprehensive and systematic release planning results in more reliable plans, better customer satisfaction, and more efficient usage of short resource ( = maximum value)

© G. Ruhe Decision Support for Product Release Planning (1/2)  Uncertainty is pervasive and unavoidable in software engineering.  Uncertain software engineering decision problems are unlikely to be explicitly modeled and completely formalized. Set of Features/Objects Interation 1 Interation 2Interation 3 Release k Release k+2 Release k+1 ! Stakeholder involvement ! Unclear objectives ! Effectiveness & efficiency ! Resource, risk, and technological constraints

© G. Ruhe Decision Support for Product Release Planning (2/2)  Decisions in software engineering are made by humans. They are based on both explicitly formulated and implicitly known objectives and constraints.  The goal of decision support is not to replace human judgment and expertise, but to assist in making better decisions.  Based on uncertain models, any formalized computational technique (subsumed as computational intelligence) in isolation is unlikely to determine meaningful results.  The advantage of the human intelligence based approach is that it is able to better handle soft and implicit objectives and constraints.  Support to solve −to solve the right problem and −to solve the problem right.

© G. Ruhe Maturity Factors for Product Release Planning  Processes rigour  Level 1  Level 5  Degree of integration into product lifecycle management  Level 1  Level 5  Planning based on measurement and defined criteria  Level 1  Level 5  Type and degree of stakeholder involvement  Level 1  Level 5  Continuity of planning and re-planning  Level 1  Level 5

© G. Ruhe Art and Science of Release Planning  Art: Focus on the human intuition and communication –Pro: Flexible, easy to perform, ability to handle soft and implicit concerns –Con: Effectiveness and efficiency for mid-size to large scale problems, not transparent and not repeatable, re-planning is not supported, stakeholder involvement limited, resource consumption hard to take into account  Science: Emphasis on formalization of the problem and application of computational algorithms to generate best solutions. –Pro: Effective and efficient, easy re-planning, transparent, repeatable, comprehensive stakeholder involvement possible, resource consumption part of the solution procedure –Con: Needs formal description of the problem, needs computer support

© G. Ruhe Analysis of Existing Approaches RP Methods Dimensions [Greer 2004] [Penny 2002][Bagnall et al. 2001] [Jung 1998][Karlsson & Ryan 1997] [Denne & Cleland- Huang 2004] [Ruhe & Ngo-The 2004] Scope 1 release Chunks of small releases 2 releases planned ahead ObjectivesBased on benefit of system changes Based on benefit of features to customers Based on weight of customers Based on value of requirements Based on customer satisfaction Based on return on investment Based on value, urgency, stakeholders weights and satisfaction Stakeholders involvement Project manager Developers, project management Project manager, customer Involvement of project manager is implied Project manager, customer, users All major stakeholders Prioritization mechanism Optimization -based No defined prioritization scheme Optimization- based OptimizationAHPIFM HeuristicsStakeholders prioritize features; Optimization heuristics used to balance conflicts Technological constraints Not available precedenceNot available Precedence (precursor) Coupling and precedence Resource constraints Cost, riskEffortCost Cost, time-to- market Effort, risk, schedule System constraintsOperational risks Not available Character and quality of solutions One solution plan One solution plan by any chosen search algorithm One solution plan generated One solution plan provided One plan spanning many release periods Several alternative solution plans are provided. These plans are diversified and fulfill some target quality level Tool supportNot available Time-tracking system Not available Partially available ReleasePlanner®

© G. Ruhe Agenda  Decision Support in Requirements Engineering  Software Release Planning  EVOLVE*  ReleasePlanner®  Case Studies  Future Research

© G. Ruhe Evolutionary Solution Approach EVOLVE* Computational Intelligence Interation 1Release 1 Release 2 Interation 2 Interation 3Release 3 Human Intelligence

© G. Ruhe  Phase 1 - Modeling: Definition/update of –Decision variables –Technological and resource constraints –Objectives –Stakeholder voting  Phase 2 - Exploration: Generate alternative solutions from the application of −Voting analysis −Specialized linear integer programming −Branch and bound algorithms −Heuristics  Phase 3 - Consolidation: Human decision maker studies solution alternatives −Maximally diversified solution alternatives −Absolute and relative degrees of stakeholder satisfaction −Comparison between solutions −Sorting capabilities −Reporting capabilities −Definition of re-planning scenarios The Three Phases of EVOLVE*

© G. Ruhe Diversification Principle Set of qualified & maximally diversified alternative solutions Value Urgency 95% Optimality alternatives A portfolio of qualified solutions being structurally diversified is more appropriate than a single solution that is formally optimal but unlikely to solve the “right problem”.

© G. Ruhe Agenda  Decision Support in Requirements Engineering  Software Release Planning  EVOLVE*  ReleasePlanner®  Case Studies  Future Research

21

© G. Ruhe Business objects & objectives Resource types & their capacities Resource consumptions Ranking of objects Stakeholder & their importance Key Components

© G. Ruhe ReleasePlanner® for Elicitation of Customer Priorities

© G. Ruhe Feature elicitation Problem specification (2) Feature repository (1) Resource utilization analysis (3) Generation of release plan alternatives (7) Evaluation of plan alternatives (8) Selection of most attractive planning scenarios and alternatives New/ev olving features Shortage of resources Stakeholder voting (4) Stakeholder voting analysis (5) Product Manager Stakeholders ReleasePlanner® Resource Estimation What-if analysis (9) Stakeholder access definition Reporting (10) Corporate Business IT Business data, products and processes (6)

© G. Ruhe Agenda  Decision Support in Requirements Engineering  Software Release Planning  EVOLVE*  ReleasePlanner®  Case Studies  Future Research

© G. Ruhe Case Studies and Lessons Learned (LL)  Corel iGrafx (2003/04): Product release planning LL: Structured process  Broader stakeholder involvement  Siemens CT SE 3 ( ): Service portfolio planning LL: Explicit process and objectives  Facilitator for understanding & improvement. Involvement of key account managers.  City of Calgary ( ): Project portfolio planning LL: Integration of release planning into business processes  WestJet (2005/06): Project portfolio planning LL: Risk mitigation from running what-if scenarios  Solid Technology (2005): Product release planning LL: Greater likelihood of feasible and stable plans  Chartwell Technology ( ): Product release planning LL: More conscious release decisions  more competitive products  Siemens Audiologische Technik (2007):  better understanding of (world-wide) customer needs

© G. Ruhe Case Study at Trema Laboratories  Case study based on release planning process at Trema Laboratories, Inc.  Used both current ad hoc planning methodology and ReleasePlanner® –Determination of the operational feasibility of the strategic release plan –Assessment of the value of using ReleasePlanner® compared to ad hoc planning  Study used one of the projects for the delivery of the next release: –Releases delivered every three months –Team size of 14 with varying levels of skills, experience and availability –Roles vary from product specialist, analyst/designer, developer, and/or tester –Adopted a 2-weekly increment –Six stakeholders with varying interests and input and geographically dispersed.

© G. Ruhe Comparison of Solutions ReleasePlanner®Adhoc Release Plan Comparing Results from ad hoc and ReleasePlanner®

© G. Ruhe Conclusions ReleasePlanner ® Adhoc Release Plan Comparing Results from ad hoc and ReleasePlanner ®  One of the five alternative plans from ReleasePlanner® is similar to the plan generated using the ad hoc method  Alternative 1 has an additional requirement implemented  Using ReleasePlanner® has taken less than a day  Ad hoc approach to arrive at same conclusions took weeks of work and meetings  ReleasePlanner® allows easy re-planning in case of changed requirements, capacities or effort estimates.  Substantial time savings  Higher likelihood to understand and address customer priorities

© G. Ruhe Added Value Quantitative V(T)= $187, for T = 1 year

© G. Ruhe Agenda  Decision Support in Requirements Engineering  Software Release Planning  EVOLVE*  ReleasePlanner®  Case Studies  Future Research

© G. Ruhe Summary and Conclusions  Requirements engineering is a decision-centric process  Decision support for product release planning aims to move from reactive  proactive mode of performance  Good release decisions are based on explicit and systematic processes aimed to balance intuition and rationality  Ongoing and future research: –Release planning for product lines –Non-linear release planning –Operational resource allocation –Release planning with flexible release dates –Learning and explanation for release planning –Light-weight re-planning –Empirical evaluation