1. CS 510 Midterm 2 Q&A Fall 2014 Barry Boehm November 10, 2014

2. Make-or-Buy Decisions: Just Cost? In EC-13 (slide 11), there is an example of a Make or Buy Decision Analysis. While this makes sense from a cost point of view, how would schedule be factored into this? For example, the ÒBuyÓ option may be ready immediately versus the ÒBuildÓ or ÒReuseÓ options which could take some time to build. As a result, having something available now might outweigh saving money. Could the analysis be augmented to include an opportunity cost (or potential loss) that reflects the impact of schedule?

3. Make-or-Buy Decisions: Just Cost? No; consider other advantages and disadvantages

4. When to Select COTS? In the ICSM Development phase, one of the activities of the Initial increment is to ÒSelect hardware, COTS products, and outsource vendorsÓ. Why wouldnÕt this be part of the Foundations phase or would it only be in the Foundations phase if we believed it to be a risk that would require early analysis or prototyping?

5. When to Select COTS? Incrementally Commit Exploration phase: Stakeholder COTS constraints – B of A Master Net: IBM Mainframe compatibility – MedFRS: Initial commit to current network COTS; work next generation network COTS during Development Valuation, Foundations: Stakeholder agreements – MedFRS: Current integrated cardiac monitor supplier Development: Next-gen networking competitions

6. What Alternatives to Consider? 3. - In EP-10 page 4 the payoff matrix is made of the ÒNet Value" for both BB (Bold) and BB (Conservative). However, this Net Value is calculated from comparing BB options to the old option A. Now, How can we construct a payoff matrix when we have only two options? Let's say I never had option A as alternative; what should we consider as Net Value for Option B in that case? (Cost, Cost-effectiveness?)

7. What Alternatives to Consider? Do Nothing always an alternative OS Options: while initially using COTS Option A Continue to use Option A (do nothing) Option B-Conservative (BC) –Sure to work (cost=$600K) –Performance = 4000 tr/sec Option B-Bold (BB) –If successful,Cost=$600K Perf.= 4667 tr/sec –If not,Cost=$100K Perf.= 4000 tr/sec Which option should we choose?

8. Operating System Development Options Option BB (Bold) SuccessfulNot Successful Option BC (Conservative) Option A Performance (tr/sec)46674000 2400 Value ($0.3K per tr/sec)14001200 720 Basic cost600 170 Total cost600700600170 Net value NV800500600550 NV relative to Option A250-50500

9. What Decision Alternatives to Choose? Based on EC-9 (Slide 22). We know there are six Cost- Effectiveness Decision Criteria which include: Maximum available budget, Minimum performance requirement, Maximum effectiveness/cost ratio, Maximum effectiveness - cost difference, Return on investment (ROI), Composite alternatives. All of these criteria have some potential pitfalls, and it may lead to failure if follow any criteria directly. How can decision- makers make the most accurate decision based on these criteria? Is there any special guidance for them?

10. What Decision Alternatives to Choose? Usually best: ROI but avoiding weak spots Maximum available budget – Weak spot: May provide inadequate capability Minimum performance requirement – Weak spot: May be unachievable Maximum effectiveness/cost ratio – Weak spot: May provide inadequate capability Maximum effectiveness - cost difference – Weak spot: May be an overkill Return on investment (ROI) – Weak spot: May provide inadequate capability Composite alternatives – Usually best: ROI but avoiding weak spots

11. How to Evaluate Prototype Utility? ÒThe Prototype Will Give Imperfect Information, i.e.: P(IB|SF) ­ 0.0; Investigation (Prototype) says choose bold state of nature: Bold will fail. P(IB|SS) ­ 1.0; Investigation (prototype) says choose bold state of nature: Bold will succeedÓ. And in the calculation, we assume ÒSuppose the states of nature are equally likelyÓ, P(SF) = 0.50, P(SS) = 0.50. These possibilities are very important because they determines EV(IB, IC). In real world, how to get the value of P(IB|SF) and P(IB|SS), and does the P(IB|SF) and P(IB|SS) only relate to the investment of prototyping? Or there are some other factors affect these two values? In addition, how to evaluate the possibilities of the states of nature? Q3: For net expected value vs cost (EP-10 Table 20-1, Fig. 20-1): Is it always worth doing the prototype when the net expected value is positive (however low)? Can there be situations where it is worth doing even if the net expected value is zero or less?

12. How to Evaluate Prototype Utility? Relative progression between endpoints generally OK PROTO COST, $K P (PS|SF)P (PS|SS)EV, $KNET EV, $K 0600 50.300.8069.34.3 100.200.9078.28.2 200.100.9586.86.8 300.001.00900 8 4 0 102030 NET EV, $K PROTO COST, $K

13. Conditions for Successful Prototyping (or Other Info-Buying) 1.There exist alternatives whose payoffs vary greatly depending on some states of nature. 2.The critical states of nature have an appreciable probability of occurring. 3.The prototype has a high probability of accurately identifying the critical states of nature. 4.The required cost and schedule of the prototype does not overly curtail its net value. 5.There exist significant side benefits derived from building the prototype. 1.Qualitative: Marketing, Teambuilding, Vendor Evaluation

14. How Much Risk Resolution Is Enough? Balance risks of doing too much, too little Based on EC-15 (Slide 11): “Potential Pitfalls and Major Risks” are listed, we know there are so many types of pitfalls and major risks. So at each development stage, risk and evidence analysis should be conducted, which may cost much time and human resource. Is it the way how the real IT industry work? Under limited budget and time schedule, what can we do to avoid these pitfalls and minimize the risks? Or it mainly based on experience of project manager?

15. 11/03/04©USC-CSE15 How Much Risk Resolution Is Enough? -A COCOMO II Analysis Percent of Project Schedule Devoted to Initial Architecture and Risk Resolution Added Schedule Devoted to Rework (COCOMO II RESL factor) Total % Added Schedule 10000 KSLOC 100 KSLOC 10 KSLOC Sweet Spot Sweet Spot Drivers: Rapid Change: leftward High Assurance: rightward

16. OK to Assume Constant Inflation, Discount Rate? From EP-9.pdf, Present value, when calculating present values do we consider the risk of possibility that interest rate can change over time. Why and how, or why not? Inflation and Discount rates often change, but it is hard to predict how much or when. If variations are critical, it would be worth doing a sensitivity analysis using different rate variations

17. When is it OK to skip phases? 1. When considering ICSM stages and phases, it is established that if there is negligible risk in a certain phase, we can skip it. 2. We also know that the outputs from each stage in the Incremental commitment spiral process is used as the input for the next stage. The question is this: After the exploration phase, if the risk is negligible enough for us to skip the valuation phase and continue from the foundation phase, How do we get the inputs we need since the exploration phase output is different from the foundation phase input?

18. When is it OK to skip phases? When from-phase outputs match to-phase inputs Development phase inputs: List of approved features/requirements Problems reported against currently deployed system Approved changes Approved Development plan Feature allocation to increments Updated risks and mitigation plans Updated stakeholder commitments/ MOAs Proposals for outsourced development Exploration Phase If outputs match relevant Development inputs, can skip to Development If most match, can combine or skip one of Valuation and Foundations

19. Weighted Sum or DSC Rating Scales 1. When we perform a weighted sum analysis and a DSC analysis, the criterion scale can be self-defined. In this case, different scales might result in different result in choosing the alternative. For example, for scale 1, alternative B is better while for scale 2, alternative A is better. Does it make sense that the solution is dependent on the scale or there is a general common cognized scale, or just it does not matter?

20. Weighted Sum or DSC Rating Scales When there are a small number of comparable capabilities to evaluate, a simple (none, poor, marginal, adequate, good, very good) rating scale is OK When there are many and non-comparable capabilities to evaluate (such as the case study in SwEngrEcon, pp. 227- 234), a one-size-fits-all rating scale is not recommended When performance or availablity are critical, a Delivered System Capability (DSC) approach is recommended – Best to use Downtime (2% vs. 4%) than Availability (98% vs. 96%)

21. Cost of Downtime Survey Industry SectorRevenue/Hour Energy$2.8 million Telecommunications $2.0 million Manufacturing $1.6 million Financial Institutions $1.4 million Information Technology $1.3 million Insurance $1.2 million Retail $1.1 million Pharmaceuticals $1.0 million Banking $996,000 Source: IT Performance Engineering & Measurement Strategies: Quantifying Performance Loss, Meta Group, October 2000. 07/22/2014 ©USC-CSSE21

22. Users Manual, Training, or Both? 1. In the Production Phase of the second stage of ICSM, is it necessary to schedule some ÒtrainingÓ activities to help potential users if you already provide the user manual for them and why it is necessary? Watch out for The American Way: “If all else fails, read the instructions.” Often training will improve the Users manual or product; one of mu users manuals said, “Don’t leave the vehicle weight blank; you’ll crash the computer.” One training session convinced me to put a check into the program for this.