University of Southern California Center for Systems and Software Engineering 09/02/09©USC-CSSE1 OCD Risk Management & DART CS 577a, Fall 2009 September 2, 2009

University of Southern California Center for Systems and Software Engineering Operational Concept Description (OCD) 09/02/09©USC-CSSE2

University of Southern California Center for Systems and Software Engineering Operational Concept Definition (OCD) 09/02/09©USC-CSSE3 Purpose of the OCD 1.To describe the success critical (key) stakeholders’ shared vision of the project being undertaken. 2.Key stakeholders typically include 1.the system’s users 2.the client 3.the customer, if different from the client 4.the maintainer 5.and the developers. 3.See OCD guidelines on how to identify other key stakeholders

University of Southern California Center for Systems and Software Engineering OCD Content and Completion Criteria VC Package FC Package OCD Content Shared Vision Success- Critical Stakeholders System Capabilities Descriptions Expected Benefits Benefit Chain, System Boundary and Environment System Transformation System Objective, Constraints & Priorities Capability Goals, LOS goals, Organization goals Constraints, relation to current system Proposed New Operational Concept Element Relationship diagram, Business Workflow Proposed New Operational Concept - Organization and Operational Transformation 09/02/09©USC-CSSE4

University of Southern California Center for Systems and Software Engineering Shared vision and context for success-critical stakeholders (1) Success- Critical Stakeholders (SCS) –Common SCS: system’s user, client, customer, maintainer, developer. –Project-specific SCS supplier, actor, volunteer, vendor, researcher System Capabilities Descriptions “ Sierra Mountainbikes, Inc’s Sales Department needs a faster, more integrated order entry system to increase sales. The proposed Web Order System will give us an e-commerce order entry system similar to Amazon.com’s that will fit the special needs of ordering mountain bicycles and their aftermarket components. Unlike the template-based system that our main competitor bought, ours would be faster, more user friendly, and better integrated with our order fulfillment system.” 09/02/09©USC-CSSE5

University of Southern California Center for Systems and Software Engineering ©USC-CSSE Shared vision and context for success-critical stakeholders (2) Benefit Chain Diagram – A good example 09/02/096

University of Southern California Center for Systems and Software Engineering ©USC-CSSE Shared vision and context for success-critical stakeholders (3) Benefit Chain Diagram A not so good example Common mistakes -Does not show the chain of benefits -Unclear initiative, outcome -Missing contribution -Incomplete benefit representation 09/02/097 Assumption: 1. No limits on no. of users 2. Stable support from CollectiveX for Network and Database functionality Implement the Web-based system depending on current system Developers, IV and V Providing Tutorials to the Client and Users. Business firms, students and teachers Use the system functionalities System to be beneficiary to the client Client Enhance the capabilities of existing system WEB- Based application

University of Southern California Center for Systems and Software Engineering Shared vision and context for success-critical stakeholders (4) System Boundary and Environment 09/02/09©USC-CSSE8

University of Southern California Center for Systems and Software Engineering System Objectives, Constraints, and Priorities System objectives –Capability Goals OC-1 Central Order Processing: Orders may be (i) entered and processed directly via the Sierra Mountainbikes (SMB) central website and Enterprise Resource Planning (ERP) system, or, in the case of telephone or fax orders (ii) entered by SMB service personnel. Orders are validated interactively, using validation criteria editable by administrators. –Level of Service Goals –Organization Goals OG-1: Increase sales and profits via more efficient order processing. 09/02/09©USC-CSSE9 LOS GoalsDesired LevelAcceptance LevelNotes Response time per entry (second)0.10.5Current system: 1

University of Southern California Center for Systems and Software Engineering System Constraints Examples: –CO-1: Windows as an Operating System: The new system must be able to run on Windows platform. –CO-2: Zero Monetary Budget: The selected NDI/NCS should be free or no monetary cost. –CO-3: Java as a Development Language: Java must be used as a development language. 09/02/09©USC-CSSE10

University of Southern California Center for Systems and Software Engineering Element Relationship Diagram 09/02/09©USC-CSSE11

University of Southern California Center for Systems and Software Engineering Business Workflow Diagram 09/02/09©USC-CSSE12

University of Southern California Center for Systems and Software Engineering Organizational and Operational Transformation Organizational Transformation –Identify any significant changes in organizational structure, authority, roles, and responsibilities that will result from transitioning to the new system. E.g. The need to hire a new system maintainer to take care of the system Operational Transformation –Identify any significant changes in operational procedures and workflows that will result from transitioning to the new system. E.g. Having the financial, delivery, and administrative processing concurrently progress rather than sequentially to decrease response time, subject to the check for payment validity before shipping an order. 09/02/09©USC-CSSE13

University of Southern California Center for Systems and Software Engineering Risk Management 09/02/09©USC-CSSE14

University of Southern California Center for Systems and Software Engineering Reactively Managing a Software Development Problem System Integration Time: We just started integrating the various software components to be used in our project and we found out that COTS* products A and B can’t talk to one another This is a problem, caused by a previously unrecognized risk, that materialized: The risk that two COTS products might not be able to talk to one another; specifically that A and B might not be able to talk to one another) We’ve got too much tied up in A and B to change Our best solution is to build wrappers around A and B to get them to talk via CORBA** This will result in: –a 3 month schedule overrun = $100K contract penalty –a $300K cost overrun *COTS: Commercial off-the-shelf **CORBA: Common Object Request Broker Architecture 09/02/09©USC-CSSE15

University of Southern California Center for Systems and Software Engineering Proactively Managing a Risk (assessment) System Design Time: –A and B are our strongest COTS choices –But there is some chance that they can’t talk to one another Probability that A and B can’t talk to one another = probability of loss: P(L) From previous experience, with COTS like A and B, we assess P(L) at 50% –If we commit to using A and B, and we find out at integration time that they can’t talk to one another Size of loss S(L) = $300K + $100K = $400K We have a risk exposure of RE = P(L) * S(L) = (.5) * ($400K) = $200K 09/02/09©USC-CSSE16

University of Southern California Center for Systems and Software Engineering Risk Management Strategy 1: Buying Information System Design Time: –Let’s spend $30K and 2 weeks prototyping the integration of A and B –This will buy information on the magnitudes of P(L) and S(L) –If RE = P(L) * S(L) is small, we’ll accept and monitor the risk –If RE is large, we’ll use one/some of the other strategies 09/02/09©USC-CSSE17

University of Southern California Center for Systems and Software Engineering Other Risk Management Strategies Risk Avoidance –COTS product C is almost as good as B, and we know, from having used A and C, that C can talk to A –Delivering on time is worth more to the customer than the small performance loss Risk Transfer –If the customer insists on using A and B, have them establish a risk reserve, to be used in case A and B can’t talk to each other Risk Reduction –If we build the wrappers and the CORBA connections right now, we add cost but minimize the schedule delay Risk Acceptance –If we can solve the A and B interoperability problem, we’ll have a big competitive edge on future procurements –Let’s do this on our own money, and patent the solution 09/02/09©USC-CSSE18

University of Southern California Center for Systems and Software Engineering Software Risk Management 09/02/09©USC-CSSE19 Checklists Decision driver analysis Assumption analysis Decomposition Performance models Cost models Network analysis Decision analysis Quality factor analysis Risk exposure Risk leverage Compound risk reduction Buying information Risk avoidance Risk transfer Risk reduction Risk element planning Risk plan integration Prototypes Simulations Benchmarks Analyses Staffing Milestone tracking Top-10 tracking Risk reassessment Corrective action Risk Resolution Risk Assessment Risk Control Risk Identification Risk Analysis Risk Prioritization Risk mgmt Planning Risk Management Risk Monitoring

University of Southern California Center for Systems and Software Engineering Top 10 Risk Categories: 1989 and /02/09©USC-CSSE Personnel shortfalls 2. Schedules and budgets 3. Wrong software functions 4. Wrong user interface 5. Gold plating 6. Requirements changes 7. Externally-furnished components 8. Externally-performed tasks 9. Real-time performance 10. Straining computer science Personnel shortfalls 2. Schedules, budgets, process 3. COTS, external components 4. Requirements mismatch 5. User interface mismatch 6. Architecture, performance, quality 7. Requirements changes 8. Legacy software 9. Externally-performed tasks 10. Straining computer science

University of Southern California Center for Systems and Software Engineering Primary CS577 Risk Categories (all on 1995 list) and Examples Personnel shortfalls: commitment (This is team member’s last course; only needs C to graduate); compatibility; communication problems; skill deficiencies (management, Web design, Java, Perl, CGI, data compression, …) Schedule: project scope too large for 24 weeks; IOC content; critical-path items (COTS, platforms, reviews, …) COTS: see next slide re multi-COTS Rqts, UI: mismatch to user needs (recall overdue book notices) Performance: #bits; #bits/sec; overhead sources Externally-performed tasks: Client/Operator preparation; commitment for transition effort 09/02/09©USC-CSSE21

University of Southern California Center for Systems and Software Engineering COTS and External Component Risks COTS risks: immaturity; inexperience; COTS incompatibility with application, platform, other COTS; controllability Non-commercial off-the shelf components: reuse libraries, government, universities, etc. –Qualification testing; benchmarking; inspections; reference checking; compatibility analysis 09/02/09©USC-CSSE22

University of Southern California Center for Systems and Software Engineering The Top Ten Software Risk Items 09/02/09©USC-CSSE23 Risk ItemRisk Management Techniques 1. Personnel ShortfallsStaffing with top talent; key personnel agreements; incentives; team-building; training; tailoring process to skill mix; peer reviews 2. Unrealistic schedules and budgets 4. Requirements mismatch; gold plating 5. User interface mismatch Business case analysis; design to cost; incremental development; software reuse; requirements descoping; adding more budget and schedule Stakeholder win-win negotiation; business case analysis; mission analysis; ops-concept formulation; user surveys; prototyping; early users’ manual; design/develop to cost Prototyping; scenarios; user characterization (functionality, style, workload) 3. COTS; external componentsQualification testing; benchmarking; prototyping; reference checking; compatibility analysis; vendor analysis; evolution support analysis

University of Southern California Center for Systems and Software Engineering The Top Ten Software Risk Items (Concluded) 09/02/09©USC-CSSE24 7. Requirements changes 9. Externally-performed tasks 6. Architecture, performance, quality 10. Straining Computer Science capabilities High change threshold; information hiding; incremental development (defer changes to later increments) Reference checking; pre-award audits; award-fee contracts; competitive design or prototyping; team-building Architecture tradeoff analysis and review boards; simulation; benchmarking; modeling; prototyping; instrumentation; tuning Technical analysis; cost-benefit analysis; prototyping; reference checking 8. Legacy softwareDesign recovery; phaseout options analysis; wrappers/mediators; restructuring

University of Southern California Center for Systems and Software Engineering Risk Exposure Factors (Satellite Experiment Software) 09/02/09©USC-CSSE25 Unsatisfactory Outcome (UO) Prob (UO) Loss (UO) Risk Exposure A. S/ W error kills experiment B. S/ W error loses key data C. Fault tolerance features cause unacceptable performance D. Monitoring software reports unsafe condition as safe E. Monitoring software reports safe condition as unsafe F. Hardware delay causes schedule overrun G. Data reduction software errors cause extra work H. Poor user interface causes inefficient operation I. Processor memory insufficient J. DBMS software loses derived data

University of Southern California Center for Systems and Software Engineering Risk Exposure Factors and Contours: Satellite Experiment Software 09/02/09©USC-CSSE26

University of Southern California Center for Systems and Software Engineering Risk Reduction Leverage (RRL) 09/02/09©USC-CSSE27 RRL - RE BEFORE - RE AFTER RISK REDUCTION COST · Spacecraft Example LOSS (UO) PROB (UO) RE B B LONG DURATION TEST $20M 0.2 $4M FAILURE MODE TESTS $20M 0.2 $4M PROB (UO) RE A A 0.05 $1M 0.07 $1.4M COST$2M$0.26M RRL = = 10

University of Southern California Center for Systems and Software Engineering Software Risk Management 09/02/09©USC-CSSE28

University of Southern California Center for Systems and Software Engineering Risk Management Plans 09/02/09©USC-CSSE29 For Each Risk Item, Answer the Following Questions: 1. Why? Risk Item Importance, Relation to Project Objectives 2. What, When? Risk Resolution Deliverables, Milestones, Activity Nets 3. Who, Where? Responsibilities, Organization 4. How? Approach (Prototypes, Surveys, Models, …) 5. How Much? Resources (Budget, Schedule, Key Personnel)

University of Southern California Center for Systems and Software Engineering Risk Management Plan: Fault Tolerance Prototyping 1. Objectives (The “Why”) –Determine, reduce level of risk of the software fault tolerance features causing unacceptable performance –Create a description of and a development plan for a set of low-risk fault tolerance features 2. Deliverables and Milestones (The “What” and “When”) –By week 3 1. Evaluation of fault tolerance option 2. Assessment of reusable components 3. Draft workload characterization 4. Evaluation plan for prototype exercise 5. Description of prototype –By week 7 6. Operational prototype with key fault tolerance features 7. Workload simulation 8. Instrumentation and data reduction capabilities 9. Draft Description, plan for fault tolerance features –By week Evaluation and iteration of prototype 11. Revised description, plan for fault tolerance features 09/02/09©USC-CSSE30

University of Southern California Center for Systems and Software Engineering Risk Management Plan: Fault Tolerance Prototyping ( concluded ) Responsibilities (The “Who” and “Where”) –System Engineer: G. Smith Tasks 1, 3, 4, 9, 11, support of tasks 5, 10 –Lead Programmer: C. Lee Tasks 5, 6, 7, 10 support of tasks 1, 3 –Programmer: J. Wilson Tasks 2, 8, support of tasks 5, 6, 7, 10 Approach (The “How”) –Design-to-Schedule prototyping effort –Driven by hypotheses about fault tolerance-performance effects –Use real-time OS, add prototype fault tolerance features –Evaluate performance with respect to representative workload –Refine Prototype based on results observed Resources (The “How Much”) $60K - Full-time system engineer, lead programmer, programmer (10 weeks)*(3 staff)*($2K/staff-week) $0K - 3 Dedicated workstations (from project pool) $0K - 2 Target processors (from project pool) $0K - 1 Test co-processor (from project pool) $10K - Contingencies $70K - Total 09/02/09©USC-CSSE31

University of Southern California Center for Systems and Software Engineering Risk Monitoring Milestone Tracking –Monitoring of risk Management Plan Milestones Top-10 Risk Item Tracking –Identify Top-10 risk items –Highlight these in monthly project reviews –Focus on new entries, slow-progress items Focus review on manger-priority items Risk Reassessment Corrective Action 09/02/09©USC-CSSE32

University of Southern California Center for Systems and Software Engineering Project Top 10 Risk Item List: Satellite Experiment Software 09/02/09©USC-CSSE33 Risk Item Mo. Ranking This Last #Mo. Risk Resolution Progress Replacing Sensor-Control Software Top Replacement Candidate Unavailable Developer Target Hardware Delivery Delays Procurement Procedural Delays Sensor Data Formats Undefined Action Items to Software, Sensor Teams; Due Next Month Staffing of Design V&V Team Key Reviewers Committed; Need Fault- Tolerance Reviewer Software Fault-Tolerance May Fault Tolerance Prototype Successful Compromise Performance Accommodate Changes in Data Meeting Scheduled With Data Bus Bus Design Designers Testbed Interface Definitions Some Delays in Action Items; Review Meeting Scheduled User Interface Uncertainties User Interface Prototype Successful TBDs In Experiment Operational TBDs Resolved Concept Uncertainties In Reusable Required Design Changes Small, Monitoring Software Successfully Made

University of Southern California Center for Systems and Software Engineering Conclusions Risk management starts on Day One –Delay and denial are serious career risks –Data provided to support early investment Win Win spiral model provides process framework for early risk resolution –Stakeholder identification and win condition reconciliation –Anchor point milestones Risk analysis helps determine “how much is enough” –Testing, planning, specifying, prototyping,… –Buying information to reduce risk 09/02/09©USC-CSSE34