1. University of Southern California Center for Systems and Software Engineering July 2008ICM and CP Workshop © USC-CSSE1 Initial Competitive Prototyping Survey Results Dan Ingold University of Southern California Center for Systems and Software Engineering http://csse.usc.edu

2. University of Southern California Center for Systems and Software Engineering July 2008ICM and CP Workshop © USC-CSSE2 Outline of Session First ~half of session –Describe survey method –Present preliminary results, some interpretations –Questions and answers Remainder of session –Participants contribute to survey base –Continue discussion

3. University of Southern California Center for Systems and Software Engineering July 2008ICM and CP Workshop © USC-CSSE3 Background John Young (OSD/AT&L) memo dated 9/17 –Need to discover issues before… SDD –During SDD… teams should be producing detailed… designs—not solving… technical issues –Programs [will] provide for two or more competing teams producing prototypes through Milestone (MS) B –AII acquisition strategies requiring USD(AT&L) approval must… include competitive, technically mature prototyping through MS B

4. University of Southern California Center for Systems and Software Engineering July 2008ICM and CP Workshop © USC-CSSE4 Expected Benefits of CP (from Young memo) Reduce technical risk Validate designs Validate cost estimates Evaluate manufacturing processes Refine requirements In total, reduce time to fielding

5. University of Southern California Center for Systems and Software Engineering July 2008ICM and CP Workshop © USC-CSSE5 CP Survey Method: Conduct surveys and interviews of leading software professionals (government, industry, academia) Goal: Gather recommendations, assess impacts, and sense expectations for Competitive Prototyping Sample: (N~90) drawn from: –Industry (recommendations of NDIA) –Government (chosen by OSD/AT&L) –Academia (researchers in Sw/Sys Engineering)

6. University of Southern California Center for Systems and Software Engineering July 2008ICM and CP Workshop © USC-CSSE6 Investigators Principal investigator: Dr. Barry Boehm –TRW Professor of Software Engineering, Viterbi School of Engineering, University of Southern California –Director, Center for Systems and Software Engineering Assistant investigator: Dan Ingold –Research Assistant, PhD Student

7. University of Southern California Center for Systems and Software Engineering July 2008ICM and CP Workshop © USC-CSSE7 Survey Construction Patterned after similar DAU survey Supplemented with questions suggested by literature search Piloted, revised in four rounds with academic & industry colleagues

8. University of Southern California Center for Systems and Software Engineering July 2008ICM and CP Workshop © USC-CSSE8 Very Preliminary Results Based on 9 completed surveys, 1 follow-up interview May not be very representative Not reviewed for internal consistency Interpretation may reflect reviewer’s biases

9. University of Southern California Center for Systems and Software Engineering July 2008ICM and CP Workshop © USC-CSSE9 Is CP good for the government and industry? Consensus: yes, in general good for both But slightly better for government Acceptable financial payoffs and ROI

10. University of Southern California Center for Systems and Software Engineering July 2008ICM and CP Workshop © USC-CSSE10 Why does/should government use CP? Because it… Major factor Minor factor Not factor Is mandated  Improves cost  Improves schedule  Improves performance  Reduces risk  Improves maturity  Increases trust DoesShould

11. University of Southern California Center for Systems and Software Engineering July 2008ICM and CP Workshop © USC-CSSE11 Expectations: Young memo vs. Survey Young memoSurvey “Reduce technical risk”Major factor “Validate cost estimates”Minor factor “Reduce time to fielding”Minor factor

12. University of Southern California Center for Systems and Software Engineering July 2008ICM and CP Workshop © USC-CSSE12 Goals of CP Resolve most important risks  Resolve all significant risks    Demonstrate key technologies    Demonstrate operational capabilities  Be operationally complete, ready for FSD–

13. University of Southern California Center for Systems and Software Engineering July 2008ICM and CP Workshop © USC-CSSE13 High-level capabilities, or low-level specifications? Q: Requirements for CP efforts should be based on high-level capabilities and levels-of-service, rather than low-level detailed specifications Majority “somewhat agree”, some “somewhat disagree” Concern may be, as expressed by one interviewee, that high-level capabilities may tilt the playing field (non-uniform requirements)

14. University of Southern California Center for Systems and Software Engineering July 2008ICM and CP Workshop © USC-CSSE14 Active Stakeholder Participation Q: CP efforts should encourage the active participation of end-users and other stakeholders throughout the life-cycle…. All agree, many strongly Supports ICM goal to “satisfice” success-critical stakeholder win conditions Addressing the (likely changing) needs of stakeholders throughout life-cycle may help CP address the Young memo goal to “refine requirements”

15. University of Southern California Center for Systems and Software Engineering July 2008ICM and CP Workshop © USC-CSSE15 Frequent demonstrations, evidence of scalability Q: CP efforts should emphasize frequent demonstrations of progress…, and presentation of evidence that the system can scale…. Majority agree, most strongly Scalability of solution is major risk in traditional acquisitions Lone dissenter was, again, concerned about a level playing field

16. University of Southern California Center for Systems and Software Engineering July 2008ICM and CP Workshop © USC-CSSE16 Evolutionary / Iterative Development Q: CP efforts should be developed in an evolutionary or iterative fashion, and designed with ability to provide interim operational capabilities…. Surprising: answers span from “strongly agree” to “disagree”, though 2:1 agree May be due to choice of phrasing, reference to “interim operational capabilities” Will be explored further in follow-up interviews

17. University of Southern California Center for Systems and Software Engineering July 2008ICM and CP Workshop © USC-CSSE17 Risk Resolution Q: CP efforts… resolve most of the important technical risks prior to SDD. Broad agreement, but some disagreement Supports Young memo goal that CP should “reduce technical risk” Consistent with other answers suggesting risk reduction is “major factor” Interviewee observes, however, that many issues still not resolved until SDD

18. University of Southern California Center for Systems and Software Engineering July 2008ICM and CP Workshop © USC-CSSE18 Prototype risky areas only? Q: Prototyping is an investment in buying information to reduce risk. In low-risk areas, there is no need to prototype. A standard explanation of the reason to prototype Yet answers split evenly between agreement and disagreement Respondents expressing multiple benefits of prototyping, not just “risk reduction”?

19. University of Southern California Center for Systems and Software Engineering July 2008ICM and CP Workshop © USC-CSSE19 Reduce requirements changes and rework? Q: CP efforts lower the volume of rework and requirements changes that occur during SDD. Broad agreement Unsurprising, agrees with expectations, and goals of Young memo

20. University of Southern California Center for Systems and Software Engineering July 2008ICM and CP Workshop © USC-CSSE20 Overemphasis on prototype? Q: Acquisitions using CP can… overemphasize the prototype [and] underemphasize other critical preparation activities. Broad agreement, some disagreement Need to understand these other critical activities, ensure CP acquisition guidance addresses them

21. University of Southern California Center for Systems and Software Engineering July 2008ICM and CP Workshop © USC-CSSE21 Retaining the Team Q: In the evaluation interval between each phase of a CP down-select, the government should continue to fund competitors at a LOE sufficient to retain core engineering skills…. Broad agreement Supports Young memo observation that CP “provide[s] a method to exercise and retain certain critical core engineering skills” Implications for (multi-year?) funding?

22. University of Southern California Center for Systems and Software Engineering July 2008ICM and CP Workshop © USC-CSSE22 Consolation prizes Q: Acquirers should consider capitalizing on investments in down-selected bidders by offering “consolation prizes” such as IV&V contracts. Respondents agree by 2:1 margin But interviewee observes that larger contractors aren’t in IV&V business Perhaps a strategy to attract smaller (more innovative?) players, who would do IV&V?

23. University of Southern California Center for Systems and Software Engineering July 2008ICM and CP Workshop © USC-CSSE23 CP Funding But respondents split on whether unlimited contractor funding should be permitted Unlimited funding by larger players may price smaller players out of market Funding sourcePercent Government75% Contractor16% Other9%

24. University of Southern California Center for Systems and Software Engineering July 2008ICM and CP Workshop © USC-CSSE24 Changes in Acquirers Will CP require greater technical competence / larger staffing of PMO’s? –Split decision, though majority agree –Greater need for technical evaluation? Will CP require more PMO investment in evaluation tools and testbeds? –Split decision –Interviewee says only in contractors

25. University of Southern California Center for Systems and Software Engineering July 2008ICM and CP Workshop © USC-CSSE25 Questions and Answers ???