2. Cumulative IOT&E Results Through FY 2008
Ideal
Effective
Number Effective
And Suitable
FY08
Suitable
Total: 28% of Systems Not Suitable
2007: 4 of 8 (50%) Not Suitable
2008: 2 of 6 (33%) Not Suitable
Cumulative Number of BLRIP Reports

3. DSB DT&E Taskforce Main Conclusion May 2008
“ the single most important step necessary to correct high suitability failure rates is to ensure programs are formulated to execute a viable systems engineering strategy from the beginning, including a robust reliability, availability, and maintainability (RAM) program, as an integral part of design and development. No amount of testing will compensate for deficiencies in RAM program formulation.”

4. Section 231 Report to Congress Core T&E Principles
T&E should concentrate on measuring improvements to mission capability and operational support based on user needs;
T&E programs should experiment learn and understand the strengths and weaknesses of a system and its components, and the effect on operational capabilities and limitations;
DT and OT activities should be integrated;
T&E should begin early, be more operationally realistic, and continue through the entire system life-cycle;
Evaluation should be conducted in the mission context expected at time of fielding to the user in terms of operational significance;
Evaluations should include a comparison against current mission capabilities;
Evaluations should take into account all available data and information;
T&E should exploit the benefits of appropriate M&S.

5. New Acquisition/T&E Policies McQueary/Finley Memo on Assessment of Op Test Readiness (21 May 2007)
The DUSD(A&T) shall conduct an independent Assessment of Operational Test Readiness (AOTR) for all ACAT ID programs and special interest programs designated by the USD(AT&L)
The CAE shall consider the results of the AOTR prior to making a determination of materiel system readiness for IOT&E.

6. New Acquisition/T&E Policies Young Memo on Competitive Prototyping (19 Sep 2007)
All acquisition strategies requiring USD (AT&L) approval must be formulated to include competitive, technically mature prototyping through MS B.

7. New Acquisition/T&E Policies Young-McQueary T&E Policy Letter - (22 Dec 2007)
DT and OT test activities shall be integrated and seamless
Evaluations shall include a comparison with current mission capabilities
T&E should assess improvements to mission capability and operational support based on user needs
To more effectively integrate DT and OT, evaluations shall take into account all available and relevant data and information, including contractor data
Operational evaluators will continue to fulfill their statutory roles in providing assessments of operational effectiveness, operational suitability, and survivability to the Milestone Decision Authority
To realize the benefits of modeling and simulation, T&E will be conducted in a continuum of live, virtual, and constructive environments.

8. New Acquisition/T&E Policies McQueary-Finley Memo on Reliability Improvement WG (15 Feb 2008)
Ensure programs are formulated to execute a viable systems engineering strategy, including a RAM growth program.
Ensure government organizations reconstitute a cadre of experienced T&E and RAM personnel.
Implement mandated integrated DT and OT, including the sharing and access to all appropriate contractor and government data and the use of operationally representative environments in early testing.

9. New Acquisition/T&E Policies McQueary-Finley Memo defining Integrated Testing (May 2008)
“Integrated testing is the collaborative planning and collaborative execution of test phases and events to provide shared data in support of independent analysis, and evaluation.”

10. New Acquisition/T&E Policies Young Memo on RAM Policy (July 2008)
The Service Secretaries are directed to establish Service policy to do the following:
Effective collaboration between the requirements and acquisition communities
Development contracts and acquisition plans must evaluate RAM during system design.
Evaluate the maturation of RAM through each phase of the acquisition life cycle.

11. Senior Leadership Buy-In of New Reliability/T&E Policies
“Having performance is important, but not as
important in most cases, as having
reliability.”
- Hon. Donald Winters, Secretary of the Navy (Sept 3, 2008)

12. Initiatives to Improve Reliability, Maintainability, and Availability
Reliability, Availability, Maintainability and Cost Manual (RAM-C Manual)
DOT&E on JCIDS Functional Control Boards
GEIA Standard 009, RFP and Contract Language, Investment Model
Reliability Growth in design phase
Ownership Cost
Number of Failures in the Field
RAM growth monitoring for incentives, Young/Bolton memos
RAM program Evaluation and Standards, testing KPP
RAM field data collection, feedback
Right Contract
and Incentives
Right Design
and Redesign
Right Validation
Right Requirements
Right Development
Right Next Increment

13. Basic Model R2 = .81 Improvement Ratio Investment/APUC
= Major System
Investment = Reliability Improvement Ratio
0.3659
2.119
X APUC 1
0.8
0.6
0.5
0.4
0.3
0.2 10 8 6 5 4 3 2
Improvement Ratio 7
100 80 60 50 40 30 20
1000
700
500
400
300
200
2000
Investment/ APUC
R2 = .81
Improvement Ratio
To create the basic model, we developed a cost estimating relationship (CER), shown in here by performing regression analysis on data from 17 programs.
Required investment increases linearly with average production unit cost (APUC) and as a power of the reliability improvement ratio.
APUC is essentially normalizing for program size and complexity.
The reliability improvement ratio is (New MTBx − Old MTBx)/Old MTBx
MTBx can be mean time between failure, mean time between removal, mean time between system abort, mean cycles between removal, or other similar measure relevant to a specific program.
The R2 for this equation is 0.81, meaning 81% of the variability among the data can be accounted for.
The required investment increases linearly with APUC and a power of the reliability improvement ratio.
The CER appears to be valid across technologies, across different types of weapon systems, and across a wide range of complexity, from components to subsystems to complete platforms.
Investment/APUC

14. Investment in Reliability Improvement, $M
Notional Example
Effect of Reliability Investment on System Cost (UAV)
500
1000
1500
2000
2500
3000 20 40 60 80
100
120
Investment in Reliability Improvement, $M
20-Year Cost, $M
20-year Cost, $ M
example is based on a notional unmanned aerial vehicle.
used the basic model to relate investment in reliability to achieved reliability,
used the dynamic model to determine how many UAVs would have to be procured to ensure that 100 are available at a 95 percent confidence level
translated that into total cost using the CASA model. (In this specific case, we regressed the support cost output from the CASA model against a range of reliability values and then used the result of the regression to estimate support cost as a function of reliability.
calculated production cost directly from the number of UAVs.
Total cost comprised the investment in reliability plus the cost to produce the required number of UAVs plus 20 years of support for each UAV
Chart shows relatively modest investments in reliability relative to production unit cost produce large reductions in total cost.
Beyond a 5:1 ratio of investment to production unit cost, the return on investment gradually decreases.
Over the range of investments shown on the chart, there is always a return on investment. However, the curve will bend back up when investments in reliability do not return savings of at least 1:1.
Investment in Reliability, $ M