1. For Official Use Only – Pre-Decisional EVM Central Repository: Reporting Compliance & Data Quality Findings January 29, 2009 Russ Vogel Acquisition Resources and Analysis, Enterprise Information & OSD Studies Office of the Under Secretary of Defense for Acquisition, Technology and Logistics Russell.vogel@osd.mil (703) 845-6677

2. For Official Use Only – Pre-Decisional 2 Overview: How We Use The Data Providing Timely and Accurate Key Acquisition Data

3. For Official Use Only – Pre-Decisional 3 EVM-CR Metrics (as of 1/5/2009) Number of programs reporting: –MDAP: 69 –MAIS: 11 Number of contracts/tasks reporting: –MDAP: 166 –MAIS: 17 Number of Submissions –Total: 3586 –Monthly: >250 New Submissions Number of Users –Total: 1303 –Active: 394 Downloads Per Months: >2000

4. For Official Use Only – Pre-Decisional 4 EVM-CR Reporting Compliance In September 2008, began first level data quality review of submission to the EVM-CR –Goals Compliance with CDRLs: timeliness and completeness (e.g., are all the required formats being submitted) Compliance with policy: data submission form factor (e.g., EDI) Alignment to other OSD (and service) data systems (e.g., DAMIR, AV SOA) –Coordinating activities with Service-level Acquisition Staff to make corrective actions as required. Findings: Most programs making effort to meet requirement –As of Jan 2009, many of the identified issues are being worked on and we’re seeing good progress. –As expected: normal, day to day “mistakes” (e.g., forgot to hit the SUBMIT button) –Beginning to tackle the more difficult issues

5. For Official Use Only – Pre-Decisional 5 EVM-CR Reporting Compliance Process & Data Quality Issues Easy fixes (requiring minor software changes, better guidance/training) –Mislabeling submission file (e.g., identifying an XML file as an X12 TS839) –Data errors (contract numbers, contractor name, “as-of date” not as expected) –Missing Components of the submission (e.g. no Format 5) For example, submitting a partial data package on due date and then on a subsequent day submitting missing file(s) as a “re-submit” Requirement: submit a complete data package as a single event –Submissions against wrong submission events Data Validations Rules / Sanity Check –Examples BCWS must be less then or equal to BAC TAB must equal CBB unless an OTB is specified –Potential Causes Incomplete submission (missing data in EDI/XML file) Some issues identified (and corrected) in EVM-CR data processing routines that read EDI/XML files Validation rules may need to be updated (theory vs. reality)

6. For Official Use Only – Pre-Decisional 6 EVM-CR Reporting Compliance Process & Data Quality Issues Submitted EDI or XML files not processing –Again, some issues identified and corrected in EVM-CR data processing routines that read EDI/XML files –Non-compliant XML Current policy / requirement is EDI X12 compliant with TS-839 format (e.g., WINSIGHT TRN format) EVM-CR also supports XML consistent with WINSIGHT XML-Schema Not all XML files are acceptable (e.g. Excel saved as XML) –What should be included in XML WINSIGHT too flexible Need: one reporting period at “reasonable” reporting level Example: In one case a single XML file is 300+ megabytes requiring “zipping” May require re-examining CDRL language and DID language to clarify or tighten requirement

7. For Official Use Only – Pre-Decisional 7 Significant Alignment Issues EVM-CR “contract tasks” not aligned with “effort numbers” –“Effort Number” is an internal OSD database key used for organizing reporting activities below the contract level –EVM-CR submissions at Contract or lower “Contract tasks” levels Used when multiple CPRs are required on a contract each month E.g. A “total CPR” plus multiple component CPRs –Reporting by CLINS / Delivery Orders (e.g. T-AKE, DDG, AB3, MPS, …) –Teaming relationships (e.g. V-22) Problems –Component CPRs may or may not sum to total properly –Multiple component CPRs provided without total (or total not EDI) –Typically ad-hoc methods (comment) for identifying total CPR Internal OSD/Government Alignment Issues –Contract reporting requirements need to be aligned with presentation and data needs of leadership (and IT systems) Effort numbers without matching tasks Tasks without matching effort numbers –Aligning CDRLs to EVM Policy (e.g., reducing tailoring such as elimination of EDI requirement)

8. For Official Use Only – Pre-Decisional 8Summary Why Is This Important? –Automation/integration with other data systems (AV SOA, DAMIR, MilDep, etc) provides timely access and visibility to contract reporting levels for stakeholders and decisions makers. –Data is being used by senior leadership: e.g., trip-wire analysis Alignment necessary: Reporting must role up to contract level properly No room for double counting or gaps Current Activities –Continue working with Services to coordinate corrective actions on data alignment and quality issues. –Updating EVM-CR and on-line documentation and training materials to clarify requirements –Working the bigger issues case by case Looking at CDRLs and alignment to policy Aligning data submissions to other OSD IT systems

9. For Official Use Only – Pre-Decisional 9Backups

10. 10 Reporting Status

11. For Official Use Only – Pre-Decisional 11 Reporting Status

12. For Official Use Only – Pre-Decisional 12 Reporting Status

13. For Official Use Only – Pre-Decisional 13 AV SOA Governance and Technical Approach SOA Separates Data from Application and Tools Business Tools Business Applications Web User Interfaces Users Defense Acquisition Decision Making D a t a G o v e r n a n c e Acquisition Services  Definition of key data elements  Assignment of responsibility for the authoritative copy of the specified data elements  Provision of access to governed data Governance of Data: Discoverable and Accessible Access to Authoritative Data Enterprise Services Enterprise Services Exposure Air ForceNavyDoDFederalOtherArmy Authoritative Data

14. For Official Use Only – Pre-Decisional 14 AT&L AV SOA Pilot Data Services Data brought under governance for the pilot include 140 elements in the following major categories, which correspond to the AT&L AV SOA services –EVM – EVM elements used in the Demo, plus contract elements included in DAMIR’s “Contract Data Point” and/or reported on the Contract Performance Report (CPR) –Nunn-McCurdy Unit Cost – Current estimate vs. APB (current and original) at total-appropriation level (RDT&E & Procurement), by fiscal year for comparison –Budget – Current President’s Budget and POM/BES submission, by appropriation and fiscal year, to provide a reference point for analysis –Milestone – Program milestones as agreed upon in the APB –Science & Technology – To compare Key Performance Parameters, thresholds, and objectives to current measurement and to identify critical technologies –Program Administration – To organize/view information by program, sub-program, budget activity, program element, budget line item, and/or project code

15. For Official Use Only – Pre-Decisional 15 Data SourceData Display Data Repository Army AIM Navy Dashboard Air Force SMART OSD/ARA DAMIR Displays Published DAMIR SPAWAR Charleston, South Carolina Authoritative Data Available  12 elements  Current APB  10  12  15  All of the Above # of Programs Application /Tools Used  Contracts  Cost & Funding  Performance  Schedule  Unit Cost  Track to Budget  Contract Details  Contract EVM  Nunn-McCurdy  Budget  Milestones  Science & Technology Data System Manager and Location 754th ELSG Gunter AFS Montgomery, Alabama NMCI Navy Annex Arlington, Virginia Radford Army Ammunition Plant Radford, Virginia AT&L Arlington, Virginia PA&E Arlington, Virginia Army WS AT&L WS AF WS DAMIR WS  3 S&T Elements SOA Enterprise Service Bus (ESB) PEO EIS Ft. Belvior, Virginia ASN RD&A (Management & Budget) Arlington, Virginia 754th ELSG Hanscom AFB Massachusetts AT&L Arlington, Virginia AV SOA Portal Navy WS OSD/PA&E CR PA&E Arlington, Virginia Repository Location  58 elements  EVM data  Data cleanup needed on some contracts CR WS  27 Unavailable or Static Data Army Static Source  1 S&T element  1 Admin element Navy Static Source 1 Admin element Air Force Static Source AT&L AV SOA Pilot – As of 1/8/2009 SPAWAR Charleston, South Carolina SPAWAR Charleston, South Carolina K-Scope  Contract EVM  Nunn-McCurdy  Milestones  Contracts by Location  Performance SPAWAR Charleston, South Carolina

16. For Official Use Only – Pre-Decisional 16 AV SOA Overview

17. For Official Use Only – Pre-Decisional AV SOA Methodology to aggregate Contract and EV data

18. For Official Use Only – Pre-Decisional AV SOA EV Data Findings To date USA AIM, CR, DAMIR have been integrated with AV SOA. For the pilot : – The most current EV data is often from Army and not from CR – The most current CR EV often exists in PDF format which is not consumable by AV SOA web services – Contract Effort Number is missing from CR data. It is either null or zero. – We will pull from the latest EV Source (MILDEP or CR) – Extra logic and error handling was required to map CR

19. For Official Use Only – Pre-Decisional 19 AV SOA EV Data Findings: As Of Date/Report Period End Date Discriminating Field for AV SOA. For reference: –As of – Central Repository –Reporting Period End Date – AV SOA –Report Date – MILDEP (AIM) Used to determine most current available CPR/EVM report. Report Date is initially pulled from the MILDEP contract and then used to align data from CR. Using a CPR As Of Date AV-SOA programmers select the most recent report. Within AIM we often find a Report End Date more current. Often our logic is defaulting to incorporating EV data from MILDEP. Recommend that MILDEP Report Date and CR As Of Date should be identical for a given Reporting Period.

20. For Official Use Only – Pre-Decisional 20 AV SOA EV Data Findings: Non Consumable Data CR has no error/status code that CPR data is not consumable. Only indicator is a blank file. It appears that consumable report availability is lagging within CR. Recommend CPR consumable status be reflected through CR Web Services.

21. For Official Use Only – Pre-Decisional 21 AV SOA EV Data Findings : Mismatch in Contracts from MILDEP and CR AV-SOA is obtaining an MDAP’s Contract list from the MILDEP system. Contract listing is used to query CR. For Example : MIM-104D (Patriot MEADS CAP) – PNO 531 –DAAH01-02-C-0075 (Not being reported by MILDEP) –DAAH01-03-C-0164 (Reported by both) –NAMEAD-04-C-6000 (Reported by both) MILDEP only shows 2 contracts.

22. For Official Use Only – Pre-Decisional 22 AV SOA EV Data Findings: Contract Effort Number AV SOA Definition : The effort number within the contract if multiple efforts within the contract are being tracked separately. DI MGMFT 81466A: Enter the contract number and the applicable Contract Line Item Number (CLINs). AV SOA “required” field Often in contracts without multiple efforts where no separate Contract Effort Number is delineated. AV- SOA is currently publishing a “-1”. Recommendations?

23. For Official Use Only – Pre-Decisional