Improving ERP Cost Estimating Presented by: Mr. Wilson Rosa Air Force Cost Analysis Agency Mr. David Cashin Naval Center for Cost Analysis Mr. Noel Bishop Office of the Deputy Assistant Secretary of the Army for Cost & Economics

Topic Speaker Introduction Mr. Wilson Rosa Estimating Challenges Way Ahead Mr. David Cashin Lessons Learned Mr. Noel Bishop

Objective Share challenges and lessons learned from five recent milestone reviews Describe a plan to improve ERP cost estimating

Introduction Procuring ERP is becoming increasingly popular within DoD

5 Programs Deployed (2000-2005) Total Investment Cost ~$1.0B (TY) # of Programs By The Way, All slide are in TY budget $ As of today there has been five pure ERP’s deployed. Added together these programs add up to approximately one billion dollars These programs were built to budget constraints Therefore our data and experience is limited Total Investment Cost ~$1.0B (TY)

17 Programs in Development (Present) # of Programs As you can see there is growing number of programs who employ ERP. These programs are full scale ERP programs with growing scope and higher costs This is why we need to prepare now, to estimate programs with improved methods Total Investment Cost >$6.0B (TY)

How ERP differs from Traditional Software Intensive Programs? Factor Traditional Software Programs ERP Technology Custom-Built Configured Commercial Off the Shelf (COTS) software Interoperability Standalone IT systems Interconnected IT systems Design Concept System Analysis and Design Business Process Re-engineering, Product Configuration Primary Cost Driver 1 Parameter (Lines of Code) Over 30 Parameters (e.g. scripts, reports, interfaces) DoD Experience +35 Years, Over 7,000 AIS Deployed 4 Years, 5 ERP Deployed 17 ERP In-development Why improved, because ERP is a process development not a product development. Let me explain…

Main Challenges with ERP Estimating Understanding Impact of Cost Drivers Software development represents only a small fraction of total cost No empirical research on specific ERP cost drivers Limited historical data to support parametric tools Because of these differences We have four main challenges 1. Brain ticklers - projets

Challenge # 1: Understanding Impact of Cost Drivers More than 30 cost parameters Still learning which are major cost drivers Currently, analysts can quantitatively estimate only 6 out of 30 Subject matter experts are unable to measure size or impact of 24 out of 30 Put 30 parameters in back up… The six parameters are similar to current estimating practices for example SLOC, help desk needs, H/W Since ERPs are based on process and not products, current recognized experts are at a lost to measure things such as Business Logic Blueprinting actives such as interviewing experts Retraining of personnel

Challenge # 2: Software Development is not the primary cost driver It is not just the parameters that differ greatly. The big difference is that an ERP includes a significant amount of non-system design task For example = Change Management: Changing the culture of your community to fit new processes Blue Printing: Understanding the current environment though mapping business logic and transactions The sheer head count of users can drive training, software licenses, and hardware needs and in some cases dwarf software development altogether Traditional Software Program ERP

Challenge # 3: No empirical research on specific ERP cost drivers No journal articles or dissertations address cost drivers 0 out of 543 ERP Articles 0 out of 56 ERP Dissertations Few “white papers” found on cost drivers but none provide supporting data Parametric tools are just now under development Did I mention that there is little to no knowledge of ERP. You can read the statistics. There is only 5-6 years of ERP history in the commercial world and much less in Government circles Another reason for the derth of research may be because ERPs are process driven and seamingly immeasurable

Challenge # 4: Limited historical data to support parametric tools Five previous ERP programs Poor EVM reporting from all five Cost presented at total level only None reporting into DCARC Inconsistent definitions/parameters Enterprise Software Initiative website provides good pricing metrics but not based on actual data You saw earlier that we have 17 programs in development but only 5 fielded programs. How are we to assess these 17 programs? Can we reach back to 1980s technology for analogy ? Can we rely on the contractors estimates ? Can we focus on SLOC for all CERs ? Say yes to any of these and

Consequence: Significant Cost Growth Program Cost Growth* 1 >5% (MS-A compared to MS-B) 2 50% (MS-B compared to IOC) 3 100% (MS-B compared to Preliminary Design Review) 4 350% (MS-A compared to Critical Design Review) 5 400% (MS-B compared to IOC) Here are the consequences. Our inability to get our hands around the cost of these programs is unacceptable. We are not sitting on our hands. Dave Cashin is going to talk about the road ahead for cost agencies *Preliminary data from 5 of 17 on-going efforts

Road Ahead

Identify Cost Drivers and Metrics Establish methodology for utilizing size metrics Establish a single data set supporting these metrics Create standard definitions and map between various industry terms The cost agencies are already working together on AIS databases. When we saw ERPs coming down the road we looked to collect data here as well We need to come up with definitions to map to all industry terms because they are all different but the same

Data Collection Efforts Ensure cost data reporting requirements are included in ERP contracts Improved CSDR and EVM reporting tailored for ERP programs Create single database across DoD: Air Force Navy Army and Marine Corps Army, Air Force, and Navy Cost Centers are working together to build a database for AIS programs starting with ERP’s

Standardize Cost Element Structures Survey industry to develop a single cost element structure Map between industry work structures and DoD CES structures Need to work with PA&E to amend policies/handbooks to address ERP specific guidelines include some sort of standard skills needed set for different activities

Develop new ERP parametric tools Establish a foothold in new data sets Standardize language and translation guides to bridge the gap Develop analogies, metrics, and parametric tools Build rigorous approach to Risk analysis

Note: ERP definitions not standardized Lessons Learned Program Office must clearly define target processes Blueprinting and Realization schedule RICE Objects by some level of complexity Business Process Master Lists by some level of complexity Legacy system integrations Requirements, (e.g., SW licenses, hardware and satellite time) Training and Fielding plans Note: ERP definitions not standardized

Lessons Learned Calibrate existing models with valid data Verify CARD metrics with other official documents Operational Requirements Document Interface Control Document GAP Analysis Process Flow Diagrams Product Assessment C4ISR Plan Business Process Master List Technical and Functional Requirements

Lessons Learned Add cost growth factor to estimate based on DoD history—fact, not risk Program Office needs ERP Subject Matter Experts on hand Ensure Change Management, Data Cleansing and Normalization, and Data Warehousing are adequately addressed

Questions? 30 metric list - definitions Expectation of completion date - ESI contracting toolset for crosscheck information and task lists - Survey results late summer - Dictionary by Fall - Data collection ongoing into winter - Some Service Data - Little federal data - Commercial Data

Acronyms AIS Automated Information System BSM Business System Modernization CES Cost Element Structure CARD Cost Analysis Requirements Description COTS Commercial Off The Shelf CSDR Cost and Software Data Reporting C4ISR Command, Control, Communication, Computer, Intelligence, Surveillance, and Reconnaissance DoD Department of Defense DCARC Defense Cost and Resource Center ECSS Expeditionary Combat Support System ERP Enterprise Resource Planning EVM Earn Value Management IOC Initial Operational Capability

Acronyms IT Information Technology MS-A Milestone A Decision Review MS-B Milestone B Decision Review OSD Office of the Secretary of Defense PA&E Program Analysis & Evaluation PMO Program Management Office RICE Report, Interface, Conversion, Extension SW Software