Why Our Approach To Acquisition Reporting Is All Wrong Blaise J. Durante Deputy Assistant SecAF (Acquisition Integration) 31 Oct 07

How to Interpret Industry’s Comments 10. GIVE US YOUR INTERPRETATION I can’t wait to hear this bull! 9. LOW MAINTENANCE Impossible to fix if broken 8. THE ENTIRE CONCEPT WILL HAVE TO BE ABANDONED The only person who understood the thing quit 7. TEST RESULTS WERE EXTREMELY GRATIFYING We are so surprised that the stupid thing works 6. MAJOR TECHNOLOGICAL BREAKTHROUGH It works OK, but looks very high-tech

How to Interpret Industry’s Comments 5. ALL NEW Parts not interchangeable with the previous design 4. RUGGED Too damn heavy to lift? 3. CUSTOMER SATISFACTION IS DELIVERED ASSURED We’re so far behind the customer will be happy with whatever we deliver 2. WE WILL LOOK INTO IT Forget it! We have enough problems for now 1. CLOSE PROJECT COORDINATION We know who to blame

The Familiar Problem Everything is running smoothly Program reports: No major risks No major issues Green ratings on cost / schedule / performance Then … BAM … a serious problem No notice … No warning Despite large investments in reporting systems What happened ?? In acquisition we've been haunted by a familiar problem Programs report problems with little or no advance notice …and the mad scramble begins to pick up the pieces It's a condition we've had since I started in acquisition many years ago …despite huge improvements in the speed and reliability of data transfer Why is this?

Current Reporting DAES reporting is useful for notification But it doesn’t predict We ask the PM to look ahead for potential issues But it’s dependent on PM intuition It’s only as forward looking as the PM It’s not inherently transparent SARs and DAES are rear-view looks at program performance What we see in DAES is useful for notifying us of problems …but it doesn't do much to help predict them DAES reports require the PM to look ahead for potential issues …but still rely on PM intuition and problem recognition to be of any value in anticipating problems They are only as forward looking and transparent as the PMs filling them out SARs and APB reports are totally "rear-view" looks at program performance Neither have any traction until after the event that triggered the breach has already occurred While we need an oversight function, of course, we should be able to do better than just be in a mode of continual reaction

Why Is This ? We ask for the wrong data Because we focus on symptoms (like cost overruns and schedule delays) We have a poor understanding of causal factors We often value “more data” But data for data sake is useless We often value bigger data systems But garbage in – garbage out Two main factors conspire to give us our present reporting dilemma 1) We ask for the wrong data. We focus on symptoms of problems (like cost overruns and schedule delays) 2) We have a poor understanding of causal factors that drive underlying problems It’s easy to fall into the mode of wanting more data …and seeking out the data that’s available to report But data for data’s sake is useless We need to seek information that will give us insight on the important issues And it’s easy to focus on more and better information systems But without the right information we just have garbage in–garbage out

We Do Know A Few Things We know a lot of program problems are caused by the introduction of immature technology A number of studies show this is a problem Yet in our routine reporting we don't examine Maturation of our technology Whether those are good choices Complexity of integration Instead we focus on The schedule slip caused by test failures How to recover But we do know a few things For example, we know that a lot of program problems are caused by attempting to introduce immature technology into a platform There are a number of studies that show this is a problem Yet in our routine reporting we don't examine the maturation of our technology and whether we’ve made good technology choices Instead we focus on the schedule slipped caused by failed tests …and how to recover The result of these two factors together is like the proverbial drunk hunting for his car keys under the street lamp "Is that where you lost your keys?" the bystander asks "No, but the light is so much better over here," the drunk replies

We Do Know A Few Things We know that once a program’s schedule variance starts to go south, cost will soon follow A consistent pattern in EVM About a 3-6 month lag Yet optimism reigns supreme “We’ve got a pretty good handle on it, and it shouldn’t slip any further” Not a lack of integrity, but a naïve optimism

We Need To Remember We should prefer leading over trailing indicators Poor government and contractor baselines make reporting ineffective Rebaselining doesn’t solve problems There is a difference between leading and trailing indicators Leading indicators will give us advance warning …and an opportunity for corrective action Asking someone to project a lagging indicator does not make it a leading indicator We suffer because of poor baselines Without a well developed baseline, we can’t tell if a program is proceeding as it should And … rebaselining makes reports look better, but it doesn’t solve the underlying problem

The Key Take-Away We need a process focus The process focus is key to: We accomplish our work by executing process Oversight needs to support the acquisition process DAMIR needs to support the process The process focus is key to: Providing the right enabling processes Providing the right enabling tools Knowing the right metrics We really need to have a process focus A process focus is fundamentally important to doing this right DAMIR needs to support the acquisition process …so we need a clear view of that process to get DAMIR requirements right DAMIR will work effectively only as an element of a well-designed process Remember that metrics will drive behavior So we really need to decide what behavior we want, We understand that when we clearly understand the process So the process focus is key to knowing the right metrics And process is key to putting this in place and having it work in real life

Reasons for Hope The Army's P(s) and the Air Force's PoPS are new attempts to report predictively Currently dependent on subjective assessments But are serious attempts to begin to anticipate Long term: good, objective leading indicators There are studies that examine relationships of program aspects and outcomes Not widely read or publicized Could lead to new measures based on established research Army P-sub-S and Air Force PoPS are new attempts to report predictively They are currently dependent on subjective assessments because it's mostly all we have But they represent deliberate effort to anticipate future problems There is a smattering of studies out that examine relationships between various program characteristics and program outcomes These would be the leading indicators we’re looking for These studies aren't widely read. But there is good work being done that could help us if we seriously undertake to develop better program metrics What we need are new measures based on established research These may not replace our current measures. (SARs, for example, are congressionally mandated), but may be in addition to them I'm trying to do my part For example, I just had RAND wrap up a study on cost overruns to draw conclusions on what drives them And I know the Army has begun a validation study on their Probability of Success reporting system to see how well it has performed in anticipating problems ahead of time

Summary We invest a lot of time and money in reporting But we don’t get the bang for the buck We need to take a process view of acquisition … and how acquisition reporting supports it We need research on cause & effect within programs We need to move to more predictive measures of program health We need metrics that mean something and drive behavior We can do this – we need to do this Closing Einstein is purported to have said that insanity is doing the same thing over and over again and expecting different results Yet with few exceptions we are reporting the same things we always have …and can't understand how we continue to get surprised It's a big challenge But we need to step back from a preoccupation with the data systems …and take a fresh look at the data that we report and review It takes courage to change something that we've grown comfortable with over the years But the fact is oversight in this era of increased stakeholder scrutiny needs to be better than it is After all, we don't want to be accused of being insane. Consider what data we need to make decisions before developing reporting systems!!!

Top 10 Physician Notes Actual notes from patient records! 10. By the time he was admitted, his rapid heart had stopped, and he was feeling better. 9. Patient has chest pain if she lies on her left side for over a year. 8. On the second day the knee was better and on the third day it had completely disappeared. 7. The patient has been depressed ever since she began seeing me in 1983. 6. The patient is tearful and crying constantly. She also appears to be depressed. 5. Discharge status: Alive but without permission. 4. Healthy-appearing decrepit sixty-nine-year-old male, mentally alert but forgetful. 3. The patient refused an autopsy. 2. She was numb from her toes down. 1. The patient has no past history of suicides. By the time he was admitted, his rapid heart had stopped, and he was feeling better. Patient has chest pain if she lies on her left side for over a year. On the second day the knee was better and on the third day it had completely disappeared. The patient has been depressed ever since she began seeing me in 1983. The patient is tearful and crying constantly. She also appears to be depressed. Discharge status: Alive but without permission. Healthy-appearing decrepit sixty-nine-year-old male, mentally alert but forgetful. The patient refused an autopsy. The patient has no past history of suicides. The patient expired on the floor uneventfully. Patient has left his white blood cells at another hospital. The patient's past medical history has been remarkably insignificant, with only a forty-pound weight gain in the past three days. She slipped on the ice and apparently her legs went in separate directions in early December. Between you and me, we ought to be able to get this lady pregnant. The patient was in his usual state of good health until his airplane ran out of gas and crashed. She is numb from her toes down. The skin was moist and dry. Patient was alert and unresponsive. When she fainted, her eyes rolled around the room.