1. Why Our Approach To Acquisition Reporting Is All Wrong
Blaise J. Durante Deputy Assistant SecAF (Acquisition Integration)
31 Oct 07

2. How to Interpret Industry’s Comments
10. GIVE US YOUR INTERPRETATION
I can’t wait to hear this bull!
9. LOW MAINTENANCE
Impossible to fix if broken
8. THE ENTIRE CONCEPT WILL HAVE TO BE ABANDONED
The only person who understood the thing quit
7. TEST RESULTS WERE EXTREMELY GRATIFYING
We are so surprised that the stupid thing works
6. MAJOR TECHNOLOGICAL BREAKTHROUGH
It works OK, but looks very high-tech

3. How to Interpret Industry’s Comments
5. ALL NEW
Parts not interchangeable with the previous design
4. RUGGED
Too damn heavy to lift?
3. CUSTOMER SATISFACTION IS DELIVERED ASSURED
We’re so far behind the customer will be happy with whatever we deliver
2. WE WILL LOOK INTO IT
Forget it! We have enough problems for now
1. CLOSE PROJECT COORDINATION
We know who to blame

4. The Familiar Problem Everything is running smoothly Program reports:
No major risks
No major issues
Green ratings on cost / schedule / performance
Then … BAM … a serious problem
No notice … No warning
Despite large investments in reporting systems
What happened ??
In acquisition we've been haunted by a familiar problem
Programs report problems with little or no advance notice
…and the mad scramble begins to pick up the pieces
It's a condition we've had since I started in acquisition many years ago
…despite huge improvements in the speed and reliability of data transfer
Why is this?

5. Current Reporting DAES reporting is useful for notification
But it doesn’t predict
We ask the PM to look ahead for potential issues
But it’s dependent on PM intuition
It’s only as forward looking as the PM
It’s not inherently transparent
SARs and DAES are rear-view looks at program performance
What we see in DAES is useful for notifying us of problems
…but it doesn't do much to help predict them
DAES reports require the PM to look ahead for potential issues
…but still rely on PM intuition and problem recognition to be of any value in anticipating problems
They are only as forward looking and transparent as the PMs filling them out
SARs and APB reports are totally "rear-view" looks at program performance
Neither have any traction until after the event that triggered the breach has already occurred
While we need an oversight function, of course, we should be able to do better than just be in a mode of continual reaction

6. Why Is This ? We ask for the wrong data
Because we focus on symptoms (like cost overruns and schedule delays)
We have a poor understanding of causal factors
We often value “more data”
But data for data sake is useless
We often value bigger data systems
But garbage in – garbage out
Two main factors conspire to give us our present reporting dilemma
1) We ask for the wrong data. We focus on symptoms of problems (like cost overruns and schedule delays)
2) We have a poor understanding of causal factors that drive underlying problems
It’s easy to fall into the mode of wanting more data
…and seeking out the data that’s available to report
But data for data’s sake is useless
We need to seek information that will give us insight on the important issues
And it’s easy to focus on more and better information systems
But without the right information we just have garbage in–garbage out

7. We Do Know A Few Things
We know a lot of program problems are caused by the introduction of immature technology
A number of studies show this is a problem
Yet in our routine reporting we don't examine
Maturation of our technology
Whether those are good choices
Complexity of integration
Instead we focus on
The schedule slip caused by test failures
How to recover
But we do know a few things
For example, we know that a lot of program problems are caused by attempting to introduce immature technology into a platform
There are a number of studies that show this is a problem
Yet in our routine reporting we don't examine the maturation of our technology and whether we’ve made good technology choices
Instead we focus on the schedule slipped caused by failed tests
…and how to recover
The result of these two factors together is like the proverbial drunk hunting for his car keys under the street lamp
"Is that where you lost your keys?" the bystander asks
"No, but the light is so much better over here," the drunk replies

8. We Do Know A Few Things
We know that once a program’s schedule variance starts to go south, cost will soon follow
A consistent pattern in EVM
About a 3-6 month lag
Yet optimism reigns supreme
“We’ve got a pretty good handle on it, and it shouldn’t slip any further”
Not a lack of integrity, but a naïve optimism

9. We Need To Remember We should prefer leading over trailing indicators
Poor government and contractor baselines make reporting ineffective
Rebaselining doesn’t solve problems
There is a difference between leading and trailing indicators
Leading indicators will give us advance warning
…and an opportunity for corrective action
Asking someone to project a lagging indicator does not make it a leading indicator
We suffer because of poor baselines
Without a well developed baseline, we can’t tell if a program is proceeding as it should
And … rebaselining makes reports look better, but it doesn’t solve the underlying problem

10. The Key Take-Away We need a process focus The process focus is key to:
We accomplish our work by executing process
Oversight needs to support the acquisition process
DAMIR needs to support the process
The process focus is key to:
Providing the right enabling processes
Providing the right enabling tools
Knowing the right metrics
We really need to have a process focus
A process focus is fundamentally important to doing this right
DAMIR needs to support the acquisition process
…so we need a clear view of that process to get DAMIR requirements right
DAMIR will work effectively only as an element of a well-designed process
Remember that metrics will drive behavior
So we really need to decide what behavior we want,
We understand that when we clearly understand the process
So the process focus is key to knowing the right metrics
And process is key to putting this in place and having it work in real life

11. Reasons for Hope
The Army's P(s) and the Air Force's PoPS are new attempts to report predictively
Currently dependent on subjective assessments
But are serious attempts to begin to anticipate
Long term: good, objective leading indicators
There are studies that examine relationships of program aspects and outcomes
Not widely read or publicized
Could lead to new measures based on established research
Army P-sub-S and Air Force PoPS are new attempts to report predictively
They are currently dependent on subjective assessments because it's mostly all we have
But they represent deliberate effort to anticipate future problems
There is a smattering of studies out that examine relationships between various program characteristics and program outcomes
These would be the leading indicators we’re looking for
These studies aren't widely read. But there is good work being done that could help us if we seriously undertake to develop better program metrics
What we need are new measures based on established research
These may not replace our current measures. (SARs, for example, are congressionally mandated), but may be in addition to them
I'm trying to do my part
For example, I just had RAND wrap up a study on cost overruns to draw conclusions on what drives them
And I know the Army has begun a validation study on their Probability of Success reporting system to see how well it has performed in anticipating problems ahead of time

12. Summary
We invest a lot of time and money in reporting
But we don’t get the bang for the buck
We need to take a process view of acquisition
… and how acquisition reporting supports it
We need research on cause & effect within programs
We need to move to more predictive measures of program health
We need metrics that mean something and drive behavior
We can do this – we need to do this
Closing
Einstein is purported to have said that insanity is doing the same thing over and over again and expecting different results
Yet with few exceptions we are reporting the same things we always have
…and can't understand how we continue to get surprised
It's a big challenge
But we need to step back from a preoccupation with the data systems
…and take a fresh look at the data that we report and review
It takes courage to change something that we've grown comfortable with over the years
But the fact is oversight in this era of increased stakeholder scrutiny needs to be better than it is
After all, we don't want to be accused of being insane.
Consider what data we need to make decisions before developing reporting systems!!!

13. Top 10 Physician Notes Actual notes from patient records!
10. By the time he was admitted, his rapid heart had stopped, and he was feeling better.
9. Patient has chest pain if she lies on her left side for over a year.
8. On the second day the knee was better and on the third day it had completely disappeared.
7. The patient has been depressed ever since she began seeing me in 1983.
6. The patient is tearful and crying constantly. She also appears to be depressed.
5. Discharge status: Alive but without permission.
4. Healthy-appearing decrepit sixty-nine-year-old male, mentally alert but forgetful.
3. The patient refused an autopsy.
2. She was numb from her toes down.
1. The patient has no past history of suicides.
By the time he was admitted, his rapid heart had stopped, and he was feeling better.
Patient has chest pain if she lies on her left side for over a year.
On the second day the knee was better and on the third day it had completely disappeared.
The patient has been depressed ever since she began seeing me in 1983.
The patient is tearful and crying constantly. She also appears to be depressed.
Discharge status: Alive but without permission.
Healthy-appearing decrepit sixty-nine-year-old male, mentally alert but forgetful.
The patient refused an autopsy.
The patient has no past history of suicides.
The patient expired on the floor uneventfully.
Patient has left his white blood cells at another hospital.
The patient's past medical history has been remarkably insignificant, with only a forty-pound weight gain in the past three days.
She slipped on the ice and apparently her legs went in separate directions in early December.
Between you and me, we ought to be able to get this lady pregnant.
The patient was in his usual state of good health until his airplane ran out of gas and crashed.
She is numb from her toes down.
The skin was moist and dry.
Patient was alert and unresponsive.
When she fainted, her eyes rolled around the room.