1. Opinion-Based Metrics
Easy to Collect &
Powerful Drivers of Process Improvement

2. Reason for Metrics
Metrics are an important element to a Process Improvement Initiative
Agree
Somewhat Agree
Somewhat Disagree
Disagree
What went through you mind when I asked you to state your opinion?
What did we learn about this crowd from the answers?
Use cards for voting – Wave them if you Strongly Agree or Strongly Disagree. (If you wave your somewhat card, you’re being cantankerous =:^)
What went through you mind when I asked you to state your opinion?
Did you think about how you would defend your answer?
Did you want to be “right”?
Were you telling me/us what you thought you wanted us to hear?
Did you establish criteria for what “important” meant?
(Listen for other answers – put them on the board if they are interesting points to be addressed later)
What did we learn?
- Homogeneous or disparate?
Were you aligned with the crowd?
Agile Quality Systems

3. Reason for Metrics
Metrics are an important element to a Process Improvement Initiative
Why did I ask? Why was a subjective question like this sufficient?
Wanted to know if we all agreed
If so, I don’t need any more information
If not, I would need to ask further questions
Sometimes, an opinion is all you need
I don’t want to spend time defending why we should agree with this, since that isn’t the point of this session.
I assumed (hoped? Did it happen) that we all were on the Agree side, and for this session, it doesn’t matter “why” you agree. As long as you agree enough to be here and participate going further, it was good enough to see that we agreed.
Agile Quality Systems

4. Metrics Programs
My organization has good Metrics to help us in our Process Improvement Initiatives
Agree
Somewhat Agree
Somewhat Disagree
Disagree
What did you consider to form your opinion?
What did we learn about this crowd from the answers?
TRANS: So, assuming we all agree Metrics are important (or you are willing to play along a while longer)
What did you consider to form your opinion?
Do you have a definition for what “good” means?
What did we learn?
Did we see the same ratings? Either way, what can you conclude from that?
Agile Quality Systems

5. Metrics Programs
My organization has good Metrics to help us in our Process Improvement Initiatives
Why did I ask?
I want you to think about Metrics from your perspective
Wanted to know if we all agreed or differed
Perspective is key to Subjective Questions
I don’t expect us to answer this question the same because we have such different perspectives.
In our case, we “KNOW” we don’t have the same perspective because we work in different places.
But what about a team that all works in the same group, on the same project, with the same history. Will they have the same perspective?
Agile Quality Systems

6. Benefits of Process Improvement
Quality
Productivity
Predictability
Team Health
Agile Quality Systems

7. Quality Metrics What do you use to measure Quality?
What’s good about those?
What’s not so good?
Agile Quality Systems

8. Productivity Metrics What do you use to measure Productivity?
What’s good about those?
What’s not so good?
Agile Quality Systems

9. Predictability Metrics
What do you use to measure Predictability?
What’s good about those?
What’s not so good?
Agile Quality Systems

10. Team Health Metrics What do you use to measure Team Health?
What’s good about those?
What’s not so good?
Agile Quality Systems

11. Benefits of Process Improvement
Quality
Productivity
Predictability
Team Health
We are good at ___
Agree
Somewhat Agree
Somewhat Disagree
Disagree
Agile Quality Systems

12. Benefits of Process Improvement
Quality
Productivity
Predictability
Team Health
We are good at ___
What would it mean if:
We all strongly agreed we were good at it?
We all strongly disagreed?
If we all strongly agreed we were good at
Maybe we don’t a process improvement initiative
Someone who already thinks they are good at something will probably not appreciate the efforts to improve – or worse, will look at process improvement with disdain
Conversely, if we all disagreed
- We can rally around the need to improve
Agile Quality Systems

13. Power of the question: “We are good at ___”
“WE” = The entire team, not individuals
Need to scope the survey – define who the “we” is
“ARE” = Where we are right now
So we can decide whether a change is warranted
“GOOD” = What we want to be
Team owns the metric
Team owns the action to improve it
Agile Quality Systems

14. Benefits of Process Improvement
Quality
Productivity
Predictability
Team Health
My organization is good at ___
Good categories, but too broad to be actionable
Agile Quality Systems

15. Benefits Quality Productivity Predictability Team Health
Agile Quality Systems

16. Benefits Quality Customer facing Internal facing
Meets customer expectations of “good”
Does what they want / need
Bug free …
Internal facing
Reliable / stable / maintainable
Find bugs early
Confident in assessing quality
Productivity
Predictability
Team Health
Agile Quality Systems

17. Benefits Productivity Productivity = Value / Cost Value Cost
Deliver high value to customer (features / functionality)
Deliver high value to business (deliverables meeting requirements of process/phases/business)
Prioritize well …
Cost
Efficient (lean)
Fast
Right amount of rework
Quality
Productivity
Predictability
Team Health
Agile Quality Systems

18. Benefits Predictability
Predictability = Making good commitments, meeting them with reasonable adjustments
Awareness
Visibility
Feedback
Control
Reality-based estimation and planning
Manage all levers of project management
Reasonable responses to change (new needs / problems) …
Quality
Productivity
Predictability
Team Health
Agile Quality Systems

19. Benefits
Quality
Productivity
Team Health = The soft benefits that lead to more tangible ones
See my impact on the project
Accountable & Empowered
Aligning skills & interests
Enjoyment
Teamwork (intra-team, across teams) …
Predictability
Team Health
Agile Quality Systems

20. Choosing the Topics for a Subjective Survey
Defining the topics for a survey forces a definition of, or even a recognition of, what matters to you
Asking about those topics tells everyone what matters
Providing focus to process-improvement efforts
Provides motivation - “Want to move the dial”
Agile Quality Systems

21. Benefits Survey
Establishes what matters to us, what we care about being good at
Provides a baseline for improvement
Motivates and empowers us to improve
Shows whether we improved
Agile Quality Systems

22. Survey Structure For each of the Topics, two questions
“We are good at ___”
“Our practices are helping with ___”
Are we set up for success, or are we hindered by our processes / practices / behaviors / …
Adds information on another factor for determining the path for improvement if we aren’t where we want to be
Two Questions:
- First one is about the current state of our execution toward the benefit.
- Second one is about our infrastructure (Policies, procedures, quality system processes, business planning procedures, common practice), identifying whether they supportive of the benefit.
- Two questions are answered equally if our practices are consistent with our execution.
Both questions answered high means we are doing well and infrastructure supports us to stay that way.
2nd one high and 1st one low means the infrastructure allows us to be good but our execution isn’t there yet.
2nd one low and the 1st one high means that despite out inhibiting infrastructure we are finding ways to execute well.
Both low means we have an inhibiting infrastructure and are not overcoming it in execution.
Agile Quality Systems

23. Survey Structure Six-point scale for answer “Agreement” – your opinion
Strongly Disagree
Disagree
Slightly Disagree
Slightly Agree
Agree
Strongly Agree
“Agreement” – your opinion
Encourages thought to explain/defend the opinion
Forces a choice– no middle ground
Enough choices to differentiate, not too many to overwhelm or quibble
“Agree” being the key word that encourages people to form and defend THEIR opinion.
As opposed to “good / bad” which someone could interpret to require them to base their answer on facts or hearsay, and not make it “their” opinion
or “1 to 6” scales with no definition, which could present scaling / comparison problems (grade inflation, one person’s 4 is another person’s 2).
Agile Quality Systems

24. An Example - Quality
4.5
4.2
4.4
3.7
3.5
3.8
3.9
4.0
Overall Question got the highest marks. How can that be?
Agile Quality Systems

25. Productivity – Team
3.9
3.8
4.2
4.7
4.1
3.2
3.4
3.5
3.3
3.6
Agile Quality Systems

26. Productivity – Management
2.9
2.3
4.6
3.7
4.0
3.4
3.2
2.8
2.6
2.4
2.2
2.1
Agile Quality Systems

27. An Example - Predictability
3.1
3.2
3.5
3.6
3.3
2.2
2.8
2.4
4.2
3.8
Agile Quality Systems

28. An Example – Team Health
4.6
4.4
4.1
3.9
4.3
4.8
3.8
4.9
3.5
Agile Quality Systems

29. Summarization: Radar Graph
Agile Quality Systems

30. Predictability Comparison
Agile Quality Systems

31. Subjective Questions “Subjective questions are useful”
Agree
Somewhat Agree
Somewhat Disagree
Disagree
Subjective, and that’s a good thing
The opinion of the team matters
The opinion of stakeholders matters
Better if supported by objective evidence
Agile Quality Systems

32. How Do You “KNOW”? The “How Do You KNOW…” story
Do you trust the opinion of the team?
Are they qualified to have such an opinion?
If the team is to own process improvement, they must also own the opinion on the need for it
Is the team’s opinion consistent with:
Other metrics?
Reality? (Who defines “reality”?)
Stakeholders (yours, management, customers, …)?
Agile Quality Systems

33. Anonymous Surveys “Surveys must be anonymous”
Agree
Somewhat Agree
Somewhat Disagree
Disagree
Why? Why Not? (columns on whiteboard)
Surveys are made to be anonymous when:
There is a perceived “right or wrong” answer
Undesired consequence comes from answering wrong
To the survey taker, or someone they care about
Undesired action after the survey
Need to be anonymous suggests mistrust
Agile Quality Systems

34. Anonymous Surveys Surveys described here must not be anonymous
Team should be comfortable sharing honest opinions within the team and with stakeholders
Team must own their opinions
And the desire to make improvements
Want to identify & explore different perspectives
Want to evaluate changes over time
Agile Quality Systems

35. Additional Survey Questions: Interest Level, Importance
How much do you care about this topic?
5 = My reason for living, the real #1 on a priority list
4 = A top priority, but accepting there are some ahead of it
3 = This matters, but it’s not a huge thing
2 = “Would be nice to be good at this, but….”
1 = This doesn’t matter to me, but it’s okay if it matters to someone else
0 = This doesn’t matter to me, and it shouldn’t matter to anyone else either
Agile Quality Systems

36. Additional Survey Questions: Interest Level, Importance
A variation of “how much do you care” could be “how good do you want to be”
5 = I want us to be excellent at this (A Student)
4 = I want us to be good at this (B Student)
3 = I want to be good at this, but not right now
2 = There is some good and some bad in this
1 = I think this will hurt us, so if it matters to someone else, keep it away from us
0 = I don’t want us to care about this at all, it would hurt us to be good at this
Agile Quality Systems

37. Additional Survey Questions: Strength of Opinion
How strongly do you believe your opinion is right, reasonable, defendable?
5 = Adamant.
Can make a strong/complete case that would convince most. Either have strong supporting evidence, or such an easy case that you don’t need evidence. Can easily explain why, and would not be swayed by differing opinions or perspectives.
4 = Certain, but…
Strong from what you can see, but possibly incomplete. As strong as #5 from your perspective, but you accept that there could be other perspectives with differing opinions.
3 = Convinced, but hard time convincing others.
You have a strong belief, but would have a hard time explaining it to others, and would change your opinion if faced with a strong counter argument.
2 = Lots of holes in the case, but a reasonable “gut feel” about it
Would readily accept a disagreeing opinion, would be easily swayed
1 = Nothing to back it up, it’s just what I think.
0 = Don’t count my vote, my opinion is meaningless.
Agile Quality Systems

38. Summary
Data-Driven Metrics can be hard, but Subjective Surveys are easy to do
Opinions may be all you need
Discussions of the opinions has huge value
Combine Subjective Surveys with other “hard data” to tell a more complete story
Agile Quality Systems

39. The End Kelly Weyrauch Kelly@AgileQualitySystems.com 763-688-0980
Connect with me on LinkedIn
Agile Quality Systems