1. Module 12 – Measuring Success
LeanSigma® Facilitator Training
Module 12 – Measuring Success

2. Topics Measuring Success
Process performance measures………………………………………
Process capability …………………………………………………………
Program performance measures …………………………………

3. Lean Sigma has six steps to optimize processes.
Lean has a methodology (similar to DMAIC for Six Sigma). You’ll will use this methodology each time you run a kaizen or lean project.

4. Introduction We’ll look at several types of measures:
Process performance: cycle time, KPIs, DPMO and yield
Process capability: capability indices
Program performance: pace, cost savings, implementation progress
Selected process measures should follow from CTQs
We want quantitative measures of a process because they’re much more objective than qualitative or ad hoc data.
Performance data can help us to set a baseline, so we know when to intervene with more process improvement work.
Establishing Baseline Performance
At this point in the improvement process you should know what you want to accomplish (project Charter), what to measure (CTQs Operational Definitions), and that the measurement system has been validated (Measurement System Analysis). The next step is to establish a baseline to tell you where you are today. From that baseline you can quantify the gap between current performance and required performance, and develop action plans to close the gap.
The baseline performance measurement should be documented as part of the Six Sigma project - a reference point to compare improvements.
Baseline from the goals defined
Use the baseline to quantify the gap between actual and desired performance & hence action plans
Document the baseline
Baselining using yield or Sigma level measures defects or defectives received by the customer (i.e. output measures of our processes) and then drives performance upstream
Baselining of a continuous response of a process will be covered in SPC & Capability

5. Cycle time is our most common measure of process improvement.
We create an estimate of future state cycle time in kaizens based on the improvements we plan to make.
After the future state is implemented, you’ll measure the actual cycle time improvement.
This validates how much time we’ve redeployed to other work. You’ll capture this measure in the Lean Sigma PMO
Annualized Labor Hours Saved
Cycle time is generally not measured on an on-going basis.
You will receive a demo of the PMO within the next few weeks.

6. Other common process improvement measures
Reduction in lead time
May support a reduction in DAP; but often process improvements must first be implemented in other, related processes
Process improvement measures are often not measured on an on-going basis, but rather before and after process changes (2x)
Typical kaizen event measures

7. On-time delivery, query responsiveness, defect rate, throughput, etc.
Key performance indicators (KPIs) are tracked to monitor the health of the process over time.
Define
Collect
Analyze
Report
Take Action
Identify the “vital few” measures that tell the story of process performance
- Define SMART Metrics:
• Specific in that your metrics are specific and targeted to the area you are measuring
• Measurable in that you can collect data that is accurate and complete
• Actionable in that the metrics are easy-to-understand and interpret
• Relevant simply means don't measure things that are not important
• Timely metrics are those for which you can get the data when you need it
- Establish a baseline to tell you where you are today. From that baseline you can quantify the gap between current performance and required performance, and develop action plans to close the gap.
- Clarify who is responsible for the measurement system
- Measures must be actionable in order to add value
On-time delivery, query responsiveness, defect rate, throughput, etc.

8. Clear metrics definitions are important!
Each KPI must have an Operational Definition:
An exact definition of the measure
How it should be measured
Measurement frequency
Who should measure
The threshold or expected service level
Reporting: KPIs are evaluated in 30 / 60 / 90 day post-kaizen reviews.
KPIs are monitored on an on-going basis by Process Owners, who are accountable for taking the appropriate action when results fall below corresponding thresholds
Clear, documented metrics definitions are needed:
- To ensure the appropriate measure-based decisions are made
- To facilitate consistent measures across different geographic locations.
Collection: Automate, where possible
KPI results may reflect process improvements, if available, both before and after process changes are incorporated

9. Defects per unit and per million opportunities are common process measures.
Term
Definition
Defective
Any transaction that contains at least 1 defect (error)
Defect
Any part of a single transaction that has an error
Opportunity
Any part of a single transaction that can cause a defect
A transaction could have any number of opportunities contained within it, a measure of complexity
Every chance to do something either “right” or “wrong”
DPU
Defects Per Unit – Calculates the percentage of the time in decimal form how many defects occurred out of the total transactions evaluated
DPMO
Defects Per Million Opportunities – Calculates the number of times out of a million a defect would occur based upon the total opportunities for error
These measures will NOT be appropriate for all processes.
These measures may be used to assess the impact of process changes/improvements, and not for on-going process monitoring.

10. Calculating DPMO
A project team is working on improving invoice accuracy
The number of defect opportunities per invoice is 10. The team evaluated 100 invoices and discovered 35 defects
DPMO = (# of defects x 1,000,000) (# of Defect Opportunities/Unit) x # of Units
DPMO = x 1,000, x 100
“Six Sigma quality” means DPMO ≤ 3.4

11. Calculating DPMO
A delivery process of a courier company has 3 defect opportunities (timeliness, accuracy of delivery, accuracy of invoice)
The company has delivered 5000 mail pieces, 8 of them were too late, 3 of them delivered to the wrong address, and 2 had a wrong amount on the invoice
The DPMO is

12. Process Yield Concepts
LeanSigma Yield concepts are:
Throughput Yield – one process step: ratio of defects vs. defect opportunities
Rolled Throughput Yield (RTY) – multiple process steps: multiplication of Throughput Yields
Normalized Yield – multiple process steps: root of the multiplied Throughput Yields (“average process yield”)
“Traditional” Yield concepts:
First Time Yield – ratio of defectives vs. units processed
Final Yield – ratio of defectives (after rework) vs. units processed
Yield measures may be used strictly to measure process improvement

13. FTY + p(d) x 100% = 100% First Time Yield
FTY – the percentage of units that pass through an operation without any defects
If you describe the yield as 80%, the true performance of the process is not measured
FTY = 20%, therefore the percentage defective = 80%
FTY + p(d) x 100% = 100%
Percentage Defective

14. “Traditional” Yield Metrics
First Time Yield (FTY)
Final Yield (FY)
Focus on Defectives and units
No. of units: 1057
No. of defective: 58
FTY: 94.5%
Step 1
Step 2
Step 3
Step 4
Step 5
Step 6
No. of units: 1057
No. of defective: 45
FY: 95.7%
Step 1
Step 2
Step 3
Step 4
Step 5
Step 6
Rework

15. FTY Example
1056 invoices were processed in one day. 924 passed and 132 failed for various reasons
# of bad units + # of good units = Total # of units
# of bad units can be expressed as a probability of a defective, p(d): or a percentage defective: p(d) x 100% = percent defective
FTY + percent defective = 100%
FTY = 87.5%, percent defective = 12.5%

16. LeanSigma Yield Metrics
Throughput Yield (TY)
Focus on Defects and defect opportunities
No. of defect opportunities per unit: 4
No. of units processed: 268
No. of total defect opportunities (4x268): 1072
No. of defects: 34
Throughput Yield: 96.8%
Step 1

17. Rolled Throughput Yield
What is the probability of producing this product error free?
Invoice Generation
Invoice Processing
Billing
Output
TY = 97%
Op1 x
TY = 98%
Op2 x
TY = 91%
Op3
= 86.5%
86.5% probability an invoice will pass through defect-free

18. How can we establish thresholds for selected process measures?
Thresholds or “specifications” should be established for all KPIs to flag the need for adjustment to the process
Specifications are usually based on:
Historical results
Compliance/regulatory requirements
Benchmark data
Management judgment
In most cases, IMS specs are
one-sided limits defined by internal customers
Coding Accuracy: Error (defect) rate < 1%
Panel fulfillment > 90%
Specifications are often documented in Service Level Agreements (SLAs)

19. How can we determine if our process measures can routinely comply with established specs?
Process capability is the inherent ability of a process to meet the specifications
The performance of the process when it is stable and predictable, or operating in a state of statistical control, i.e., the mean and the variation in the process are not shifting
Assumes no “special cause” variation, e.g., a “market event”, or a hurricane and resulting production delays. Verify this!
Process capability measures provide insight as to whether the process has:
A centering issue relative to the specifications
A variation issue
A combination of centering and variation issues
Inappropriate specifications
Process capability measures provide a baseline assessment
No special (or, “assignable”) cause variation: Can often be verified by generating and assessing plots of the data and by statistical tests for normality. Common Cause: Variation caused by unknown factors resulting in a steady but random distribution of output around the average of the data. It is the variation left over after Special Cause variation has been removed and typically (not always) follows a Normal Distribution.

20. When specifications cannot be changed, actions must be taken to improve process capability
Incapable Process:
- This scenario occurs often in the six sigma projects; it’s more difficult to resolve than a centering issue
- In this scenario, the process is centered between the specifications at the mean; however, process variability is high
- Resolution: Identify the variables that will reduce variability in the process outputs and take the necessary action to modify them
- Example: Data Investigation (DI) TATs are inconsistent. Variables that impact DI TATs may include availability of the mainframe or UNIX environments, training of investigators, etc.
Off-Target Process:
- In this scenario, process variability is relatively low, but the process consistently shows too many defects
- Resolution: Identify the variables that will shift the process over to the mean
- If we cannot change the specs, this is usually the easiest option to improve process capability
LSL = Lower Spec Limit
USL = Upper Spec Limit
Target is usually the midpoint between the specs

21. Standard deviation describes how far results in a process vary from the mean result.
Processes with higher standard deviations have more variation.
They are less predictable and less stable than healthy processes.
Processes with less variation will have a higher “sigma level” and fewer defects.
Variation
(Dispersion)
Normal
Distribution
(“Bell”)
Curve
The bell curve is defined / characterized by the mean and standard deviation
Standard
Deviation =

22. We assess process capability using process capability indices.
Oct 2010
We assess process capability using process capability indices.
Cp = =
What is possible if your process is perfectly centered
The process potential
Cpk = minimum ( , )
The reality of your process performance
How the process is actually running relative to the specifications
The “k” is the penalty for being off-center
specification width
process width
(USL – LSL) 6s
(USL – X) 3s
(X - LSL) 3s
(c) IMS Health Confidential Six Sigma Green Belt Certification

23. Oct 2010
The “sigma level” of the process provides another measure of process capability.
σlevel = minimum ( , )
Number of standard deviations between the center of the process and the nearest specification
“Six-sigma quality” means:
Cp ≥ 2.0
Cpk ≥ 1.5
σlevel ≥ 4.5
(USL – X) s
(X - LSL) s
Sigma level at 4.5 allows for process drift
Process capability should be periodically reassessed, especially following the implementation of process improvements.
(c) IMS Health Confidential Six Sigma Green Belt Certification 23

24. Program metrics help us monitor the health and benefits of the LeanSigma program.
Are we achieving our mission to deliver more value to our customers and stakeholders by identifying and implementing measureable process improvements?
Regular Lean governance meetings help us:
Align on priorities and targets
Validate results meet expectations
Ensure support
Mitigate risks
Costs:
Lean facilitator & SME time
Travel $
Implementation costs
Benefits:
Enable IMS to deliver on strategic priorities
Transform the culture
LeanSigma Mission:
Pursue process opportunities that directly enable IMS to deliver on strategic priorities (transformation, revenue, EBIDTA, and sales excellence).
Transform the culture so employees see opportunities for continuous improvement in everything they do.
Governance meetings: With the lean council (Jim Berkshire, Paul Thomson, lean leads), with GTSO management, with Ari and his team.

25. Lean program reporting summarizes results from all Lean events.
Cadence
Pace of events by month, department & geography
Benefits
Net EBIDTA savings
Annualized hours saved (capacity redeployed, FTE savings)
Percent improvement cycle / lead time, delays, etc.
Highlights of exceptional events, case studies
Implementation
Percent of identified savings removed from budget
Percent of kaizen action items completed
Focus areas
Prioritization of processes based on exec sponsor discussions
Pipeline of Lean events
Cultural entrenchment
Number of employees trained / certified
Number of employees participating in kaizens
Communications
95% of reporting is enabled by web queries against the Lean PMO

26. Annualized Labor Hours Saved
Sample Dashboard
U.S. Events by Month
Process Focus to Date
n= 110
completed events
planned
2011
2012
80.23% of all U.S. kaizen action items completed
2012 U.S. EBIDTA Contribution: $817K
*100% of savings realized
Annualized Labor Hours Saved
= 31,086

27. Cost Savings Detail Example

28. Qualitative Reporting Sample

29. Case Study: U.S. Market Research Client Support
February 2011
Sponsor: Manny Triggiano
Process Owner: Brenda Capobianco
Lean Process Lead: Carissa Waltenbaugh
Team: Kevin Downing, Chris Davies, Cathy Pileggi-Jefferson, Katrina Kulp, Kyle Burke, Michele Agnoli, Joy Cross, Gerard Avillo, Lisa Cooney, Jeff Northern
Key Kaizen Findings
90 Days After the Kaizen
Issues Noted
Process Changes
A pilot was completed to test the central support pool for smaller clients.
The centralized model went live June 13, 2011.
QMS was enforced as a central information system.
Tier 1 (simple) and 2 (complex) resource pools were created.
Checklists and decision trees were created to clarify info needs.
Clients are assigned a CSA even if they call only a few times a year.
Pool CSA resources so questions are answered by the first available CSA.
Some CSAs less able to handle complex questions.
Triage client questions and assign CSA resources to a Level 1 pool (simple) or Level 2 pool (complex).
CSAs sometimes call multiple times to refine the client question.
Create a checklist to guide CSAs when refining details of a client’s question.
Create a decision tree to identify what research is required for a particular question. Improve how info is organized on ARC.
CSAs sometimes do unnecessary research.

30. Case Study: U.S. Market Research Support Findings 15 months after implementation
Results:
The central service center remains in place successfully.
Clients talk to CSAs directly more often.
Very few clients have questioned the change.
Three positions were eliminated.
In addition, the team took on additional work without adding headcount. (SMART roll-out).
Cycle time savings were estimated at 65% initially. The savings observed are about 50%.
Critical factors made the change possible:
Pilot to test client feedback
Excellent communication to the team and clients about changes
Extreme organization and focus

31. Focus areas – Completed Events
MMS Screening & Validation
MAS RFP Response mini kaizen
Billing Initiation Process
MAS Custom Project Process
Performance Analytics Deployment Standardization
Opportunity Management / CRM Improvement
SAS Incentive Compensation Implementation Process
Reduce time spent screening & validating data
Improve quality & consistency of RFP responses
Reduce admin burden for GTM resources
Improve process workflows for custom deliverables
Reduce time to implement
Reduce admin burden
Improve the process for IC implement-ations
Objectives
MAR 6-8
APR 16
MAY 1-3
MAY 15-17
MAY 15-17
JUN 11-12
JULY
31% reduction in cycle time
Process improve-ments imple-mented
Identified 3 improve-ment work streams
Results to be rolled out at July 10 report out
Delivery time reduced by 25%
Eliminated time spent entering detail into early stage opps
Results

32. Process Priorities – Sample from Operations
On an annual basis, lean leads and functional area management identify focus areas within each functional area.

33. Evidence of Cultural Entrenchment
Leadership
Lean continues to be a high priority for Seyed and team
Lean Events
U.S. has held more Lean events than any other country (110 events since Feb 2011)
30 additional events are in the pipeline
Skill Development
More than half of all certified Lean facilitators are in the US
19 certified and 48 in progress
Results
Over 25% of Lean’s global savings come from the U.S.
Implementation is strong (>80% kaizen action items complete)
Overview of the program and key projects including timeline and status
Tracking metrics e.g. savings, efficiency improvements
Our plan to go back and audit after the event was implemented (we may want to show at next ELG)

34. Lean Measurement & Reporting Tools
Lean Event Executive Summary Template:
Guidelines for 30 / 60 / 90 Day Reviews
Lean PMO
PMO Users Guide:
Project Statistics (PMO Web Query):
Guidance on standard metrics and metrics-monitoring tools
Under development. Contact Tamra Kirkpatrick.

35. US & Canada: ISO Compliance
In the US and Canada, the Kaizen management process is maintained in compliance with ISO standards
Facilitators are required to comply with the Kaizen Management procedure
The procedure is written to provide a great deal of flexibility in selecting, planning and conducting kaizen events
Process effectiveness is assessed:
Results are reviewed in quarterly internal management reviews, and periodically, by external ISO auditors
This slide is N/A to countries outside of the US & Canada.
Reference to lean kaizen events are made in process documentation for other US and Canada processes; hence the need to bring our kaizen management procedure under ISO as well.
A key benefit of ISO compliance: It requires on-going training of lean facilitators (training records must be maintained).
Number of kaizen events per quarter, based on a rolling four quarters ≥ 15