Choosing Performance Indicators Savannah River Nuclear Solutions, LLC

Slides:



Advertisements
Similar presentations
Quality control tools
Advertisements

Project leaders will keep track of team progress using an A3 Report.
Post Incident/Injury Response Presented by:. Purpose To ensure management/supervision responds appropriately and with confidence in the event of an incident.
Computer Engineering 203 R Smith Project Tracking 12/ Project Tracking Why do we want to track a project? What is the projects MOV? – Why is tracking.
Quality Management Philosophies
Chapter 10 Quality Control McGraw-Hill/Irwin
Ensuring Quality and Productivity If you forget the customer, nothing much else matters. —Anne Mulcahy, CEO, Xerox Corporation Chapter 2 Copyright © 2010.
Bureau of Workers’ Comp PA Training for Health & Safety (PATHS)
Effectively applying ISO9001:2000 clauses 5 and 8
© 2007 Breakthrough Systems1 Deming’s Red Bead Experiment AICE Quality Conference Tuesday, 06MAR07 Jim Clauson Breakthrough Systems
THE MANAGEMENT AND CONTROL OF QUALITY, 5e, © 2002 South-Western/Thomson Learning TM 1 Chapter 8 Performance Measurement and Strategic Information Management.
Chapter 16 Problem Solving and Decision Making. Objectives After reading the chapter and reviewing the materials presented the students will be able to:
IT Risks and Controls Revised on Content Internal Control  What is internal control?  Objectives of internal controls  Types of internal controls.
Information, Analysis, and Knowledge Management in the Baldrige Criteria Examines how an organization selects, gathers, analyzes, manages, and improves.
Chapter 6: THE EIGHT STEP PROCESS FOCUS: This chapter provides a description of the application of customer-driven project management.
ISM Best Practices Aurora, CO Dave Jackson September 2006 Dashboards and Control Charts.
HSA 171 CAR. 1436/5/10 3  Concept Of Controlling.  Definition.  Controlling Process. 4.
| +44(0) © ICE LTD 2009 All rights reserved. August 2009 version 1.3 Systems Thinking Facilitators.
Quality Control Copyright © 2015 McGraw-Hill Education. All rights reserved. No reproduction or distribution without the prior written consent of McGraw-Hill.
Lies, Damned Lies, and Statistics Hanford Performance Indicator Forum December 11, 2003 Steven S Prevette Fluor Hanford Safety and Health
Safety Management Across Large Organizations The Meeks Lumber Way.
6 6 Controlling Describe the control process Contrast two types of corrective action Compare preventive, concurrent, and corrective.
Ensuring High Quality and Productivity
Getting Started with Flow
By Prof. Dr. Salahuddin Khan
Best Practices in using a Balanced Scorecard
Implementing Quality Concepts
Gauging Your Safety Culture
Employability Skills.
WRITING A WINNING STATEMENT OF PURPOSE (Project Statement)
Six-Sigma : DMAIC Cycle & Application
Organisation Control KPI’s & an industry Review
Address the complexity in a Structured way
The Office Procedures and Technology
Steven S Prevette ASQ Certified Quality Engineer Fluor Hanford
What is performance management?
HOW TO BE A GOOD STUDENT 101.
Overview – Guide to Developing Safety Improvement Plan
Overview – Guide to Developing Safety Improvement Plan
Quality Management Six Sigma
The Scientific Method: How to solve just about anything
Title: Owner: Ver: Date:
Title: Owner: Ver: Date:
Here are some top tips to help you bake responsible data into your project design:.
in Construction Industry
KPI Familiarisation
Objectives 1. A definition of planning and an understanding of the purposes of planning 2. Insights into how the major steps of the planning process are.
The Scientific Method.
DMAIC Roadmap DMAIC methodology is central to Six Sigma process improvement projects. Each phase provides a problem solving process where-by specific tools.
Leading with Leading Indicators
Building and Sustaining Total Quality Organizations
Presentation to - Management Team Javier Garza, HRM B-02
Learning with Leading Indicators
Hanford Performance Indicator Forum Statistical Process Control June 2005 Steven S Prevette
Choosing Performance Indicators Savannah River Nuclear Solutions, LLC
Safety Self-Inspections
Hanford Performance Indicator Forum
LESSON 3 Job Analysis Dr. Salim Al-Shukaili.
Power point presentation
Karien Uys, M.Soc.S, BNS, RN, CQI&PS. The Journey Continues.
Scientific Method Questions of the Day...
Zimbabwe 2008 Critical Thinking.
Performance Measures and Statistics
Savannah River Nuclear Solutions, LLC
Steven S Prevette ASQ Certified Quality Engineer
The Life Cycle of a Trend Savannah River Nuclear Solutions, LLC
Trending Requirements and Results
TS
Business Case Template
Six Sigma Introduction 1 1.
Presentation transcript:

Choosing Performance Indicators Savannah River Nuclear Solutions, LLC Steven S Prevette Senior Statistician Savannah River Nuclear Solutions, LLC SPC Trending Primer / Two Day Training http://www.efcog.org/wg/esh_es/Statistical_Process_Control/index.htm

Management F-Law from Dr. Ackoff Managers who don’t know how to measure what they want, settle for wanting what they can measure Dr. Russell L Ackoff and Herbert J Anderson at http://www.f-laws.com/pdf/A_Little_Book_of_F-LawsE.pdf

Introduction to Measurement It is more important how the measure is used than what the measure is Self-fulfilling prophecies can prevent us from gathering any data We are drowning in data, but little knowledge is derived Context and Operational Definitions are crucial Don’t ignore “gut feelings” and emotions

Creating a Management System Dr. Ackoff, Creating the Corporate Future Three Management Functions: ID of actual and potential problems (threats and opportunities) Decision making (what to do and doing it, or having it done) Maintenance and improvement of performance under changing and unchanging conditions

Five Critical Issues – Dr. Ackoff (1, 2, 3) Managers suffer from overabundance of irrelevant information. Managers don’t know what information they need. Need to look at the decision process to determine this. Even if given the information they need, decision making will not necessarily improve.

Five Critical Issues – Dr. Ackoff (4 and 5) More communication does not necessarily lead to better performance. Information can be used destructively. Managers do need to know how the information system works. Just because it came from a computer doesn’t mean it is right.

Designing a Management System - Ackoff The information system should be designed as an integral part of the management system Most information systems are designed independently, leading to failure Information systems should serve management, not vice versa

Dr. Ackoff’s message We do need to know the context within which the performance indicators will be used Forecasting and living with the forecasted future is important, but what about designing a better future?

Three Information Sources Worker and Customer Opinion Expert Review Process Measures We will focus on Process Measures for this topic. Note that opinions can be converted to measurement data with survey analysis, and results can be converted to measurement data through grading criteria.

Survey Processing Sidebar Surveys usually are not analyzed well Most arithmetically average 1 to 5 Likert scale and don’t asses for variation This assumes people think linearly This assumes each category is equal width Can over-react to random variation For a better idea, see http://www.hanford.gov/rl/uploadfiles/VPP_AnaSurveyData.pdf

Some Approaches for Choosing PI’s Top-Down Process Approach Bottom-Up Customer Focus Idealized Design Leading Indicators

Top-Down Approach Look at your Mission and Vision What are your Products, Services and Customers What is your Business Objective What are desired Outcomes What are the Processes that accomplish the above (drawing a flow chart may help) Decide on Measures (see next page) Go set up data sources, gather data

Process Process Approach REWORK Credit to Phil Monroe, DEMCOM Product Service Schedule OUTCOMES Mission Progress Commitments Met Stewardship Profit Safety Satisfaction Compliance THEORY Cycle Time In-Process Inventory Procedure W ASTE I D L E Dollars Hours Materials Data Budget REWORK Credit to Phil Monroe, DEMCOM

Connecting Process, Output, and Outcome Outcomes are only achieved as a result of a process Focusing ONLY on outcomes is a sure path to failure Ignoring outcomes is a sure path to failure When I provide a Product or a Service, what is my THEORY that connects this to a favorable outcome? Example – I provide statistical training to you. My product is you, as you leave this room. My theory is that you will apply the knowledge you have been provided, apply it to performance indicator work, which will cause continual improvement to occur and have a positive impact on accomplishing the Mission of your Organization.

Examples Input Rate (units per time or Dollars, Hours) Efficiency (Input vs. Budget, Input vs. Idle Cycle Time (Baldrige Criteria pushes cycle time) Backlog (inventory) Procedure Compliance, Completion without Stoppage Output Rate (units per time of Product or Service) Productivity (Output divided by Input) Defect Rate (Waste + Rework vs. Output) Effectiveness (Outcome measures, Outcome per Input, Percent Compliant, Output vs. Schedule)

Bottom Up Approach Go find out what data you currently have Why are you collecting it? What could it tell you? Choose measures from available data Refine by trial and error Advantage – Cost Effective, utilizes existing resources Disadvantage – Only focuses on “visible” data, is Reactive, Not “designed” (re: Ackoff)

3rd Meet in the middle, identify gaps Combination Approach Work down from the Aim of the Organization 1st 3rd Meet in the middle, identify gaps 2nd Work up from the data on hand

Customer Focus Put yourself in your customer’s place. What is important to the customer? Customer Satisfaction, Loyalty. Can it be measured or inferred? Do the same for your employees.

Idealized Design Your current organization and systems are gone What would you put in place TODAY? You are free to replace it with anything you like as long as: It is technologically feasible It must be operationally viable Dr. Russ Ackoff

Low Frequency - High Impact Events There are several “outcome” based measures such as Deaths, Bankruptcy, Property Damage, Environmental Damage which are infrequent occurrences but carry a high perceived risk and emotional reaction Even lucky success – “Dow Jones breaks 12,000” can be problematic When one of these events occurs, we are susceptible to over reaction to the event itself and justify reasons for it in hindsight The Black Swan: The Impact of the Highly Improbable Nassim Nicholas Taleb

Jump Start with Leading Indicators Just what are leading indicators, anyway? Predictions of the future? or A means to create a better future?

Creating a Better Future Distracted by calls to predict future, we delay development of leading indicators At low injury rate, little information exists in outcome indicators Trending response time is long at low rates Use leading indicators to measure lower threshold data and activities Quickens trend response and improves outcomes

Choosing Leading Indicators What are you doing to “Create a Better Future”? What are your program elements, your grass-root efforts to improve? What are the activities you are conducting which you expect will lead to better performance on outcomes? What is your theory for achieving better outcomes? What are lower threshold indicators that could be measured or inferred?

Safety Leading Indicators at Fluor and SRNS Events – first-aid cases, occurrences, near misses Safety inspections – number and score Employee input – safety concerns and survey responses Behavior Based Safety – number of observations and at risk behavior rates Management Observations – number and results Senior management reviews weekly and published in company scorecards

Low Threshold Events (example at Fluor Hanford)

Safety Inspections (Fluor Hanford)

Employee Sentiment (Fluor Hanford and SRNS)

Behavior Based Safety (SRNS)

Allows focus on doing the right things right Results Leading indicator trends at both sites allowed management to stay ahead of issues Fluor Hanford OSHA Recordable Case Rate (TRC) dropped 28% since start of use (May 03 – Apr 04 compared to Feb 05 – Jan 06) and 2007 lowest TRC rate SRNS corrective actions from adverse events of summer 2009 included leading indicators SRNS Operations 53% reduction in TRC, FY 2010 best rate since 1985 Allows focus on doing the right things right

Correlations at SRNS Note: Black line is OSHA Case Rate, Green line is the Leading Indicators

Correlations at Fluor Hanford

Leading and Lagging Indicators Lagging Indicators dominate at the higher levels, reflecting outcomes. Tend to be standardized and dictated from above. Leading Indicators dominate at the lower levels, reflecting processes that achieve the outcomes. Tend to be customized, and driven from the bottom-up.

Hierarchy of Indicators LEADING LAGGING Corporation Project Facility Team Lagging indicators dominate at high levels, leading at lower levels.

Performance Indicator Evolution As a process matures, one may end up evolving the indicators used. For example, if interested in completing actions by commitment dates, one may end up using (as the process matures): Percent of Actions completed by due date in effect at time of completion Percent of Actions completed without missing any due dates during their life Percent of Actions completed by the original due date Average days Actions completed ahead of original due date

Can we really measure anything? Perhaps, but there are limits There is a good book on “how to measure anything” Don’t become a Robert MacNamara “whiz kid” Vietnam war “body counts” Pay attention to gut feelings, subjective risk Reconcile any differences with the numbers Can we support or refute the statement “I love my spouse more than you love your spouse” with a measurement? Should we?

Barrier: The Search for the “Perfect” Indicator When committees get together and try to table-top the perfect indicator, paralysis often sets in. Realize all data are flawed, there is no “true value”, indicators can always be “gamed.” Putting the right culture of HOW to use performance indicators in place minimizes adverse impacts. Gain experience with simple indicators, then move on to more complex indicators if needed. With proper analysis, flaws with existing data can be detected and fixed. If you never look at the data, there will never be an incentive to fix the data.

Barrier: Fear Higher ups will use it as a “hammer” Subjected to quotas and targets imposed from above Fear (“accountability”) used as a “motivator” Actions and Explanations as a result of random fluctuations Perceived loss of control over portrayal of performance Must develop “perfect” indicator the first time Use of SPC can minimize these fears

Barrier: 3 Kinds of Numbers for Management Facts of life. If we don't make this profit figure, we will go out of business. Planning, prediction and budget. Can be used to compare alternative plans. Arbitrary numerical targets. Generally used to judge workers by. Avoid the use of the 3rd kind of number Henry Neave The Deming Dimension

“Just Do It” All data are flawed Make good use of your data Endless conference table discussions won’t cause any data to appear Initial prototype successes will lead to experience, and will further the spread of the use of indicators

Data Gathering Plan ahead Establish Operational Definitions Check data quality Use existing databases before creating new

Plan Ahead “It’s absolutely vital for business that you settle this method of counting, measuring, definition of faults, mistake, defect, before you do business. It’s too late afterwards” -Dr. W. Edwards Deming How many initiatives have we embarked upon, without a clear set of indicators established up front, only to be left with, a year afterwards, trying to figure out “what happened”?

Operational Definitions “Clean the Table”

Operational Definitions Most arguments about conflicting data come down to the definition of how to count the data Try to be precise in your definitions, but likely something unforeseen will arise Record the detail someplace: EM-SR—SRNS prefix occurrence reports Counted by categorization date Does not include canceled reports

Context Do not look at a chart (or any data) in a vacuum Reconcile any differences between the data and “gut feeling” Combine experience and the data Lessons from the data should lead to insight in the field, and vice versa

Data Quality Data should be replicable Operational Definitions are a must Source Data must be defined There is no “true value” of any measure, but a good operational definition can save much trouble in the future ANYONE at ANYTIME in the future should be able to apply the same operational definition to the same source data, and get the same results.

Conclusion Performance Indicators are part of the Management System Good use of data will encourage Performance Indicator Development All data are flawed – Set up good Operational Definitions to minimize flaws “Just Do It” – Start collecting data and use it There is no such thing as a bad performance indicator, only bad use of performance indicators Good use of performance indicators will lead to continual improvement