Download presentation
Presentation is loading. Please wait.
Published byMelissa Perry Modified over 9 years ago
1
1 2012 EXAMINER TRAINING 2012 EXAMINER TRAINING Introduction to Evaluating a Baldrige Application Presented by The Granite State Quality Council and The Northern New England Alliance for Excellence
2
2 Agenda Stages of a Baldrige Evaluation The Evaluation Process –Before you begin –Key Factors –Key Themes –Evaluating each Item Process Item Demonstration Results Item Demonstration Review of the Assignment
3
3 What is a Baldrige Application? A structured response to the questions posed by the Criteria –May be for internal use or for state/national award Contains: –Eligibility forms –Glossary of Terms –Organizational Profile (5 pages) –Categories 1-7 (50 pages)
4
4 The Role of a Baldrige Examiner Provide objective feedback on Strengths and Opportunities for Improvement (OFIs) –First pass is based solely on the Application –A Site Visit may follow to verify and clarify feedback from Application
5
5 The Goal of an Application Evaluation Unbiased feedback on Strengths and Opportunities For Improvement (OFIs) Baldrige feedback is –Non-prescriptive –Non-judgmental –Grounded in the Criteria & Core Values –Weighted on the Organization’s Profile Feedback may be used to determine award level
6
6 The Baldrige Application Evaluation Process - Stages Goal: Value Added Feedback To the Applicant Consensus Comments Independent Observations Site Visit Comments Stage 1 Stage 2 Stage 3
7
7 Application Evaluation Process
8
8
9
9
10
10 The Application Evaluation Process – Independent Observations Check for Conflict of Interest Assemble Materials and Read Application Draft Initial Key Factors Draft Initial Key Themes Observations Finalize Key Themes Observations Complete Checklist and Assemble Scorebook Evaluate Each Criteria Item
11
11 Contents of a Baldrige Application Case Study : Tillingate Living Eligibility Information –Pages i – x Additional Information –Org Charts, Glossary/Abbreviations (pages xi – xv) Responses to Baldrige Criteria Questions –Preface: Organizational Profile (pages xvi – xx) –Categories 1 – 7 (pages 1-50)
12
12 The Application Evaluation Process Check for Conflict of Interest Assemble Materials and Read Application Draft Initial Key Factors Draft Initial Key Themes Observations Finalize Key Themes Observations Complete Checklist and Assemble Scorebook Evaluate Each Criteria Item
13
13 What you will use to Evaluate the Application Application Criteria No independent research or contact with Applicant On-line Worksheets
14
14 What you will use to Evaluate the Application On-line Worksheets
15
15 The Narrative Application Evaluation Process Check for Conflict of Interest Assemble Materials and Read Application DraftInitial Key Factors Draft Initial Key Themes Observations Finalize Key Themes Observations Complete Checklist and Assemble Scorebook Evaluate Each Criteria Item
16
16 Key Factors What are they? –Significant attributes of an organization –that influence the way the organization operates Where do they come from? –The Organizational Profile –Criteria Items –Eligibility Forms Guidelines –Group under the five Areas to Address in the Organizational Profile –Write in phrases rather than complete sentences –Numbered list (for use in rest of analysis) –Revise throughout the evaluation process –Limit to 1–3 pages in length
17
17 How to Identify Key Factors Answers to “What” questions from the Organizational Profile Org Profile Criteria Items Key Factors No Evaluation of Key Factors
18
18 Key Factors: Example Key Factors: –Mission, vision, and values –Employee/staff profile –Customer and market segments and customer requirements –Competitive position and critical success factors –Strategic challenges –Governance structure Not Key Factors: –Strategic planning approach –Complaint management process –Rewards and Recognition System –Methods to promote and ensure ethical behavior Answers “What” - facts or attributes that affect the way the organization operates Answers “How” - processes that the applicant uses in managing its work
19
19 Case Study page xvi Criteria Page 4
20
20
21
21 The Key Factors Worksheet Key Factors Worksheet P.1a Organizational Environment For-profit, privately held organization/Care Model - For-profit, privately held organization providing assisted living and skilled nursing care in 23 facilities located in Pennsylvania, Kentucky, Tennessee, and Virginia whose facilities are known for a sense of societal responsibility. Care model is centered on advancing seniors’ independence and quality of life utilizing the Aging Actively Consortium's (AAC’s) dimensions of wellness: emotional, intellectual/cognitive, physical, spiritual, social, professional/vocational, and environmental. Mission, Vision, Values - Mission (Figure P.1-2): Provide ageless care and timeless living to individuals in a homelike environment that supports their lifestyles and need for care with dignity and respect Vision (Figure P.1-2): Be among the top 10% of skilled nursing facilities (SNFs) and assisted living facilities (ALFs) and be a top choice for care Values (Figure P.1-2): Agility, Patience, Empathy, and Excellence (APEX) Core Competencies (Figure P.1-2) - Designing, innovating, and managing facilities to support various lifestyles and deliver excellent clinical outcomes Developing clinical and service competencies for a caring and exceptional staff Designing and delivering rehabilitation services to support residents’ activities of daily living Creating an educational environment to support a sense of mastery for residents Key Factors: a bulleted list of Applicant responses to questions in the Organizational Profile.
22
22 The Narrative Application Evaluation Process Check for Conflict of Interest Assemble Materials and Read Application Draft Initial Key Factors Draft Initial Key Themes Observations Finalize Key Themes Observations Complete Checklist and Assemble Scorebook Evaluate Each Criteria Item
23
23 What are Key Themes? A Summary of Key Points found in the Evaluation Key Theme: A high-level strength or opportunity for improvement that –Is common (cross-cutting) to more than one Item/Category and/or –Is significant in terms of the applicant’s key factors and/or –Addresses a Core Value of the Criteria –Must be traceable to comments found in the Item Worksheets As a whole Key Themes: –Summarize how well an applicant has addressed the Criteria requirements, Key Factors, and Core Values –Serve as an Executive Summary Importance: –Key Themes are usually the first and sometimes the only part of the feedback report read by senior leaders –Used by examiners to focus site visit –Used by judges to get a “big picture” of the Applicant
24
24 Organization of the Key Themes Worksheet A. What are the most important strengths or outstanding practices found in Process Items? B. What are the most significant opportunities, concerns, or vulnerabilities identified in Process Items? C. what are the most significant strengths found in its response to Results Items? D. What are the most significant opportunities, vulnerabilities, and/or gaps found in the Results Items?
25
25 Steps in Developing Independent Key Themes 1. Read application & capture initial impressions. 2. Evaluate application at the Item-level. 3.Key Themes developed at Consensus –Refined after Site Visit
26
26 Agenda Review Stages of a Baldrige Evaluation The Evaluation Process Before you begin Key Factors Key Themes Evaluating each Item Process Item Demonstration Results Item Demonstration Review of the Assignment
27
27 Baldrige National Quality Program Developing Independent Review Observations Process Item Demo
28
28 The Application Evaluation Process Check for Conflict of Interest Assemble Materials and Read Application Draft Initial Key Factors Draft Initial Key Themes Observations Finalize Key Themes Observations Complete Checklist and Assemble Scorebook Evaluate Each Criteria Item
29
29 Evaluate Each Criteria Item Read Criteria Requirements Determine 4-6 Key Factors Develop ~6 Strengths and OFI’s and Supporting Evidence Read Applicant's Response Write 2 Feedback Ready Comments Determine Score
30
30 Step 1: Read Criteria Requirements
31
31 Item 5.1 Requirements Page 18 in Criteria Booklet Support Info: Notes Category & Item Description Page 42
32
32 Step 2: Key Factors
33
33 P.1a Organizational Environment P.1b Organizational Relationships P.2a Competitive Environment P.2b Strategic Context P.2c Performance Improvement System Item 5.1 Independent Review Worksheet – Key Factors What 4-6 Key Factors do we most expect to be addressed by this Item? (or – which Key Factors do you most want to keep in mind as you evaluate this Item) From Key Factors Worksheet – Handout 4-6 Key Factors are requested at the top of every Item Worksheet All Key Factors Item Requirements Item Key Factors
34
34 Item 5.1 Independent Review Worksheet – Key Factors 4-6 Key Factors are requested at the top of every Item Worksheet
35
35 Step 3: Read Application Item Response Categories 1-6 Demonstrate Processes “HOW?”
36
36 Read through the Item: Example - Item 5.1a(1) What does the Criteria ask? How does the Applicant respond? Case Study: Page 22 & 23
37
37 Definition – PROCESS Linked activities with the purpose of producing a product or service for a customer (user) within or outside the organization Involves combinations of people, machines, tools, techniques, materials, and improvements in a defined series of steps Service situations - used more generally –May spell out what must be done including a preferred or expected sequence Knowledge work (e.g., strategic planning, research, development, and analysis) –does not always imply formal sequences of steps –implies understandings regarding competent performance, such as timing, options to be included, evaluation, and reporting. Page 62 in Criteria Book
38
38 Analyzing a Process Item The purpose of Process Items is to permit diagnosis of an organization’s most important processes — the ones that contribute most to organizational performance improvement and contribute to key outcomes or performance results. KEY Processes
39
39 Read through the Item: Example - Item 5.1a(1) What does the Criteria ask? How does the Applicant respond? Case Study: Page 22 & 23 What Processes can you identify in 5.1a(1)?
40
40 Step 4: Develop Around 6 Total Strengths and OFI’s and Supporting Evidence
41
41 Developing Strengths and OFIs Strengths –Practices/processes or results to be reinforced & build upon Opportunity for Improvement –Potential gaps, blind spots & vulnerabilities Categories 1-6 reflect Processes Category 7 reflects Results
42
42 Main IdeaSupporting Evidence First Pass: Focus on identifying… A-D-L-I Relationship with Key Factors Relationship with Criteria Strengths
43
43 Gaps and OFIs
44
44 From our Example - Item 5.1a(1)
45
45 Example OFIs from - Item 5.1
46
46 Evaluating a Process Item Processes include –Approach (A) –Deployment (D) –Learning (L) –Integration (I) A-D-L-I is a tool to help in application evaluation and scoring A-D-L-I is the P-D-S-A of High Performing Organizations
47
47 Approach The methods and processes used by an organization to accomplish its work HOW an organization addresses the item requirements Approach considers… –Appropriateness for organization –Effectiveness –Alignment with the organization’s needs –Appropriate review and improvement Approaches are systematic if they –Are well ordered –Are repeatable –Use data and information so learning is possible –Build in the opportunity for evaluation, improvement, and sharing thereby permitting a gain in maturity
48
48 Deployment Refers to the EXTENT to which an organization’s approach is applied The breadth and depth of the approach to relevant work units throughout the organization Implementation on steroids… Is the approach deployed from the President to the janitor?
49
49 Deployment Appropriate Deployment is process specific. Appropriate deployment varies from item to item Examples: Category 1 – Leadership Processes –Deployed to workforce segments, stakeholders, other key processes Category 3 – Customer Processes –Deployed to customer segments, work force segments Category 5 – Workforce Processes –Deployed to workforce segments, locations, departments
50
50 Learning: Organizational and Personal New knowledge or skills acquired through evaluation, study, experience, and innovation. Criteria looks for both personal and organizational learning There are three dimensions of organizational learning: – Continuous improvement of existing processes – Innovation leading to breakthrough changes – Knowledge sharing of such improvements and innovations
51
51 Integration The harmonization of plans, processes, information, resource decisions, actions, results, and analysis to support key organization-wide goals. –Begins with alignment –Culminates in interconnectivity Integration asks: –How well do the individual components of a system operate as a fully interconnected unit? –How well is the approach aligned with the applicant’s organizational needs? –How well is the approach integrated with these needs?
52
52 Indicating A-D-L-I A-D-L-I
53
53 Identifying Process Strengths & OFIs 1.Criteria –Strength? –Gap? 2.Key Factors –Importance & Appropriateness 3.A-D-L-I –Specifics
54
54 Step 5: Write 2 Feedback Ready Comments
55
55 Feedback Ready Comments Turns Strength/OFI & Evidence into feedback the applicant understands –Polite, Professional, Positive –Not: Judgmental or Prescriptive –Concise, one message per comment –Indicates Criteria-based strength or potential OFI Potential significance of strength/OFI
56
56 Creating a Feedback Ready Comment: Strength 2,4,5,6Systematic integrated approach to employee health & safety - Exec Director of each facility = risk management/safety officer, serve on system wide Safety Committee - Review all facility/system health safety, security policies & procedures - Meet monthly to review key measures; APEX Goals - Integrated with daily rounds with Collaborative Care Teams - Use of Data to drive improvements b(1) b(1) A variety of methods and associated goals (Figure 5.1-2) help ensure workplace health, safety, and security. These methods include a system-wide Safety Committee that meets monthly to review progress toward goals, unexpected events, and best practices. Daily rounding by Collaborative Care Teams helps identify potential safety hazards.
57
57 Creating a Feedback Ready Comment: OFI 1,2,3, 4, 6 Approaches to management of workforce capability/capacity & to maintain a safe, secure & supportive work climate do not address all segments of workforce - Little discussion of physicians, volunteers, students --Unclear if security is an issue at some of the locations (e.g. SLFs) -Approach for capacity planning is somewhat unclear – they just post the positions and move people around? a,b a, b It is unclear how the applicant manages volunteers, physicians, and students to fully support its work. For example, it is unclear how volunteers are trained and managed and how approaches are deployed to precepted students and credentialed physicians. Approaches for managing these workforce groups may help the applicant deliver high quality care and services and maintain a safe, secure environment.
58
58 b(1) A variety of methods and associated goals (Figure 5.1-2) help ensure workplace health, safety, and security. These methods include a system-wide Safety Committee that meets monthly to review progress toward goals, unexpected events, and best practices. Daily rounding by Collaborative Care Teams helps identify potential safety hazards. Polite? Professional? Positive? JudgmentalPrescriptive Nugget: Key point based on Criteria Indicates Relevance to this applicant Examples for clarity & context
59
59 Polite? Professional? Positive? JudgmentalPrescriptive a, b It is unclear how the applicant manages volunteers, physicians, and students to fully support its work. For example, it is unclear how volunteers are trained and managed and how approaches are deployed to precepted students and credentialed physicians. Approaches for managing these workforce groups may help the applicant deliver high quality care and services and maintain a safe, secure environment.’ Nugget: Key point based on Criteria Indicates Relevance to this applicant Examples for clarity & context
60
60 Step 6: Scoring
61
61 Scoring a Process Item Two components –A-D-L-I –Level of Response to Item Requirements
62
62 Baldrige Scoring: Reflects Organization’s Maturity in the Journey Level of Response to Item Requirements Two dimensions of scoring –For Process Items (Categories 1-6) Approach Deployment Learning Integration –For Results Items (Category 7) Levels Trend Comparisons Integration
63
63 The Performance Excellence Journey BeginningOn the Road Mature Anecdotal evidence, beginning of systematic approaches Clear evidence of effective, systematic approaches fully responsive to current and changing business needs Little or no deploymentApproaches consistently deployed to all relevant work units Reaction to problems; early stages of an improvement system Fact-based, systematic evaluation and improvement process; strong refinement and innovation Little or no alignment of key processes and work units Approaches, processes, and work units fully aligned and integrated with organizational needs Maturity of Processes
64
64 Basic Requirements
65
65 Overall Requirements
66
66 Multiple Requirements
67
67 Item Format Criteria – Page 29
68
68 UNDERSTANDING SYSTEMATIC APPROACH Page 68 - Criteria
69
69 UNDERSTANDING DEPLOYMENT
70
70 UNDERSTANDING IMPROVEMENT/LEARNING/INNOVATION
71
71 UNDERSTANDING INTEGRATION
72
72 Scoring Basics Scores should align with comments. Base the scores on a holistic view of the item and the applicant’s responses Avoid requiring more evidence than is reasonable given the applicant’s 50-page limit. From low to high, the scoring ranges reflect a growth in organizational maturity. Note that the scoring ranges reflect a growth in maturity. In the scoring ranges, note how the language for each scoring factor (ADLI) changes to reflect higher levels of organizational maturity Tip: Start at the 50 – 65% band
73
73 Holistic Scoring Rubric based on Maturity ADLI Basic – Overall - Multiple
74
74 Master Scoring Grid Criteria Point Values Page 3 - Criteria Determined by Examiners
75
75 Baldrige National Quality Program Developing Independent Review Observations Results Item Demo
76
76 The Model for Performance Excellence is a Results Driven Model!
77
77 Definition of Results “Outputs and outcomes achieved by an organization in addressing the requirements of a Baldrige Criteria Item”
78
78 Results Results document how the applicant measures – its performance over time – against competitors’ performance – against relevant comparative data
79
79 Results Measure A-D-L-I… Make note of those things that the applicant says are important and that the criteria calls for in Categories 1-6 Check to see if they are reported in Category 7
80
80 Not all Results are created equal… Current Performance Segmentation Trends Comparisons
81
81 Results - Current Levels Employee survey question – I am satisfied with my job. % Strongly Agreed
82
82 Current Levels and Segments % Strongly Agreed Employee survey question – I am satisfied with my job.
83
83 Results - Trends % Strongly Agreed Employee survey question – I am satisfied with my job.
84
84 Results - Comparisons % Strongly Agreed Employee survey question – I am satisfied with my job.
85
85 Results: Evaluate Each Criteria Item Read Criteria Requirements Determine 4-6 Key Factors Develop ~6 Strengths and OFI’s and Supporting Evidence Read Applicant's Response Write 2 Feedback Ready Comments Determine Score
86
86 Step 1: Read Criteria Requirements
87
87 Results Example – Item 7.3 Page 24 of Criteria Support Info: Notes Category & Item Description Pages 47 - 48
88
88 Step 2: Key Factors
89
89 Item 7.3 Key Factors for Tillingate Living What 4-6 Key Factors do we most expect to be addressed by this Item? (or – which Key Factors do you most want to keep in mind as you evaluate this Item) From master Key Factor Worksheet 1. Mission, Vision, Values (Figure P.1 ‐ 2) ‐ Mission: Provide ageless care and timeless living in a homelike environment, dignity, and respect Vision: Be among the top 10% of Skilled Nursing Facilities (SNFs) and Assisted Living Facilities (ALFs) and be a top choice for care Values: Agility, Patience, Empathy, and excellence (APEX) 2. Employee and Workforce Demographics (Figure P.1 ‐ 3) ‐ 3,718 employees; 3. Workforce Groups: Employees, physicians, 700 volunteers; Engagement Factors ‐ Good work environment; Good benefits; Positive relationships with coworkers; Pride in the organization; A voice in resident care 4. Workforce Health and Safety Requirements ‐ Protection for exposure to communicable diseases; Protection form injury; Support in managing residents; Support for a healthy lifestyle; 5. Key Competitors: all other CMS 5 ‐ Star ‐ rated facilities within 50 miles of each facility; 6. Sources of comparative data: CAHPS questions, CMS Nursing Home Compare, USSN Data Dispatcher, Caring Colleagues, NursQM
90
90 Step 3: Read Application Item Response
91
91 Results Example – Item 7.3a(1) What Results are provided FOR 7.3a(1)?
92
92 Step 4: Develop Around 6 Strengths and OFI’s and Supporting Evidence
93
93 Defining “Key” Results? “Key” refers to elements or factors most critical to achieving the intended outcome In terms of results, look for those responsive to the Criteria requirements most important to the organization’s success results that are essential elements for the organization to pursue or monitor in order to achieve its desired outcome
94
94 Key Results Results evaluated by different factors
95
95 Le-T-C-I: Let’s see: Evaluating outputs and outcomes Levels The current level of performance Trends Rate or slope of improvement Breadth or extent of deployment of results Comparisons Performance relative to appropriate comparisons (industry, similar organizations) benchmarks or industry leaders Integration Linkage of results to important requirements or processes as identified in the Organizational Profile and Process Items Valid indicators of future performance Harmonized across processes and work units to support organization goals
96
96 Tillingate Living Item 7.3a(1) Strength Note that examiners found evidence outside of a(1) related to capability & capacity
97
97 Tillingate Living Item 7.3a(1) OFI
98
98 Step 5: Write 2 Feedback Ready Comments
99
99 Creating a Feedback Ready Comment: Results OFI a(1) The applicant has limited results for workforce capability and capacity which may challenge its ability to achieve strategic objectives and goals. For example, results on competencies or skill levels are not reported. Also, results for capacity are limited to ratios of nursing care time to health care outcomes. Additional capability and capacity outcomes may support advancing the quality of care and achieving top-decile resident satisfaction while achieving role-model financial stability.
100
100 Polite? Professional? Positive? JudgmentalPrescriptive Nugget: Key point based on Criteria Indicates Relevance to this applicant Examples for clarity & context a(1) The applicant has limited results for workforce capability and capacity which may challenge its ability to achieve strategic objectives and goals. For example, results on competencies or skill levels are not reported. Also, results for capacity are limited to ratios of nursing care time to health care outcomes. Additional capability and capacity outcomes may support advancing the quality of care and achieving top-decile resident satisfaction while achieving role-model financial stability.
101
101 Creating a Feedback Ready Comment: Results Strength
102
102 Creating a Feedback Ready Comment: Results Strength a(3) Top-decile performance on employee satisfaction and engagement survey measures, as well as low vacancy and turnover rates (Figure 7.3-7), help the organization sustain its strategic advantage of high employee retention. Overall satisfaction has been at or better than the top-decile level since 2008 (Figure 7.3-4). Engagement results for "I am proud to work [here]" and "I would recommend [applicant] to family members“ survey items were at top-decile levels in 2010 and 2011 (Figure 7.3-6). Polite? Professional? Positive? JudgmentalPrescriptive Nugget: Key point based on Criteria Indicates Relevance to this applicant Examples for clarity & context
103
103 Step 6: Scoring
104
104 Results Scoring Guidelines – Page 69 SCORING DIMENSION DESCRIPTOR LEVELSNo results; poor results reported Few results reported for Basic requirements Good results for Basic requirements Good results for Overall requirements Good to excellent results for Multiple requirements Excellent results fully responsive to Multiple requirements TRENDSTrend data not reported or show adverse trends Some trend data, with some adverse trends Some trend data, with majority presented as beneficial Beneficial trends in areas of importance Beneficial trends sustained over time in most areas of importance Beneficial trends sustained over time in all areas of importance 0-5% 90-100% 0-5% 90-100%
105
105 SCORING DIMENSION DESCRIPTOR COMPARISONSComparative data not reported Little or no comparative data reported Early stages of obtaining comparative data Some current performance levels evaluated against relevant comparisons Many to most trends evaluated against relevant comparisons or benchmarks; show levels of leadership and very good relative performance Industry and benchmark leadership in many areas INTEGRATIONResults are not reported for areas of importance Results are reported for a few areas of importance Results are reported for many areas of importance Results reported for most key customer, market, and process requirements Results reported for most key customer, market, and process requirements and action plans Results and projections reported for most key customer, market, and process requirements and action plans 0-5% 90-100% 0-5% 90-100%
106
106 Determine an overall scoring range and score
107
107 The Application Evaluation Process Check for Conflict of Interest Assemble Materials and Read Application Draft Initial Key Factors Draft Initial Key Themes (optional) Finalize Key Themes Complete Checklist and Assemble Scorebook Evaluate Each Criteria Item
108
108 Agenda Review Stages of a Baldrige Evaluation The Evaluation Process Before you begin Key Factors Key Themes Evaluating each Item Process Item Demonstration Results Item Demonstration Review of the Assignment
109
109 Materials for Assignment Tillingate Living Case Study –GSQC Website Criteria for Performance Excellence –http://www.nist.gov/baldrige/publications/upload/2011_2012_Health_ Care_Criteria.pdfhttp://www.nist.gov/baldrige/publications/upload/2011_2012_Health_ Care_Criteria.pdf Worksheet Packet ( e-mailed with slides, on websites ) Completed Key Factors Blank Item 5.2 Partially Completed Item 7.4 Reference materials –Slides from today’s workshop(s) –Completed Item 5.1
110
110 The Assignment… Read Tillingate Living Case Study Review Key Factors Worksheet (Assignment Worksheets) –Any questions? Any changes? Prepare Independent Review Worksheets for –Item 5.2: 4-6 Key Factors, ~ 6 Strengths/OFIs worksheets, Score. (2 comments are optional) –Complete Item 7.3: Review/Change Key Factors, ~ 6 Strengths/OFIs worksheets, Score. (2 comments are optional) Plan on 5+ hours of work on Worksheets Submit your completed Worksheets by October 22nd (anne@gsqc.com)
111
111 Follow the 6-step process to complete Worksheets
112
112 Overview of mid-October Session Review and Refinement of assignment Questions and Clarification Overview of Scorebook Navigator tool Assignment for November 15th Establish 2 hour webinar(s) for week of October 22 nd …..
113
113 Final Questions & Feedback?
Similar presentations
© 2025 SlidePlayer.com. Inc.
All rights reserved.