Michigan Quality Leadership Award New Examiner Training 2014

Agenda Welcome and Introductions Begin with the End in Mind Code of Ethical Conduct Key Factors Six-Step Process Evaluating Items Preparing for March 12

Clever Introductions Find a partner to introduce to the class. Share name, affiliation, Baldrige-like experience. Share another item of your choice, or complete this sentence: My zodiac sign is ______ and it will help me be an awesome Examiner because ……

Examiner Roles and Responsibilities for Today’s Learning Take an active role in your own learning by asking questions, sharing information, viewpoints, and roles during the training program. Take an active role in helping others to increase knowledge and develop skills.

Award Process Cycle Application Submission Independent Review Consensus Review Selected Not selected Judges’ Meeting Judges’ Meeting Feedback Report to Applicant Site Visit Judges’ Meeting Judges Name Applicant an Award Recipient Not selected Feedback Report to Applicant Selected

Begin with the End in Mind Feedback Report –Constructive, actionable feedback that helps applicant Identify and leverage Strengths Prioritize and address Opportunities for Improvement (OFIs)

Baldrige Criteria Framework: A Systems Perspective

Ensuring Examiner Ethical Behavior A key success factor for the Award program is promoting and ensuring ethical behavior. Code of Ethical Conduct Conflict of Interest

Code of Ethical Conduct Four Principles: 1.Protect the Integrity of the Award Process. 2. Exhibit Professional Conduct at All Times. 3. Protect the Promise of Confidentiality. 4. Protect the Program’s Intellectual Property.

Conflict of Interest Statement Upon notification of application, immediately review for conflicts of interest. Sign and back the Conflict of Interest statement. Call Geri Markley, Michigan Quality Council, with all questions.

Exercise: Code of Ethics In your small table groups: Assign roles. Discuss your assigned ethics scenario. Prepare to share your consensus response.

Scenario 1: the answer is …… Examiners should not exploit their selection to the Board of Examiners. There are no “secrets” for receiving the award and no guarantees for receiving a site visit. Also, the program prefers to understate the “winning” aspect of the process; it encourages a focus on continued self-improvement and the sharing of best practices to help organizations achieve performance excellence. Examiners may not use the MQC logo in any advertising/promotion (and business cards may not include “MQLA examiner” or the MQC logo). Principles: Protecting the integrity of the award process, the program’s intellectual property, and the reputation of the BOE.

Scenario 2: the answer is …… Any assistance with an organization’s application would result in a conflict of interest for an examiner. The examiner would be considered an employee of the organization. If the organization reapplies, the examiner would not be permitted to evaluate the application. Examiners advising or participating with an organization in preparing an award application shall not reveal or discuss that participation with other examiners during training or at any other time. Principles: Protecting the promise of confidentiality and the integrity of the award process.

Scenario 3: the answer is …… No. At no time during Independent Review or Consensus Review should an examiner contact the applicant. Examiners shall not communicate with the applicant organization or in any manner seek additional documentation, information, or clarification. This includes Internet searches or other external sources of information. Examiners shall safeguard the confidences of all parties involved in the judging or examination of current or former applicants. Principles: Protecting the promise of confidentiality and the integrity of the award process.

Scenario 4: the answer is …… No. Examiners must avoid disclosures that may in any way influence the award integrity or process, currently or in the future, and treat as confidential all information about the applicant. While this information has been publicly released, it is not part of the application, and, therefore, it is not relevant to the evaluation. While potentially interesting, this information should not be discussed and must not be considered in the evaluation. The program shares with the team leader any known information that could affect selection as a role model, and the team leader seeks clarification during the site visit. Principle: protecting the integrity of the award process.

Scenario 5: the answer is …… Yes. Examiners may use the Internet and other sources to familiarize themselves with common or generic terminology. Examiners shall not communicate with the applicant organizations or in any manner seek additional documentation, information, or clarification specific to the applicant. This includes Internet searches or other external sources of information. Principle: protecting the promise of confidentiality.

Master List of Key Factors “ Facts” about the organization –Indicates relevance and importance Not processes or evaluation of processes Mainly from the Organizational Profile

Examiners View Applicants through Two Lenses: Criteria and Key Factors

Examples of Key Factors Size a) 39 employees located in one building b) 73,000 employees located in twelve states Strategic Challenges a) Heavily regulated and intensely competitive b) Rapid expansion from acquiring many small competitors Products a) Highly technical and changes often b) Has not changed in 37 years

Key Factors How are key factors used? Why are they important? Case Study Key Factors

Exercise: Key Factors In your small table groups: Assign roles. Identify key factors for your assigned section of the Organization Profile. Prepare to share your key factors.

Six-Step Item Evaluation Process

Independent Review Steps 1.Ground yourself in the criteria. 2.Select the relevant 4-6 key factors for the item. 3.Read the relevant section of the application. 4.Draft around 6 strengths & OFIs. 5.Write two feedback-ready comments.  One strength  One OFI 6.Determine the scoring range and score for the item.

Step 1: Ground yourself in the Criteria item.

Step 1 – Ground Yourself in Criteria Requirements Read the Criteria item to understand the Item requirements. –Look at 1.1a(1) – Senior Leadership (p.7) How do SENIOR LEADERS set your organization’s VISION and VALUES? How do SENIOR LEADERS DEPLOY the VISION and VALUES through your LEADERSHIP SYSTEM, to the WORKFORCE, to KEY suppliers and PARTNERS, and to CUSTOMERS and other STAKEHOLDERS, as appropriate? How do SENIOR LEADERS’ actions reflect a commitment to those VALUES?

Step 2: Select the relevant key factors for the item.

Step 2 –Select the Relevant Key Factors for the Item Review master list of key factors; determine the attributes of the organization that would influence its responses to the item requirements. Select the most relevant 4-6 key factors for that item. Example: 1.1a(1) would likely have a key factor related to the organization’s vision and values.

Step 3: Read the relevant section of the application.

Step 3 – Read the Relevant Section of the Application Identify the processes or approach the applicant uses to meet item requirements. Flag, mark up and/or take notes as needed. –Item 1.1a(1) may describe an approach used to set vision and values, how they are deployed, etc.

Step 4: Draft around six strengths and opportunities for improvement.

Step 4 – Draft Around Six Strengths and OFIs Select relevant key factors for each approach. Provide brief statement of approach. Provide evidence. Select appropriate evaluation factors (ADLI). Provide reference to specific areas of the item (e.g., a(1), b(3)).

Step 5: Draft feedback-ready comments.

Step 5 – Draft one each Feedback- Ready Strength and OFI Use Comment Guidelines to help craft comments that capture your findings and analysis. Include these key elements: –Brief “nugget” –An example or two –Why this feedback is important

Step 6: Determine the scoring range and score for the item.

Step 6 – Determine Scoring Range and Score for the Item Review balance of observations. Use scoring guidelines (pp ). Determine “best fit” scoring range. Determine score.

Analyzing a Process Item The purpose of process items is to permit diagnosis of your organization’s most important processes: the ones that contribute most to organizational performance improvement and to key outcomes or performance results.

Assessing Process Maturity Process = the methods the applicant uses to address item requirements in Criteria categories 1-6. Four factors are used to evaluate process maturity: Approach Deployment Learning Integration

Approach Approach refers to Appropriateness of methods Effectiveness of methods Degree to which the approach is systematic (repeatable and based on reliable data and information) Degree to which the approach addresses basic, overall, or multiple requirements of the criteria

Criteria Item Format

Item 1.1 – Approach (p. 7) a.(1) How do SENIOR LEADERS set your organization’s VISION and VALUES? How do SENIOR LEADERS DEPLOY the VISION and VALUES through your LEADERSHIP SYSTEM, to the WORKFORCE, to KEY suppliers and PARTNERS, and to CUSTOMERS and other STAKEHOLDERS, as appropriate? How do SENIOR LEADERS’ actions reflect a commitment to those VALUES?

Deployment Deployment refers to the extent to which The approach is applied consistently The approach is used by all appropriate work units The approach is applied to all appropriate areas, customers, suppliers, partners, etc. Tip: Key factors are important in assessing deployment.

Learning Learning refers to Refining the approach through cycles of evaluation and improvement Encouraging breakthrough change to the approach through innovation Sharing refinements and innovations with other work units and processes in the organization

Process Maturity - Learning Three dimensions of organizational learning: – Continuous improvement of existing processes – Innovation leading to breakthrough changes – Knowledge-sharing of such improvements and innovations

Integration Integration refers to the extent to which The approach is aligned with organizational needs. Measures, information, and improvement systems are complementary across processes and work units. Plans, processes, results, analyses, learning, and actions are harmonized across processes and work units to support organization-wide goals.

Process Maturity - Integration

Alignment and Integration

Capturing Observations U se r Ty pe D o u bl e DescriptionEvidenceADLI Item Ref BRBR S tr e n g t h The applicant has an integrated and systematic approach for assessing its workforce capability and capacity needs, including the skills, competencies, certifications, and staffing levels. - HR Capability Assessment is conducted annually - The quarterly Capacity Study ensures a match between position requirements, required competencies, and qualified EOs for each position. - The capabilities are a specific element of the strategic (long-term) and action (short-term) plans resulting from the SPP - action (short-term) plans resulting from the SPP (Figure 2.1-1) A,D,Ia(1)

How do I evaluate Process Items? 1.Ground yourself in the criteria item. 2.Select the relevant key factors for the item. 3.Read the relevant section of the application. 4.Draft around 6 strengths and opportunities for improvement (OFIs). 5.Draft feedback-ready comments (Two for IR). 6.Determine the scoring range and the score for the item. By using the Six-Step Process!! Independent Review

Exercise: Evaluate a Process Item (Steps 1 - 4) In your small table groups: Assign roles. Complete Steps –Ground yourselves in the Criteria by summarizing –Identify 4 – 6 most important Key Factors –Review the applicant’s response –Identify around six observations with evidence Prepare to report out.

Well-Written Comments Why are comments important? Feedback is the basis for improvement. They demonstrate understanding of the applicant and its key factors. What are characteristics of well-written comments? Concise Precise Show insight Actionable

Applicant Requirements A concise opening sentence that expresses a single thought—the “nugget” (or essence, or main point) of the comment One or a few examples (not every example) The relevance or importance of the nugget to the applicant (think key factors)

Sample Comment 6.2b(1) The organization ensures that work processes meet key requirements day-to-day by utilizing technology for real-time monitoring of patient length of stay, test results, turnaround time, and bed availability. Other methods of evaluation include day-to-day rounding by administration, audits, and scorecard reporting. This approach supports the core competency of operational excellence and helps address the strategic challenge of cost control.

Comments Should Not Be prescriptive Parrot the application Go beyond the criteria

Exercise: Write a Comment (Step 5) In your small table groups: Assign roles. Review observations. Select one and write a feedback-ready comment. Prepare to report out.

Scoring System Each Item is assigned a point value. Scoring guidelines describe percentage range of the points based on evaluation factors. The Examiner selects a scoring range that is “most descriptive” of the applicant’s achievement level in that item. The Examiner selects the best score within that range.

Exercise: Score (Step 6) In your small table groups: Assign roles. Review observations and comment. Discuss and agree on scoring range. Agree on score. Prepare to report out.

How do I evaluate Results Items? 1.Ground yourself in the criteria item. 2.Select the relevant key factors for the item. 3.Read the relevant section of the application. 4.Draft around 6 strengths and opportunities for improvement (OFIs). 5.Draft feedback ready comments (Two for IR). 6.Determine the scoring range and the score for the item. By using the Six-Step Process!! Independent Review

What Should You Be Looking For? Let’s See – “LeTCI” Levels Trends Comparisons Integration

Importance What results are required? How do you know what results are important to this applicant? What comparisons are appropriate? What segmentation should you expect?

Appropriate Comparisons Industry mean Best competitor Best-in-class performance

12,000 14,000 Rate of Improvement & Comparisons

Meaningful, Well-Written Results Observations Describe How you grouped various Key Results Whether you think Levels and Trends shown represent favorable or unfavorable performance and why Whether you think the Comparisons provided are appropriate and represent favorable or unfavorable performance and why Whether you think the results demonstrate Alignment and Linkage to the applicant’s key factors and key process requirements

Meaningful, Well-Written Results Observations What Missing Results (Gaps) you noted and why it matters to the applicant, including expected results based on Criteria, process, or other stakeholder requirements What Overall Item observations you have noted, if any. Tip: Be sure to consider results embedded in the text as well as the charts and graphs provided.

7.1a Comparisons to local competitors are limited for customer outcomes, although there are comparisons to top - decile performance levels from national or state databases. Regularly comparing performance to others in the marketplace may help the organization find opportunities to outperform competitors and gain market share, a key strategic challenge. Results OFI Comment

Exercise: Evaluate a Results Item (Steps 1–6) In your small table groups: Assign roles. Complete Steps 1-6 for assigned Item. Prepare to report out.

Preparing for March 12 Prework using Baldrige Online Scoring System (BOSS) Demo

Prework Read Collin Technology Case Study. In BOSS: –Complete master list of key factors. –Complete evaluation (Steps 1 – 6) for these items: 1.1, 5.2, 6.2, 7.4. In Word: –Write a brief description of these items (format provided): 1.2, 2.1, 2.2, 3.1, 3.2, 4.1, 4.2, 6.1, 7.1, 7.2, and 7.3 were provided as examples.

Prework Tips Don’t procrastinate and expect to do it all at once. Use notes from today to help you complete your item evaluations. Contact Geri to take a look at your prework progress and provide feedback or offer tips. Don’t spend more than 20 hours. –If you’re at 16 hours and think you’ll go over, contact Geri to help brainstorm ways to move forward.

Resources Participant Notebook Online Baldrige presentations (prework) Welcome to BOSS Powerpoint Conference Call Feb 7 at 10:00 Examiner Preparation Team Mentors Geri BOSS monitoring and follow-up s by Geri

Questions?