1. Welcome, Examiners! Washington State Quality Award Return Examiner Training 2009

2. WSQA Return Examiner 2009 2 First things first! THANK YOU! WSQA’s mission: Improving the way we live, learn, and work in Washington by helping organizations improve through the use of the Baldrige Criteria for Performance Excellence

3. WSQA Return Examiner 2009 3 Objectives For This Class Review Changes to the Baldrige Criteria Strengthen Skills in Areas of Opportunity Review Resources for Examiner Software

4. WSQA Return Examiner 2009 4 Agenda: 1.WSQA organization overview 2.Baldrige criteria changes 3.WSQA ethics review 4.WSQA process Full Vs Lite 5.Review process 1.Individual review worksheets 2.Consensus process 3.Improvements 6.Examiner Software 7.Summary

5. For a Successful Course… Be an active participantBe an active participant Give only constructive feedbackGive only constructive feedback Be courteous to your colleaguesBe courteous to your colleagues Honor diverse opinionsHonor diverse opinions Seek to understandSeek to understand Ask questionsAsk questions Have fun!!Have fun!!

6. WSQA Return Examiner 2009 6 Introductions Name, organization, what you do Questions and expectations? What do you like to do when you’re not working?

7. WSQA Return Examiner 2009 7 What is WSQA? Created by State law to: Promote excellence Recognize achievement Educate and train Patterned after the Baldrige National Quality Award Awards presented annually by the Governor Awarded to private, public, and not-for-profit organizations in manufacturing, service, education, and healthcare

8. WSQA Return Examiner 2009 8 What is WSQA? WSQA is a 501(c)(3) nonprofit organization supported by Corporate donations Individual memberships Workshop, collaborative and conference fees Many (many, many!) volunteer hours

9. WSQA Return Examiner 2009 9 Who is WSQA? Examiners Lite Assessment Full Assessment Certification follows assessment Senior Examiners Level 3 Team Leads Scorebook Editors Panel of Examiner & Process Development (PEPD) Mentors Panel of Judges Board of Directors Executive Director and support team

10. 2. Baldrige criteria changes

11. WSQA Return Examiner 2009 11 Baldrige criteria booklet Beginning in 2009 revised every 2 years

12. WSQA Return Examiner 2009 12 Baldrige 2007 Criteria Changes Strategic advantages Workforce engagement Leadership- performance measures Strategic Planning-innovation, strategic advantages

13. WSQA Return Examiner 2009 13 Baldrige 2008 Criteria Changes Order of information in the booklet Scoring system description Diagrams added for core values and maturity levels Added strategic advantages as a term to the Glossary

14. WSQA Return Examiner 2009 2009 Criteria Changes Customer Focus/ Engagement Organizational Core Competencies Sustainability and Social Responsibilities 14

15. WSQA Return Examiner 2009 Customer Focus/ Engagement Customer commitment to your brand and offerings Characterized by loyalty and willingness to make an effort to do business with you, willingness to do business with you 15

16. WSQA Return Examiner 2009 Core Competencies Areas of greatest expertise Strategically important Provide market place advantage May involve technologies, unique offerings, market place niche, business acumen 16

17. WSQA Return Examiner 2009 Sustainability Organization’s ability to address business needs and agility and strategy to address the future Considers external and internal factors Considers impact on society and well being of environment, social and economic systems 17

18. WSQA Return Examiner 2009 18 Conflict of interest issues WSQA seeks to avoid conflict of interest to protect process integrity Conflicts are frequent! First step in assignment process: determine any conflicts

19. WSQA Return Examiner 2009 19 Code of Ethical Conduct Why are ethics of utmost importance to WSQA and this process?

20. WSQA Return Examiner 2009 4. WSQA Full and Lite Process Comparison table 20

21. 5. WSQA Process Objective: Review key steps in the process. Review key process opportunities.

22. Stages of the Assessment Process

23. WSQA Return Examiner 2009 23 Preparing for independent review Read the criteria Read the application Familiarize yourself with the online tool All information on the WSQA website: webinars and manuals http://www.wsqa.net/extimetable.php Check in with Team Lead on Key Factors and after entering category 1 and associated results

24. WSQA Return Examiner 2009 24 Independent review 1.Select the most relevant Key Factors (from list) 2.Read the criteria for the category 3.Read (and annotate) the relevant section of the application 4.Identify and record the applicant’s processes 5.Record observations. Note missing responses and significant observations 6.Recommend a scoring range for the category

25. WSQA Return Examiner 2009 25 Independent review steps - process 1.Review related criteria 2.Read the category under review, take notes 3.Select relevant key factors from list 4.Identify & record processes 5.Record observations on ADLI: Approach Deployment Learning Integration 6.Identify any significant or missing responses

26. WSQA Return Examiner 2009 26 Record observations on “ADLI” “Approach” refers to Methods used to accomplish the process How appropriate the methods are to the item requirements The effectiveness of their use The degree to which the process is repeatable and is based on reliable data and information (i.e., systematic)

27. WSQA Return Examiner 2009 27 Record observations on “ADLI” “Deployment” refers to the extent to which The approach is applied in addressing relevant and important item requirements It is applied consistently It is used by all appropriate work units

28. WSQA Return Examiner 2009 28 Record observations on “ADLI” “Learning” refers to Refining the approach through cycles of evaluation and improvement Encouraging breakthrough change to your approach (innovation) Sharing refinements and innovations with other relevant work units and processes

29. WSQA Return Examiner 2009 29 Record observations on “ADLI” “Integration” refers to the harmonization of Plans Processes Information Resource decisions Actions Results Analyses to support key, organization-wide goals

30. WSQA Return Examiner 2009 30 Record observations on “ADLI” Questions to ask in analyzing “Integration” Do the individual components of a performance management system operate as a fully interconnected unit? Is the approach aligned with the organizational needs identified in the Organizational Profile?

31. WSQA Return Examiner 2009 31 Process diagnosis: key concepts Integration examples: Alignment of objectives and action plans with strategic challenges AND mission, vision, values Alignment of product/service delivery methods with KEY customer requirements Alignment of key process measures with KEY customer requirements

32. WSQA Return Examiner 2009 32 Process diagnosis: key concepts 1. Approach 3. Learning 4. Integration 2. Deployment PROCESS

33. WSQA Return Examiner 2009 33 Building the ADLI worksheet ItemProcessKF’sADLI 1.1a(2)Foster/ require legal & ethical behavior #, #, #... ++ Annual overview of obligations, online training, post-course survey + Covers staff, board, partners, volunteers + Daily Huddle addresses concerns quickly - None seen + Addresses HIPAA, patient confiden- tiality How would you score this single response?

34. WSQA Return Examiner 2009 34 Independent review steps - results 1.Review the Criteria to identify required results 2.Review the relevant process in the application to identify KEY results important to the applicant (use KF’s, application, your diagnosis) 3.Review the Results in the application What IS there? What ISN’T there that should be? 4.Evaluate & score on the basis of LeTCI

35. WSQA Return Examiner 2009 35 Results evaluation: key concepts Levels Numbers that position results on a meaningful measurement scale Permit evaluation relative to Past performance Projections Goals Appropriate comparisons

36. WSQA Return Examiner 2009 36 Results evaluation: key concepts Trends Numbers that indicate direction and rate of change Provide a time sequence of performance Require at least three data points Criteria call for trend data on Product, service performance Customer, workforce satisfaction Financial performance Market performance Operational performance (cycle time, productivity)

37. WSQA Return Examiner 2009 37 Results evaluation: key concepts Comparisons External requirements External benchmarks Similar OR different organizations with similar characteristics

38. WSQA Return Examiner 2009 38 Results evaluation: key concepts Integration Results align with key factors, e.g., Strategic challenges, Workforce requirements Vision, mission, values Results presented for Key processes Key products, services Strategic accomplishments What examples can you think of? Strong integration? Not-so-strong integration?

39. WSQA Return Examiner 2009 39 Results evaluation: key concepts Strong integration: results presented for Key areas addressing strategic challenges, Key competitive advantages Key customer requirements Not-so-strong integration: Results missing for the above Results presented that the Examiner can’t match to process items or Organizational Profile

40. WSQA Return Examiner 2009 40 “Key” results Refers to elements or factors most critical to achieving the intended outcome In terms of results, look for Those responsive to the Criteria requirements Those most important to the organization’s success Those results that are essential elements for the organization to pursue or monitor in order to achieve its desired outcome

41. Break time!

42. WSQA Return Examiner 2009 Stages of the Assessment Process

43. WSQA Return Examiner 2009 WSQA Consensus: Step-by-step

44. WSQA Return Examiner 2009 WSQA Consensus: Step-by-step

45. WSQA Return Examiner 2009 WSQA Consensus: Step-by-step

46. WSQA Return Examiner 2009 Stage 2 Consensus Webinar http://www.wsqa.net/extimetable.php Comments from all Independent reviews Ensure comments reflect scoring. - what is preventing applicant from moving to next level? Is this explained in the comments 46

47. WSQA Return Examiner 2009 47 Consensus preparation and call(s) Each Examiner responds with agreement, suggestions for changes, or disagreement, stating rationales Category Lead facilitates consensus Team Lead and/or PEPD mentor leads final scoring discussion Can take anywhere from 4-8 hours (depending on what?)

48. WSQA Return Examiner 2009 48 Consensus and conclusion If using online tool during consensus, changes may be made in real time If not, Category Leads update comments & scores after call according to consensus decisions Category Lead notifies Team Lead and Scorebook Editor when done Scorebook Editor compiles consensus scorebook with final edits, notifies WSQA Consensus scorebook should be finalized ASAP

49. WSQA Return Examiner 2009 49 Comment, scoring considerations “Benefit of the doubt” Give credit for what’s in the application Don’t penalize for incidental exceptions Not every process must show complete “DLI” Not every results example must include “TCI”

50. WSQA Return Examiner 2009 50 Comment, scoring considerations For LITE, only one score for entire results Category 7 Propose a score for results related to your categories Be prepared to reach team consensus for all of Category 7 Using scoring language can help the applicant understand comments and score

51. WSQA Return Examiner 2009 51 Examiner training Completing the stage 1 grid Using the gaps and notes tab Comment writing Consensus Meetings Scoring consistency Integration Learning Comparative Data Considerations for small organizations Opportunities

52. WSQA Return Examiner 2009 Baldrige National Quality Program National Examiner Questions 2009 Harry Hertz Responses

53. WSQA Return Examiner 2009 Further clarification of “performance projections” vs “goals” Inclusion of goals in results category – why are they not included in the evaluation? Why are projections expected for all measures in all scoring ranges? Why are projections not in the Results Criteria? Question 1

54. WSQA Return Examiner 2009 Why don’t we align results items with process items? Currently 1.1 is aligned with 7.6 instead of being aligned with 7.1. Question 2 Question 3 l Without comparisons, how do you interpret levels? Without a point of reference, how can an Examiner understand a “level”?

55. WSQA Return Examiner 2009 Question 4 l How are key themes assessed (from applicant’s perspective)? Question 5 l How do we evaluate applicants that are part of a larger system? For example an applicant might say that they use a process established by the system for which they are a part.

56. WSQA Return Examiner 2009 Why don’t we ask applicants to define/describe their work system(s), e.g., in 6.1.a.1, “What is/are your work system(s)?” Question 6 Question7 l Clarify the difference between work systems and work processes. There appears to be overlap and the category is labeled process management.

57. WSQA Return Examiner 2009 Core Competencies

58. WSQA Return Examiner 2009 58 Information resources WSQA comment guideline handout Your back-up, team lead, PEPD mentor Reference materials in Baldrige web site http://baldrige.nist.gov/Examiner_Resources.htm http://baldrige.nist.gov/Examiner_Resources.htm WSQA Website http://www.wsqa.net/extraining.php

59. WSQA Return Examiner 2009 59 What we’ve covered today WSQA overview Baldrige criteria changes WSQA ethics review WSQA Lite versus Full process WSQA process review Questions from National Training

60. WSQA Return Examiner 2009 60 THANK YOU!! Your support and participation as Examiners helps us all by helping WSQA fulfill its mission!