Copyright Minnesota Council for Quality Revision 08/21/06 1 Minnesota Quality Award 2006 Site Visit Training

Copyright Minnesota Council for Quality Revision 08/21/06 2 Today’s Agenda Site Visit Introduction Site Visit Process –Advanced Preparation –Final Preparation –Conducting the Site Visit –Interviewing Techniques –Site Visit Products What Happens Next

Copyright Minnesota Council for Quality Revision 08/21/06 3 Baseline Questions Narrative or Baldrige-Express? Which Criteria? Length of site visit? (Usually 3 days.) Anything unusual?

Copyright Minnesota Council for Quality Revision 08/21/06 4 Purpose of the Site Visit Understand key organizational factors Verify and clarify facts and site visit issues Prepare feedback report Score the items Assess commitment, sustainability, and momentum Prepare and present exit presentation

Copyright Minnesota Council for Quality Revision 08/21/06 5 Elements of a Successful Site Visit People ProcessProducts Roles, Ground rules, Communication Assignments, Controlled agenda, Consensus building Exit presentation, Feedback report, Judges’ presentation

Copyright Minnesota Council for Quality Revision 08/21/06 6 Presentation to Judges Information Flow + =

Copyright Minnesota Council for Quality Revision 08/21/06 7 Evaluator Role & Responsibilities Conduct interviews to gather information Manage and track resolution of site visit issues Revise and update comments Backup other team members as needed Review and comment on all feedback comments Lead scoring discussion for assigned category Provide input to exit presentation and judges presentation Remember that you represent the council!

Copyright Minnesota Council for Quality Revision 08/21/06 8 Team Lead Role and Responsibilities Lead site visit planning meetings and caucuses Coach team members Interface with applicant representative and executive Monitor progress of category owners in closing issues and updating comments Oversee process for team refinement/approval of final comments Oversee preparation and delivery of exit presentation Deliver final feedback report to council

Copyright Minnesota Council for Quality Revision 08/21/06 9 MCQ Role and Responsibilities Ensure communication is occurring between team and applicant Participate in opening presentation Monitor team progress & provide logistical support Monitor applicant satisfaction and concerns Resolve process issues that may arise Moderate the exit presentation

Copyright Minnesota Council for Quality Revision 08/21/06 10 Applicant Rep. Role & Responsibilities Contribute information for findings Arrange availability of interviewees and materials Help team complete its agenda Provides: List of key contacts Organization chart Facility layout Optional requested material See Site Visit Preparation Checklist for Applicants

Copyright Minnesota Council for Quality Revision 08/21/06 11 Today’s Agenda Site Visit Introduction Site Visit Process –Advanced Preparation –Final Preparation –Conducting the Site Visit –Interviewing Techniques –Site Visit Products What Happens Next

Copyright Minnesota Council for Quality Revision 08/21/06 12 Site Visit Sequence Advance Preparation Final Preparation Site Visit 1b. Exit Presentation 1a. Executive Preview 2. Feedback Report 3. Presentation to Judges

Copyright Minnesota Council for Quality Revision 08/21/06 13 Advance Preparation Evaluators - prepare Site Visit Issue worksheets Team leads: –Compile list of documents to be reviewed –Compile list of individuals to be interviewed Team leads and applicant representative – Draft site visit agenda and interview schedule Finalize & confirm logistics for: –Hotel and meeting rooms –Security requirements –Emergency contact number –Name tags (as needed) –Expense accounting

Copyright Minnesota Council for Quality Revision 08/21/06 14 Site Visit Issues Information that is missing or vague Deployment of the practices Cross-cutting issues or themes Site Visit Issues should be written at the basic or overall level Limit SVI’s to 3 – 4 per item The total SVI’s need to be broad enough to verify or clarify all the comments in the consensus report

Copyright Minnesota Council for Quality Revision 08/21/06 15 Example Site Visit Issues Verify and Clarify: Role and activities of senior management (driving customer focus, empowerment, innovation, learning) Degree of involvement and empowerment of employees Reliable and accessible data and information Use of facts in decision-making Focus on customers Training effectiveness and on-the-job reinforcement Use of strategic objectives, action plans, and related measures to align work at all levels—links to performance feedback and employee rewards Evidence of evaluation and improvement Integration of all processes—both core and support Supplier involvement in performance improvement activities Unclear or missing results and continuing trends

Copyright Minnesota Council for Quality Revision 08/21/06 16 Site Visit Issues Worksheet

Copyright Minnesota Council for Quality Revision 08/21/06 17 Site Visit Issues Strategies Acceptable techniques Reviewing data, reports, documents, presentations (hard- copy or electronic) as provided directly by the applicant Interviewing individuals and teams Asking “walk-around questions” to check deployment and communication Unacceptable techniques Conducting focus groups, surveys, or interviews with customers, suppliers, or dealers Anything that disrupts work processes Interviewing sources or collecting data beyond that provided by the applicant This includes conducting library or internet research and visiting web sites

Copyright Minnesota Council for Quality Revision 08/21/06 18 “Typical” Walk-around Questions What are the values, mission, and vision of the organization? What are the goals of your work unit and what role do you play in achieving them? What kind of training do you receive? –Is it useful to your work? –How is training reinforced on the job? How are you empowered to take actions needed to carry out or improve your work? What is recognized or rewarded here? Who are your customers and how do you know what they need?

Copyright Minnesota Council for Quality Revision 08/21/06 19 Site Visit Interview Matrix Individual to InterviewSite Visit Issue Ref. Sheila Knight, Marketing1.1.1, 2.2.1, 3.1.1, 3.1.2, 3.1,3, 3.2.1, 3.2.4, 4.1.1, 4.1.3, 4.2.1, 4.2.2, Ron McDonald, Administration1.1.1, 1.1.3, 1.2.1, 1.2.2, Mary Stewart, Employee Development1.1.1, 2.1.1, 5.1.1, 5.1.3, 5.2.1, 7.3 all Alex Hamilton, Operations1.1.1, 2.1.1, 6.1.3, 6.2.1, 6.2.2, 7.5 all John Jones, Finance4.1.4, 4.2.1, 4.3.2, 7.2 all Leader/manager of Process Improvement Team & members of PI Team Individual responsible for coordinating leadership communication 1.2.5, Individual responsible for Support Services Process Improvement Meetings with Leadership 6.2.5

Copyright Minnesota Council for Quality Revision 08/21/06 20 Today’s Agenda Site Visit Introduction Site Visit Process –Advanced Preparation –Final Preparation –Conducting the Site Visit –Interviewing Techniques –Site Visit Products What Happens Next

Copyright Minnesota Council for Quality Revision 08/21/06 21 Final prep meeting held evening before site visit –Review SV issue worksheets –Review and update documentation list –Review and update interview matrix –Set up Key Themes Storyboard –Establish and schedule final walk-around questions –Finalize the agenda, assignments, pairs Evaluators bring copies of site visit issue worksheets, to prepare a composite set for all team members Handoff issues or questions as needed Applicant representative meets with the team Aid in planning the time on site Share any last minute schedule conflicts Final Preparation

Copyright Minnesota Council for Quality Revision 08/21/06 22 Key Themes Storyboard A key theme is a Strength or OFI that is common to more than one Item or Category, is especially significant in terms of the applicants Key Factors, and/or addresses a Core Value of the Criteria. Provides key points and an overall summary of the team’s evaluation of the applicant. Divided into three sections to address three questions concerning important Strengths, significant OFIs, and key Results Will evolve throughout the site visit Will be used to help prepare the exit presentation

Copyright Minnesota Council for Quality Revision 08/21/06 23 Site Visit Schedule Template

Copyright Minnesota Council for Quality Revision 08/21/06 24 Today’s Agenda Site Visit Introduction Site Visit Process –Advanced Preparation –Final Preparation –Conducting the Site Visit –Interviewing Techniques –Site Visit Products What Happens Next

Copyright Minnesota Council for Quality Revision 08/21/06 25 Site Visit Schedule Day 1 –Opening meeting Introductions to the applicant (MCQ/Team Leads) Applicant presentation (up to one hour) –Facilities tour (optional) All Days –Interview meetings –Walk-around interviews of employees –Team discussions (caucuses) –Meeting with applicant representative and executive at end of day Final Day –Executive preview and exit presentation

Copyright Minnesota Council for Quality Revision 08/21/06 26 Typical Daily Progress (Day 1) Advanced level of understanding of business factors Close some Site Visit Issues (typically Cat. 1 and 2) Establish remaining agenda Confirm the Day 2 schedule will closeout all remaining SVI’s Brief applicant on progress and outstanding issues Category Owners begin editing their comments

Copyright Minnesota Council for Quality Revision 08/21/06 27 Typical Daily Progress (Day 2) Confirm key themes Close 100% of remaining Site Visit Issues Identify any remaining information/interviews needed Finalize plan for Day 3 Brief applicant on progress, outstanding issues, and plans for Day 3 Finalize 100% of the feedback comments

Copyright Minnesota Council for Quality Revision 08/21/06 28 Typical Daily Progress (Day 3) Close remaining issues (if any) Prepare and deliver Executive Preview and Exit Presentation Complete the feedback report Identify any Pretty Good Practices Score items Assess commitment, sustainability, and momentum Complete logistics (hotel checkout, return of applicant materials, collection of shredding, 360 feedback)

Copyright Minnesota Council for Quality Revision 08/21/06 29 Evaluator Team Caucuses Review progress against the schedule Report important findings Handoff notes to category leaders Hand-off any outstanding questions Update SVI worksheets and key themes storyboard Review and update feedback comments Be sure to schedule enough time for caucuses plus some personal break time

Copyright Minnesota Council for Quality Revision 08/21/06 30 Managing Issues and Handoffs Create 7 flip charts, one for each category. Ask Evaluators to use yellow sticky notes to record issues they have to hand off to the category owner in the “open” area and red for new issues. When the information has been collected, record the answer and move the sticky to “closed” portion. This way everyone can keep track of open and closed issues and effectively manage handoffs. Category 1 Open  Closed  Verify deployment between shifts Ask CEO about employee focus (4)

Copyright Minnesota Council for Quality Revision 08/21/06 31 Today’s Agenda Site Visit Introduction Site Visit Process –Advanced Preparation –Final Preparation –Conducting the Site Visit –Interviewing Techniques –Site Visit Products What Happens Next

Copyright Minnesota Council for Quality Revision 08/21/06 32 Interview Structure Begin the interview with: –Introduce yourself and why you’re there We will respect confidentiality of information We will be looking for both Strengths and OFIs –Ask if they have questions about the interview –Explain why you are taking notes –Describe the time limits End the interview with: –Do you have any questions for me? –Thank them for their time and honesty

Copyright Minnesota Council for Quality Revision 08/21/06 33 Effective Interviewing Tips Ask open-ended questions, probe when necessary, bring it to a conclusion Script your lead and follow-up questions Maintain eye contact and relaxed body language One person interviews; backup takes notes Backup may clarify or ask missed point Refocus when they ramble – remind them of time limits Be frank and respectful –Use the applicant’s terminology –Actively LISTEN to the applicant Be patient with shy interviewees

Copyright Minnesota Council for Quality Revision 08/21/06 34 Asking Questions Use open-ended questions Tell me about … How do you … Help me understand … Please walk me through … Avoid Leading questions. Why don’t you – At my office we do it this way. What do you do? Have you always done it this way? Why did you change? How did you know what to change? Don’t you think it would be a good idea to re-engineer your manufacturing process? How do you determine how frequently to check this process? Why do you only audit this process every two years? How do you know customer requirements are correctly defined? Why don’t you involve customers in your design-build teams?

Copyright Minnesota Council for Quality Revision 08/21/06 35 Today’s Agenda Site Visit Introduction Site Visit Process –Advanced Preparation –Final Preparation –Conducting the Site Visit –Interviewing Techniques –Site Visit Products What Happens Next

Copyright Minnesota Council for Quality Revision 08/21/06 36 Prepare Exit Presentation Review the Key Themes Storyboard –Significant Strengths, including Pretty Good Practices –Significant Opportunities for Improvement Prepare the PowerPoint presentation using the Exit Presentation Template Identify presenters

Copyright Minnesota Council for Quality Revision 08/21/06 37 Conduct Executive Preview Applicant’s option Team Leads only Council representative also present Summary review of Exit Presentation content Opportunity for clarifying questions Expression of thanks for cooperation and participation Followed by final modification of the Exit Presentation if required

Copyright Minnesota Council for Quality Revision 08/21/06 38 Deliver Exit Presentation One hour maximum (often shorter) Moderated by MCQ representative Audience determined by applicant Review key themes –Significant Strengths and Pretty Good Practices –Significant Opportunities for Improvement Clarifying Q&A Thanks

Copyright Minnesota Council for Quality Revision 08/21/06 39 Complete the Feedback Category owners prepare final wording of comments Team reviews and signs off on all comments Team scores the items Team identifies Pretty Good Practices Team reviews judges’ questions –Commitment –Sustainability –Momentum Be sure to follow all writing rules!

Copyright Minnesota Council for Quality Revision 08/21/06 40 Score Items Assign scoring facilitator Determine order for scoring items Allow 5-10 minutes maximum per item Review scoring guidelines, terminology (see Clarifying Requirements for Basic-, Overall-, and Multiple-Level Scoring) For each item –Category Owner reviews key points of comments –Team members individually determine scores –Team displays scores on facilitator’s signal –Team negotiates variances until consensus reached –Be mindful of item point values - Don’t bog down over minor points! Record percent scores on appropriate scoring matrix Remember--It’s not over till it’s over

Copyright Minnesota Council for Quality Revision 08/21/06 41 Scoring Calibration Remember from evaluator training? –Few: <15% –Some: 15% < 30% –Many: 30% < 50% –Most: 50% < 80% –Nearly All: 80% <100% –All: 100%

Copyright Minnesota Council for Quality Revision 08/21/06 42 Scoring Diagram

Copyright Minnesota Council for Quality Revision 08/21/06 43 Finalize the Feedback Report After leaving site (within one week) –Team Leads write Executive Summary based on Exit Presentation using Key Themes Worksheet –Feedback editor: collates comments into the appropriate feedback report template (Business, Healthcare, Education) incorporates Executive Summary and Scoring chart distributes report to team for feedback –Team lead forwards finished feedback report to lead judges Next steps (approximately two weeks) –Judges review and edit final report and forward to Council –MCQ forwards finished report to applicant

Copyright Minnesota Council for Quality Revision 08/21/06 44 Judges’ Presentation Agenda Key Factors – 2 minutes Key Themes: Process Strengths – 3 minutes Key Themes: Process OFIs – 3 minutes Key Themes: Results – 6 minutes Scoring Summary – 2 minutes Significant Differences (if any) – 1 minute Commitment, Sustainability, and Momentum – 3 minutes Questions – 10 minutes

Copyright Minnesota Council for Quality Revision 08/21/06 45 Judges’ Presentation - Commitment The extent to which leaders are willing to invest time, resources and energy in making the journey An example of low commitment: leaders spend little time on quality or planning for improvements – few resources are made available for improving quality systems and approaches An example of high commitment: leaders consistently encourage and lead efforts to improve the organization – leaders plan for high levels of performance – leaders allocate resources to the effort

Copyright Minnesota Council for Quality Revision 08/21/06 46 Judges’ Presentation - Sustainability The extent to which the organization is likely to continue making progress on its journey An example of low sustainability: The quality journey may be seen as just a management fad – “something the boss wants us to do.” If the leader of the quality effort were to disappear from the scene, the journey would stop. An example of high sustainability: “Managing quality is everyone’s job. We need to excel as an organization in order to survive and prosper. It’s not just the boss’ thing.” If there were a change in senior leadership, progress would continue.

Copyright Minnesota Council for Quality Revision 08/21/06 47 Judges’ Presentation - Momentum The direction and rate of progress on the quality journey An example of low momentum: There may be slow or no progress overall from year to year, although some pockets of rapid or steady growth may exist. The organization may be little aware of its opportunities for improvement or how to close the gaps. Perhaps, improvements in the past may have been lost and are no longer apparent. An example of high momentum: Innovation and improvement in business practice is frequent and pervasive. Baldrige scores may have been rising each year through multiple cycle of self-assessment and improvements.

Copyright Minnesota Council for Quality Revision 08/21/06 48 Today’s Agenda Site Visit Introduction Site Visit Process –Advanced Preparation –Final Preparation –Conducting the Site Visit –Interviewing Techniques –Site Visit Products What Happens Next

Copyright Minnesota Council for Quality Revision 08/21/06 49 What Happens After the Site Visit Judging to assign a recognition level –Team leads present and respond to judges’ questions –All team members invited to attend and participate –Judges deliberate privately and assign award level Applicant receives feedback report Improvement Planning Session –Facilitated by council –Team leads walk applicant through report –All team members invited to attend and participate Celebration!

Copyright Minnesota Council for Quality Revision 08/21/06 50 Other Tools to Help with Site Visit See the Evaluator Resource Center at: See the Site Visit Packet provided by the council prior to site visit

Copyright Minnesota Council for Quality Revision 08/21/06 51 Thank you for all your time and effort! The MQA couldn’t exist without you!