1. California Team Excellence Award 2012 Judges Training
2012_CTEA_Judges_Training

2. 2012 CTEA Judges Training Outcomes - What Will You Learn?
Your participation is a critical part of encouraging and recognizing team excellence in California!
Thank you for your service!
Outcomes - What Will You Learn?
How the CTEA Award Program works.
How to evaluate and score team presentations.
How to reach consensus among the members of your judging panel.
How to write effective feedback comments to help teams and their organizations improve.
2012_CTEA_Judges_Training

3. Two-Phase Training
This training is required of all CTEA judges for the 2012 cycle, even those who have had prior experience as CTEA judges.
Phase 1 – Perform this self-administered training prior to Phase 2.
Phase 2 – Join one of several scheduled judges conference calls.
Dates are listed in Lesson 4 and on the website:
2012_CTEA_Judges_Training

4. Outline of Phase 1 Training
Lesson 1:
Overview of the CTEA Program and the Judging Activities.
Lesson 2:
How to Score a Team’s Project Presentation.
Lesson 3:
How to Write Effective Feedback Comments.
Lesson 4:
Next Steps – Phase 2.
2012_CTEA_Judges_Training

5. Assemble the Necessary Materials
Before proceeding download all the documents listed below from the CTEA Judges Information webpage so you can refer to them during this training.
CTEA Program Brochure.pdf (on CTEA homepage)
ASQ-ITEA Award Criteria.pdf (on Team Resources page)
ASQ-ITEA Glossary of Terms.pdf (on Team Resources page)
CTEA Judges Workbook.xls
ASQ-ITEA Scoresheet Sample.pdf
ASQ-ITEA Presentation Sample.pdf
CTEA Judges Guidelines.pdf
CTEA Feedback Report Template.doc
ASQ-ITEA Feedback Report Sample.pdf
2012_CTEA_Judges_Training

6. Overview of the CTEA Program and the Judging Activities
Lesson 1
Overview of the CTEA Program
and the Judging Activities
2012_CTEA_Judges_Training

7. State & International Recognition Programs (1)
International Team Excellence Award (ITEA)
This Is administered by the American Society for Quality (ASQ).
There is an annual cycle with a live competition and award ceremony at ASQ’s International conference in May.
California Team Excellence Award (CTEA)
California Council for Excellence (CCE) administers the CTEA.
Several countries and states administer local award programs that generally mirror the international program.
There is an annual cycle with a judging event in October and an awards ceremony the following spring.
2012_CTEA_Judges_Training

8. State & International Recognition Programs (2)
The CTEA Program is officially affiliated with the ITEA Program.
CTEA uses the ITEA criteria for judging.
CTEA also functions as a feeder program in which one or more high-scoring CTEA teams are eligible to compete as finalists in the ITEA Program.
2012_CTEA_Judges_Training

9. CTEA Mission Statement
We promote overall effectiveness through team-focused management.
We provide the training, tools, knowledge, and assessment skills needed to excel through team participation, recognition, and celebration.
2012_CTEA_Judges_Training

10. Review the CTEA Program Brochure
This brochure explains how the CTEA program operates.
Key points are that you will be assessing a team presentation of an improvement project that:
Covers one completed team project from the previous 24 months;
Describes how the team addressed the ITEA criteria during the course of their project;
Lasts no more than 30 minutes;
Is usually a live presentation but can be in DVD form.
2012_CTEA_Judges_Training

11. Benefits of Being a CTEA Judge
Observing use of tools and methods you can apply in your own organization.
Honing your knowledge and skills about what constitutes best practice in team excellence as embodied in the ITEA Criteria.
Networking with fellow judges from various sectors who are highly competent professionals in the application of team- based continuous improvement.
Obtaining recognition from your own organization for achieving the status of CTEA judge.
Being energized by seeing the outstanding results accomplished by these teams as well as the professional and personal impacts these achievements have on team members.
2012_CTEA_Judges_Training

12. Your Tasks as Judges Judges have two primary tasks:
Score the presentation according to the criteria, striving for accuracy so that the scoring is done consistently across judges.
Write actionable and non-prescriptive feedback comments that will be useful to the team and their sponsoring organization.
2012_CTEA_Judges_Training

13. Your Obligations as Judges
Having no conflict of interest with the organization(s)/team(s) that you are judging.
Applying only the ITEA criteria and not applying any other opinions or professional requirements.
Holding the information confidential about which team/organization(s) you judged and the contents of their presentations.
Be willing to reach consensus with your fellow judges by being open to other points of view.
Giving teams the benefit of the doubt.
2012_CTEA_Judges_Training

14. Organization of Judges
The judges are placed into panels of 3-5 people.
The composition is balanced with people of varying degrees of judging experience and varying organizational backgrounds.
Each panel has a lead judge assigned, an experienced judge who will lead the panel’s activities during the day of the judging event.
Coordinates with any follow-up activities.
Completion of final Feedback Report.
2012_CTEA_Judges_Training

15. Judging Event Timeline
During the morning of the judging event, listen to and individually score a team presentation.
At the end of the presentation you have the option of asking questions of the presenters (or review the DVD) for up to 5 minutes.
The questions must only be for clarification or verification of what was said in the presentation relative to the criteria.
They cannot pertain to issues outside the criteria.
Then adjourn to a private location to finish individual scoring, reach consensus on the scores, and then write draft feedback comments.
Note: Some judging panels may be asked to judge two team presentations rather than just one. This will be discussed during the Judges Conference call if need be.
2012_CTEA_Judges_Training

16. Presentation Time Limit - 30 Min.
What happens if the presentation
goes beyond 30 minutes?
The lead judge tells the judging panel to stop scoring at the 30 minute mark but allows the team to finish its presentation.
Anything presented after the 30 minute mark is not considered in judging and scoring.
2012_CTEA_Judges_Training

17. Your Customers The organizations sponsoring the participating teams.
Other judges because achieving consistency across judges is important for accuracy and fairness of assessment.
2012_CTEA_Judges_Training

18. How to Score a Team’s Project Presentation
Lesson 2
How to Score a Team’s Project Presentation
2012_CTEA_Judges_Training

19. ASQ-ITEA Criteria Summary
Read the ASQ-ITEA Award Criteria
The following 5 slides show the summary form of the ASQ-ITEA Award Criteria along with the point scores.
2012_CTEA_Judges_Training

20. CTEA Award Criteria Summary (1)
1. Project Selection and Purpose (9 3 points = 27 points possible)
1A. Explain the methods used to choose the project. (Provide specific examples of techniques and data used.)
a. Describe the types of data and/or quality tools used to select the project, and why they were used.
b. Explain the reasons why the project was selected.
c. Describe the involvement of potential stakeholders in project selection.
1B. Explain how the project supports/aligns with the organization's goals, performance measures, and/or strategies.
a. Identify the affected organizational goals, performance measures, and/or strategies.
b. Identify the types of impact on each goal, performance measure, and/or strategy.
c. Identify the degree of impact on each goal, performance measure, and/or strategy, and how this was determined.
1C. Identify the potential stakeholders (who may be impacted by the project) and explain how they may be impacted by the project.
a. Identify potential internal and external stakeholders and explain how they were identified.
b. Identify the types of potential impact on stakeholders and explain how these were determined.
c. Identify the degree of potential impact on stakeholders and explain how this was determined.
2012_CTEA_Judges_Training

21. CTEA Award Criteria Summary (2)
2. Current Situation Analysis (6 4.5 points = 27 points possible)
2A. Explain the approach/process the team used to identify the potential root cause(s)/improvement opportunity(ies).
a. Describe the methods and tools used to identify possible root causes/improvement opportunities.
b. Describe the team’s analysis of data to identify possible root causes/improvement opportunities.
c. Describe how or if any of the stakeholders were involved in identifying the possible root causes/improvement opportunities.
2B. Describe how the team analyzed information to identify the final root cause(s)/improvement opportunity(ies). (Include any appropriate validation.)
a. Describe the methods and tools used to identify the final root cause(s)/improvement opportunity(ies).
b. Describe the team’s analysis of data to select the final root cause(s)/improvement opportunity(ies).
c. Identify the root cause(s)/improvement opportunity(ies) and explain how the team validated the final root cause(s)/improvement opportunity(ies).
2012_CTEA_Judges_Training

22. CTEA Award Criteria Summary (3)
3. Solution Development (9 3 points = 27 points possible)
3A. Explain the methods used to identify the possible solutions/improvement actions.
a. Describe the methods and tools used to develop possible solutions/improvement actions.
b. Describe the team’s analysis of data to develop possible solutions/improvement actions.
c. Indicate the criteria the team decided to use in selecting the final solution(s)/improvement action(s).
3B. Explain how the final solution(s)/improvement action(s) was/were determined.
a. Describe the methods and tools used by the team to select the final solution(s)/improvement action(s).
b. Describe the team’s analysis of data to select the final solution(s)/improvement action(s).
c. Describe the involvement of stakeholders in the selection of the final solution(s)/improvement action(s).
3C. Explain the final solution(s)/improvement action(s), validation, and the benefits expected to be realized by implementing the team’s solution(s)/improvement action(s).
a. Describe the final solution(s)/improvement action(s) and explain how the team validated the final solution(s)/improvement action(s).
b. List the types of tangible and intangible benefits that are expected to be realized by implementing the team’s solution(s)/improvement action(s).
c. Explain how the team used data to justify the implementation of the team’s solution(s)/improvement action(s).
2012_CTEA_Judges_Training

23. CTEA Award Criteria Summary (4)
4. Project Implementation and Results (9 3 points = 27 points possible)
4A. Explain how buy-in/agreement was achieved for implementation.
a. Indicate the types of internal and external (if applicable) stakeholder involvement in implementation.
b. Describe how various types of resistance were identified and addressed.
c. Explain how stakeholder buy-in was ensured.
4B. Explain the approach used by the team to implement its solution(s)/improvement action(s) and to ensure the results.
a. Describe the plan developed by the team to implement its solution(s)/improvement action(s).
b. Describe the procedure, system, or other changes that were made to implement the solution(s)/improvement action(s) and to sustain the results.
c. Describe the creation and installation of a system for measuring and sustaining results.
4C. Describe the results achieved.
a. Indicate the types of tangible and intangible results that were realized.
b. Explain how the project’s results link with the organization’s goals, performance measures, and/or strategies.
c. Describe how results were shared with stakeholders.
2012_CTEA_Judges_Training

24. CTEA Award Criteria Summary (5)
5. Team/Project Management and Project Presentation
(4 4.5 points = 18 points possible)
5A. Explain how the team members were selected and how they were involved throughout the project.
5B. Explain how the team was prepared to work together in addressing the project.
5C. Explain how the team managed its performance to ensure it was effective as a team.
5D. The team will also be judged on the clarity and organization of its presentation.
2012_CTEA_Judges_Training

25. ASQ-ITEA Criteria in Detail
Read the ASQ-ITEA Criteria in Detail
(pay attention to the details provided for each item and the Notes)
Criteria Summary covered in previous slides.
Criteria in Detail.
The detailed version includes more information about what is expected for each criterion item.
The detailed version also includes Notes for you to use to enable you to do more accurate scoring.
2012_CTEA_Judges_Training

26. Scoring the Team Presentation
Judges evaluate the extent to which each item in the Criteria is addressed in the presentation and assign one of the four possible scores:
Not Covered (0 Points)
Totally missing (No approach evident)
Unclear (1 or 1.5 Points)
Touched upon, but not clear. Not enough information is provided to determine if the team’s approach met, or could meet, the criterion requirements. (Approach evident)
Meets Criteria (2 or 3 Points)
Sufficient information is provided to determine that the team’s approach met the criterion requirements. (Approach applied with results evident)
Exceeds (3 or 4.5) Points
The team’s approach goes beyond meeting the criterion and provides additional clarity indicating increased accuracy in the team’s analysis, actions, and/or conclusions. Integration with other criterion items is apparent and enhances the team’s overall results. A “Best Practice” or “Role Model” approach. (Approach broadly integrated and evident)
2012_CTEA_Judges_Training

27. Scoring Tips (1)
Note that “Exceeds” scoring level descriptions are cumulative.
What is looked for in “Exceeds” must include what is in “Meets”.
The scoring level of “Unclear” is when not enough information is provided to determine if the team’s approach met or could meet the criteria requirements.
The scoring level of “Not Covered” is not noted in the guidelines because it receives “0” points.
These are meant to be general guidelines for scoring to serve as a standard for evaluating team presentations.
All teams are still encouraged to be as creative and as innovative as possible in sharing their stories.
2012_CTEA_Judges_Training

28. Scoring Tips (2)
Criteria 2Aa, 2Ba, 3Aa, and 3Ba ask the team to describe their tools/methods.
Criteria 2Ab, 2Bb, 3Ab, and 3Bb ask the team to describe their analysis of data.
There are 13 criteria with two parts.
If you rate the two parts at different levels, you must score it at the lower level.
For example, if one part is “Meets” and the second part is “Exceeds”, the item is scored at “Meets”.
2012_CTEA_Judges_Training

29. Read the ASQ-ITEA Glossary of Terms
Scoring Tips (3)
Read the ASQ-ITEA Glossary of Terms
ASQ-ITEA has constructed a Glossary of Terms that is very important for judges.
In particular be sure to study the information given for “Describe” because each of the four words listed there requires a different level of information.
Also be sure to study the term “Integration” because teams that demonstrate this characteristic should score better and are more likely to attain the “Exceeds” level than those that do not.
Integration explained on the next two slides.
2012_CTEA_Judges_Training

30. Integration Explained (1)
The Criteria in Detail draw attention to where an item detail has a connection (cross-linkage) to other criterion item(s).
The Criteria show that addressing a criterion item detail in one area has an impact on one or more other criterion areas.
How the team addresses these cross-linkages indicates the degree of integration within the team project, stakeholder, and organization direction.
This is a significant factor for a team to score at the “Exceeds” level.
2012_CTEA_Judges_Training

31. Integration Explained (2)
“1Cb. Identify the types of potential impact on stakeholders and explain how these were determined.
Stakeholders may be impacted in many different ways by the project, from quality to timeliness.
Demonstrating that the team understands how the project may have different impacts on the different stakeholder groups is important.
As with 1Bb (type of goal impact), the impact is implied to be directional.
NOTE: This section could later tie to 4Ab, as correctly predicting potential negative impacts early in the project can help the team prepare to overcome resistance later.
The Cross-Linkage in 1Cb is with 1Bb and 4Ab, if addressed per the criteria, would indicate there is integration in the team’s approach across the project and/or organization.
2012_CTEA_Judges_Training

32. Review the CTEA Judges Workbook (all 3 tabs)
Scoring Guidelines
Review the CTEA Judges Workbook (all 3 tabs)
During the team presentation, each judge uses their Judges Worksheet to take notes and to evaluate the presentation.
You will have the forms in all 3 tabs given to you at the judging event.
2012_CTEA_Judges_Training

33. Judges Worksheet (Tab 1 in the Judges Workbook)
This is the Judges Worksheet to be used during the teams presentation.
Refer to the worksheet during the independent review and during consensus with your team.
2012_CTEA_Judges_Training

34. Reaching Consensus (1)
After doing your individual scoring, the judges must reach consensus on their scores across the panel.
A panel member collects and posts all scores to the Consensus Score Sheet.
No consensus discussion is necessary on an item if there is only one scoring level deviation among the judges’ scores.
2012_CTEA_Judges_Training

35. Review the Consensus Scoresheet (Tab 2 of the Judges Workbook)
Reaching Consensus (2)
Review the Consensus Scoresheet
(Tab 2 of the Judges Workbook)
If there is more than one scoring level deviation, the panel members must adjust their individual scores to be within one deviation of each other (example on next slide).
No judge can have a score that deviates two levels from another judge.
This is done through dialogue about reasons for the respective scoring.
Each judge supports their rational for the scoring and discuss as a group.
Then the judges’ scores are averaged on the Consensus Scoresheet.
2012_CTEA_Judges_Training

36. Consensus Scoresheet Sample
This is sample of an initial Consensus Scoresheet.
Notice 2 scoring level deviation on items 1B.b and 1C.b.
2012_CTEA_Judges_Training

37. Final Scoresheet Sample Tab 3 in Judges Workbook
Review the ASQ-ITEA Scoresheet Sample
The results of the Consensus Scoresheet are automatically populated in the Final Scoresheet.
Review the CTEA Final Scoresheet (Tab 3).
2012_CTEA_Judges_Training

38. Familiarizing Yourself with a Presentation
Review the ASQ-ITEA Presentation Sample
The final step in understanding scoring is to familiarize yourself with what a presentation looks like.
While each team presentation is different, one is provided as part of this training package and available on the CTEA website.
2012_CTEA_Judges_Training

39. How to Write Effective Feedback Comments
Lesson 3
How to Write Effective Feedback Comments
2012_CTEA_Judges_Training

40. Purpose of Feedback Reports
A valuable tool teams and their organizations can use to take action.
Celebrate Strengths
Consider Suggestions for Improvement (SFIs)
The feedback report has two components.
Total Score (from judges consensus)
Feedback Comments
Strengths
Suggestions for Improvement
2012_CTEA_Judges_Training

41. Qualities of ‘Well Written’ Feedback Comments
Read the CTEA Judges Guidelines
Succinct, specific narrative.
Constructive and actionable.
Clear to the team why included.
Complete sentences w/o grammar errors.
Non-prescriptive.
Address important elements.
Supports score and evaluation.
2012_CTEA_Judges_Training

42. Do’s & Don’ts of Comment Writing
Use Facts w/o judgment.
Be specific about what is missing (‘it is not clear’ comments).
Use Criteria language and link to examples in the team’s presentation.
Be impactful and answer the ‘so what’.
Use Opinions.
Be prescriptive (Avoid “could, should or would”).
Be overly technical in your writing so that the team is unable to understand feedback.
2012_CTEA_Judges_Training

43. Feedback Comments Should Have the Following Items
Number of the Criteria item.
Number on the presentation slide(s) that the comment addresses.
Language of the criteria item in the form of a brief quote or paraphrase from the item.
Language of the team in the form of a brief quote or paraphrase from the relevant slide(s).
2012_CTEA_Judges_Training

44. Adhere to Style Guidelines
Use a polite, professional, positive tone.
Use active voice, present tense.
Use the team's terminology.
Avoid jargon or acronyms except when used by the team.
Refer to “the team”, not name.
Reference figure numbers when appropriate.
2012_CTEA_Judges_Training

45. Determining What to Emphasize in Feedback Comments
Strength Comments should be the highest scoring items.
SFI Comments should be the lowest scoring.
Comments on integration should only be written if there are noticeable issues with respect of cross-linkages (either as Strengths or SFIs).
2012_CTEA_Judges_Training

46. Examples: Well Written Strength Comments
“The presentation effectively identified the internal and external stakeholders and clearly assigned rated measures to show how much they will be impacted by the project (Slide 17)(1Cb).”
“Excellent information was provided explaining why this project was selected (Slide 3). Six sigma tools were used to justify the project selection and were clearly explained and illustrated in slides 6 – 9. Additionally the use of data to substantiate this project’s selection was highly informative.”
2012_CTEA_Judges_Training

47. Example: Poorly Written Comment
“The team did a good job when they presented their project.”
Issues with Strength Comment:
No support with examples to let the team know what was good.
Does not reinforce practices.
Does not describe the impact.
Does not relate to the criteria.
Not in present tense, active voice.
2012_CTEA_Judges_Training

48. ‘Tips’ for SFI Opening Statements
Although the team addresses/states/indicates…
While the team demonstrated … it is not clear/apparent/evident how/what the team...
The team does not address/report how/what…
2012_CTEA_Judges_Training

49. Examples: Well Written SFI Comments
“The team clearly explained the types of impact and the degree of impact of the project on both internal and external stakeholders; however in order to strengthen this item the judges were searching for information on “how the scores and measurable data were obtained.” (1Cb & 1Cc).”
“While the team showed how they used six sigma tools DMADV and SIPOC to identify potential stakeholders; there was no evidence provided to describe how these potential stakeholders were involved in the selection of the project (Slide10).”
2012_CTEA_Judges_Training

50. Example: Poorly Written SFI Comment
“It was impossible for us to tell what the team achieved. Goals and measures were glaringly omitted. Without an action plan that includes goals it will be impossible for the team to achieve meaningful results. The organization should be using the balanced scorecard to help them.”
Issues with Poorly Written SFI Comment:
Critical tone “glaringly omitted”.
So what is opinion, not fact “Without an action plan that includes goals it will be impossible for the team…”.
Prescriptive.
No linkage to criteria.
2012_CTEA_Judges_Training

51. Refining Your Comments
What is the real strength or SFI?
Is it clear?
What team practices in the comment support the Strength/SFI?
Is the “so what” linked to the main point of the comment?
Is the “so what” factual?
2012_CTEA_Judges_Training

52. Feedback on Integration
Earlier the need to include integration (or lack thereof) in the scoring was mentioned
If the team showed a high score due to integration where the Criteria call for a cross- linkage, then you should write a Strength feedback comment to that effect
On the other hand, if the team showed a direct lack of integration where the Criteria call for a cross-linkage, then you should write an SFI comment to that effect
2012_CTEA_Judges_Training

53. Reaching Consensus on Feedback Comments
How the judging panel does the work of writing draft feedback comments is up to the panel, and the lead judge will provide suggestions and guidance.
There is a requirement for the judges to reach consensus on their feedback comments.
2012_CTEA_Judges_Training

54. Supporting Materials for Writing Feedback Comments
Review the CTEA Feedback Report Template
Read the ASQ-ITEA Feedback Sample
The two supporting documents named above are provided to you for writing feedback.
The first is an actual feedback report for the ASQ- ITEA presentation provided earlier (HD Supply).
The second is a template you will be provided at the judging event.
2012_CTEA_Judges_Training

55. Next Steps – Phase 2 Lesson 4 2012_CTEA_Judges_Training

56. Schedule (1)
Attend one of the following judge conference calls using the call-in information: x
Tuesday, September 18, 8: :30 am
Wednesday, September 19, 12:30 - 2:30 pm
Thursday, September 20, 8: :30 am
(Back-up Call) Tuesday, September 25, 8: :30 am
We will answer questions regarding the process that you may have at that time.
We will discuss logistical issues, locations, schedules, etc.
We will verify that you have all documentation.
2012_CTEA_Judges_Training

57. Schedule (2) Attend your assigned judging event(s):
October 2, 2012: Los Angeles - Good Samaritan Hospital
(Site Coordinator: John Corbett)
October 3, 2012: San Diego - Sharp HealthCare
(Site Coordinator: Rusty Robinson)
October 4, 2012: Sacramento - National University
(Site Coordinator: Ron Bane)
Respond to your lead judge’s request for comments on the final draft of the feedback report within 1 week of receipt.
The Lead Judge will submit the final feedback report to the CTEA Council Chair by October 21, 2012.
2012_CTEA_Judges_Training

58. Schedule (3)
The CTEA Council Chair provides the feedback reports to the teams and their sponsoring organizations:
November 14, 2012
CTEA Recognition and Awards Ceremony:
April 19-20, 2013
ASQ ITEA Finalist Presentations and Awards Ceremony, ASQ World Conference:
May 6-8, 2013
2012_CTEA_Judges_Training

59. Thank you for taking the
Conclusion
Thank you for taking the
2012 CTEA Judges Training
See you in October!
2012_CTEA_Judges_Training