1. California Team Excellence Award (CTEA) 2013 Judges Training

2. 2013 CTEA Judges Training Outcomes - What Will You Learn?
Your participation is a critical part of encouraging and recognizing team excellence in California!
Thank you for your service!
Outcomes - What Will You Learn?
How the CTEA Award Program works.
How to evaluate and score team presentations.
How to reach consensus among the members of your judging panel.
How to write effective feedback comments to help teams and their organizations improve.
2012_CTEA_Judges_Training

3. Two-Phase Training
This training is required of all CTEA judges for the 2013 cycle, even those who have had prior experience as CTEA judges.
Phase 1 – Perform this self-administered training prior to Phase 2.
Phase 2 – Join one of several scheduled judges conference calls.
Dates are listed in Lesson 4 and on the website:
2012_CTEA_Judges_Training

4. Outline of Phase 1 Training
Lesson 1:
Overview of the CTEA Program and the Judging Activities.
Lesson 2:
How to Score a Team’s Project Presentation.
Lesson 3:
How to Write Effective Feedback Comments.
Lesson 4:
Next Steps – Phase 2.
2012_CTEA_Judges_Training

5. Assemble the Necessary Materials
Before proceeding download all the documents listed below from the CTEA Judges Information webpage so you can refer to them during this training.
CTEA Program Brochure.pdf (on CTEA homepage)
ASQ-ITEA Award Criteria.pdf (on Team Resources page)
ASQ-ITEA Glossary of Terms.pdf (on Team Resources page)
CTEA Judges Workbook.xls
ASQ-ITEA Scoresheet Sample.pdf
ASQ-ITEA Presentation Sample.pdf
CTEA Judges Guidelines.pdf
CTEA Feedback Report Template.doc
ASQ-ITEA Feedback Report Sample.pdf
2012_CTEA_Judges_Training

6. Overview of the CTEA Program and the Judging Activities
Lesson 1
Overview of the CTEA Program
and the Judging Activities
2012_CTEA_Judges_Training

7. State & International Recognition Programs (1)
International Team Excellence Award (ITEA)
This Is administered by the American Society for Quality (ASQ).
There is an annual cycle with a live competition and award ceremony at ASQ’s International conference in May.
California Team Excellence Award (CTEA)
California Council for Excellence (CCE) administers the CTEA.
Several countries and states administer local award programs that generally mirror the international program.
There is an annual cycle with a judging event in October and an awards ceremony the following spring.
2012_CTEA_Judges_Training

8. State & International Recognition Programs (2)
Since 1999, the CTEA Program has been an authorized showcase of the ITEA program and uses the ITEA criteria for judging.
CTEA also functions as a feeder program in which one or more high-scoring CTEA teams are eligible to compete as finalists in the ITEA Program.
The program criteria provide a time proven Best Practice for planning, management, and measuring the performance of Problem Solving and Process Improvement teams.
2012_CTEA_Judges_Training

9. CTEA Value
CTEA Mission: We promote overall effectiveness through team- focused management, providing the training, tools, knowledge, and assessment skills needed to excel through team participation, recognition, and celebration.
This is a great product for both management and teams to use to improve and move to their next level of performance.
2012_CTEA_Judges_Training

10. Review the CTEA Program Brochure
This brochure explains how the CTEA program operates.
Key points are that you will be assessing a team presentation of an improvement project that:
Covers one completed team project from the previous 24 months;
Describes how the team addressed the ITEA criteria during the course of their project;
Lasts no more than 30 minutes;
Is usually a live presentation but can be in DVD form.
2012_CTEA_Judges_Training

11. CTEA Recognizes Teams at Two Levels - Gold Nugget and Gold Rush -
Gold Nugget Award Level - Full extent of the criteria applied at the section, item, and item detail question level (5 Sections, 37 Items & 47 item questions to address).
Gold Rush Award Level - Use of the criteria applied at the section and item level (5 Sections & 26 item questions to address).
This training prepares you to judge at the Gold Nugget Level. Once you become proficient, you may be requested to judge Gold Rush Level applications as well.
2012_CTEA_Judges_Training

12. Benefits of Being a CTEA Judge
Observing use of tools and methods you can apply in your own organization.
Honing your knowledge and skills about what constitutes best practice in team excellence as embodied in the ITEA Criteria.
Networking with fellow judges from various sectors who are highly competent professionals in the application of team- based continuous improvement.
Obtaining recognition from your own organization for achieving the status of CTEA judge.
Being energized by seeing the outstanding results accomplished by these teams as well as the professional and personal impacts these achievements have on team members.
2012_CTEA_Judges_Training

13. Your Tasks as Judges Judges have two primary tasks:
Score the presentation according to the criteria, striving for accuracy so that the scoring is done consistently across judges.
Write actionable and non-prescriptive feedback comments that will be useful to the team and their sponsoring organization.
2012_CTEA_Judges_Training

14. Your Obligations as Judges
Having no conflict of interest with the organization(s)/team(s) that you are judging.
Applying only the ITEA criteria and not applying any other opinions or professional requirements.
Holding the information confidential about which team/organization(s) you judged and the contents of their presentations.
Be willing to reach consensus with your fellow judges by being open to other points of view.
Giving teams the benefit of the doubt.
2012_CTEA_Judges_Training

15. Organization of Judges
The judges are placed into panels of 3-5 people.
The composition is balanced with people of varying degrees of judging experience and varying organizational backgrounds.
Each panel has a lead judge assigned, an experienced judge who will lead the panel’s activities during the day of the judging event.
Coordinates with any follow-up activities.
Ensures completion of final Feedback Report.
2012_CTEA_Judges_Training

16. Judging Event Timeline
During the morning of the judging event, listen to and individually score a team presentation.
At the end of the presentation you have the option of asking questions of the presenters (or review the DVD) for up to 5 minutes.
The questions must only be for clarification or verification of what was said in the presentation relative to the criteria.
They cannot pertain to issues outside the criteria.
Then adjourn to a private location to finish individual scoring, reach consensus on the scores, and then write draft feedback comments.
Note: Some judging panels may be asked to judge two team presentations rather than just one. This will be discussed during the Judges Conference call if need be.
2012_CTEA_Judges_Training

17. Presentation Time Limit - 30 Min.
What happens if the presentation
goes beyond 30 minutes?
The lead judge tells the judging panel to stop scoring at the 30 minute mark but allows the team to finish its presentation.
Anything presented after the 30 minute mark is not considered in judging and scoring.
2012_CTEA_Judges_Training

18. Your Customers The organizations sponsoring the participating teams.
Other judges because achieving consistency across judges is important for accuracy and fairness of assessment.
2012_CTEA_Judges_Training

19. The ITEA/CTEA Award Criteria
Lesson 2
The ITEA/CTEA Award Criteria
2012_CTEA_Judges_Training

20. Criteria Sections & Score Point Values (Same for Both Problem Solving & Process Improvement Teams)
Project Selection Purpose
9 3 points = 27 points possible
Current Situation Analysis
6 4.5 points = 27 points possible
Solution Development
Projection Implementation and Results
Team/Project Management & Project Presentation
4 4.5 points = 18 points possible
Total possible points 126
2012_CTEA_Judges_Training

21. Scoring
The judges evaluate how well the team’s presentation addresses each criterion. Judges decide if the team Exceeds, Meets, provides Unclear evidence, or did Not Cover the item at all. Scoring is as shown below.
2012_CTEA_Judges_Training

22. Scoring, Cont’d 2012_CTEA_Judges_Training

23. Criteria Levels of Detail Section – Item – Item Detail 3 Levels of detail example:
1, 2 &3 Levels of Criteria Detail Level 1 – Blue; Level 2- Green; Level 3 - Red
Section 1 - Project Selection/Purpose (9 3 points = 27 points possible)
Project Selection/Purpose gives the team a chance to talk about the importance of the project to the organization, potential project stakeholders, and the team itself.
1A. Guiding Question: How was the project selected? Either the team or management selected the project. In either case, the process used to select the project must be clear and well stated. As with all team responses, providing specific examples of techniques and data used will strengthen the team’s presentation.
2012_CTEA_Judges_Training

24. Criteria Levels of Detail, Cont’d
1Aa. (1) Describe how data and/or tools were used to support the selection, even if the project was assigned to the team, and
(2) Explain why these specific tools and/or data were used to select the project.
1Ab. (1) Explain how the project was selected, and
(2) Explain why the project was selected: what discerned gap or observed opportunity lead to the launching of this project?
NOTE: 1Aa addresses the tools and data used. The emphasis in 1Ab is on the decisions made because of the tool usage/data analysis in 1Aa.
1Ac. Describe the potential stakeholders for the project and how they participated in the selection process. If the stakeholders were not directly involved, then describe how their interests were known and taken into account.
NOTE: this should reflect the history of the project. The actual/final stakeholders may be different from the potential stakeholders at the time the project was being defined. The goal here is to be sure that the project was being defined. The goal here is to be sure that the project was not selected in a vacuum without an understanding of the organizational environment.
2012_CTEA_Judges_Training

25. Purpose of the Notes
Note: The Italicized words supply supplemental notes. These notes are not part of the criteria; rather they are to help guide the team through the process.
2012_CTEA_Judges_Training

26. Descriptors
The “descriptors” are terms such as “Identify” (what), “describe” (how), and “explain” (the why).
The descriptors are in bold type, and they let the team know the level of detail with which they must address each item.
2012_CTEA_Judges_Training

27. Two-part Criteria
When there is more than one descriptor, the criterion is a two-part criterion. Each two-part criterion is clearly marked with a (1) and a (2).
Example:
1Aa. (1) Describe how data and/or tools were used to support the selection, even if the project was assigned to the team, and
(2) Explain why these specific tools and/or data were used to select the project.
1Ba. Identify the affected organizational goals, performance measures, and/or strategies.
Two-part items must be scored Unclear if a team Meets only one part, regardless of how well the other part was addressed.
2012_CTEA_Judges_Training

28. Types of Teams
There are many types of teams (Lean, Improvement, Six Sigma, Innovation, and Quality Circle, to name a few). Regardless of the type of team you are or of any team methodology your company may use, teams always focus on one of two things - either Problem Solving or Process Improvement.
Problem Solving teams have a “problem” to solve. Typically these teams will focus on identifying and eliminating or overcoming “root causes.”
Process Improvement teams have a new process to develop or an existing process that needs to be improved. Typically these teams will focus on identifying and optimizing key process parameters to design or improve an overall process.
Sometimes a team may get confused by the words they use to describe their project. For example, they may say, “We had a problem: our customers were demanding quicker turn-around time and we needed to improve our processes.” This is clearly a case where a Process Improvement is needed. There would be no expectation of a Root Cause unless something had recently caused the turn-around time to get suddenly worse.
2012_CTEA_Judges_Training

29. Changes to Prior CTEA Criteria
There are no changes as to what each criterion is asking to be addressed and the criteria section(5) quantity (37) and order remain the same. All criteria retain their respective scoring values; total possible score remains at 126.
Major adjustment is a reformatted document to improve the distinction between the criteria and supporting information. To simplify the sentence structure, the document was split into a version for each of the two focus paths – Problem-Solving and Process Improvement.
Recognized 1Ab as a two part criterion item. The criteria always had the How and Why requirement, but the criteria summary left the Why part off. That brings the total number of two part criteria to 14.
Minor sentence clarification to facilitate correct interpretation of the criteria and ensure consistent use of the primary descriptors: List/Identify, Describe, and Explain.
2012_CTEA_Judges_Training

30. Changes to Prior Criteria, Cont’d
Some minor languages differ. For example in item 2B the problem solving version reads:
2Ba. Describe the methods and tools the team used to narrow down the list of potential root causes (identified in 2A), to the final root cause(s).
2Bb. Describe the team’s analysis of data to select the final root cause(s). Provide clear, specific examples of what data were analyzed (from the tools/methods described in 2Ba), and how they were analyzed, to help identify the final root cause(s).
2Bc. (1) Identify what the root cause(s) was/were, and
(2) Explain how the team validated that the final root cause(s) determined in 2Bb was/were in fact the final root cause(s). What tools and/or methods did the team use to validate (prove) to themselves (and others if needed) that they identified the true root cause(s)?
The process improvement model reads: 
2Ba. Describe the methods and tools the team used to narrow down the list of potential improvement opportunity(ies) (identified in 2A), to the final improvement opportunity (ies).
2Bb. Describe the team’s analysis of data to select the improvement opportunity(ies). Provide clear, specific examples of what data were analyzed (from the tools/methods described in 2Ba), and how they were analyzed, to help identify the final improvement opportunity(ies).
2Bc. (1) Identify what the improvement opportunity(ies) was/were, and
(2) Explain how the team validated that the final improvement opportunity(ies) determined in 2Bb was/were in fact the final improvement opportunity(ies). What tools and/or methods did the team use to validate (prove) to themselves (and others if needed) that they identified the true improvement opportunity(ies)
2012_CTEA_Judges_Training

31. Criteria for Problem Solving Teams
2012_CTEA_Judges_Training

32. Section 1. Project Selection/Purpose (9 items @ 3 points = 27 points possible)
Project Selection/Purpose gives the team a chance to talk about the importance of the
project to the organization, potential project stakeholders, and the team itself. Section 1
is divided into three sub-sections: (1A) allows the team to discuss how the project was
selected. (1B) provides an opportunity for the team to demonstrate the importance of the
particular project to the organization. (1C) gives the team a chance to discuss how potential
stakeholders were selected and how those stakeholders might be impacted.
1.A.Guiding Question: How was the project selected? Either the team or management selected the project. In either case, the process used to select the project must be clear and well stated. As with all team responses, providing specific examples of techniques and data used will strengthen the team’s presentation.
1Aa. (1) Describe how data and/or tools were used to support the selection, even if the project was assigned to the team, and
(2) Explain why these specific tools and/or data were used to select the project.
1Ab. (1) Explain how the project was selected, and
(2) Explain why the project was selected: what discerned gap or observed opportunity lead to the launching of this project?
NOTE: 1Aa addresses the tools and data used. The emphasis in 1Ab is on the decisions made because of the tool usage/data analysis in 1Aa.
1Ac. Describe the potential stakeholders for the project and how they participated in the selection process. If the stakeholders were not directly involved, then describe how their interests were known and taken into account.
NOTE: this should reflect the history of the project. The actual/final stakeholders may be different from the potential stakeholders at the time the project was being defined. The goal here is to be sure the project was not selected in a vacuum without an understanding of the organizational environment.
2012_CTEA_Judges_Training

33. 1B. Guiding question: Why is this project important to the organization? A good project will support, or at least align with, one or more of the organization's goals, performance measures, and/or strategies. Organizational goals can vary in type and style, from broad vision or mission statements to very specific objectives related to their goals and/or strategies. These items are looking for the link between the team’s project and those goals and/or strategies.
1Ba. Identify the affected organizational goals, performance measures, and/or strategies.
Note: Goals may be present at various levels in an organization and the team is being asked to identify the goals/performance measures impacted by their project. If the team’s project is linked to a lower-level goal, it may be helpful for the team to demonstrate the linkage between the project and the lower-level goal as well as the linkage between the lower-level goal and the organizational-level strategy or goal.
1Bb. Identify types (WHAT) of impact the project will have on each goal, performance measure, and/or strategy presented in 1Ba. NOTE: “Type” implies a directional impact of some sort, such as increased/decreased costs, efficiencies, safety, associate satisfaction, etc.
1Bc. (1) Identify the degree of impact (HOW MUCH) the project will have on each goal, performance measure, and/or strategy, and
(2) Explain how the team quantified the predicted degree of impact. If the team provided relative degrees of impact such as high, medium, or low, they should provide their rationale for deciding “high” vs. “medium” or “low.”
2012_CTEA_Judges_Training

35. Section 2. Current Situation Analysis (6 items @ 4
Section 2. Current Situation Analysis (6 4.5 points = 27 points possible)
Current Situation Analysis focuses on how the team analyzed the current situation.
This includes the processes, data, and information. This section also covers how
stakeholders were involved. The second half of this section covers how the team
determined and validated their final root cause(s). If the organization uses a specific
methodology as part of the quality process, the team may want to consider sharing
how that methodology was applied to identify the possible and final root causes.
Guiding question: How did the team identify the potential root cause(s)? A high-performing team will explore multiple explanations for their current situation, not assuming that the first explanation is correct. 2Aa. Describe the methods and tools used to identify possible root cause(s). Teams may use a variety of approaches/processes to identify the possible root causes of a problem they face. 2Ab. Describe the team’s analysis of data to identify possible root cause(s). Provide clear, specific examples of what data were analyzed (from the tools/methods described in 2Aa), and how they were analyzed, to help identify the possible root cause(s). 2Ac. Describe how the stakeholders were involved in identifying the possible root cause(s). Provide supporting information. NOTE: “Involved” does not necessarily mean how stakeholders participated directly in the process of selecting the root causes. “Involved” can be addressed by describing how the team took the interests of the stakeholders into account in determining the possible root causes.
2012_CTEA_Judges_Training

36. Guiding question: How did the team analyze information to identify the final root cause(s)? Having identified a variety of possible root causes, in 2A, these items give the team a chance to share how they went about narrowing down the possibilities to identify the true root cause(s) for this project. As is true elsewhere, by sharing any appropriate validation techniques, the team could strengthen their presentation
2Ba. Describe the methods and tools the team used to narrow down the list of potential root causes (identified in 2A), to the final root cause(s). 2Bb. Describe the team’s analysis of data to select the final root cause(s). Provide clear, specific examples of what data were analyzed (from the tools/methods described in 2Ba), and how they were analyzed, to help identify the final root cause(s). 2Bc. (1) Identify what the root cause(s) was/were, and (2) Explain how the team validated that the final root cause(s) determined in 2Bb was/were in fact the final root cause(s). What tools and/or methods did the team use to validate (prove) to themselves (and others if needed) that they identified the true root cause(s)?
2012_CTEA_Judges_Training

37. Section 3 Solution Development (9 3 points = 27 points possible) Solution Development allows the team to demonstrate how they logically moved from the causes to their final choice. Initially the team should share how they developed a set of possible root causes. From there, the team should share how t hey arrived at their specific solution(s) that would meet the project objective(s). Finally, the team should share how they convinced themselves and others that their proposals would be both practical and acceptable to the organization.
Guiding questions: How did the team identify the possible solutions? What criteria did the team decide on for selecting their final solutions? An excellent team is one that explores multiple options to arrive at the best one. They will also have specific, supportable criteria against which to judge those options.
3Aa. Describe the tools and methods the team used to identify the possible solutions for the root cause(s) identified and verified in 2B. There may be a number of ways to address the root cause(s). What tools and/or methods did the team use to identify these possible solutions?
3Ab. Describe the team’s analysis of data to develop possible solutions. Provide clear, specific examples of what data were analyzed (from the tools/methods described in 3Aa), and how they were analyzed, to help identify the possible solutions.
3Ac. Identify the criteria the team decided to use to assess the relative effectiveness of the various possible solutions they identified in 3Aa and 3Ab above.
NOTE: The criteria should help the team narrow down the list of possible solutions to enable them to identify the best solution(s).
2012_CTEA_Judges_Training

38. 3B. Guiding questions: How did the team select/determine the final solution(s) from the possibilities generated in 3A above? How did they assure that their final choice(s) met the needs of the stakeholders? Having identified a variety of possible solutions, the team should share what lead them to decide on their final approach. 3Ba. Describe the methods and tools used by the team to select the final solution(s). Now that the team has identified the criteria they used in selecting the final solution(s) item asks the team to describe the tools and methods they used along with the criteria identified in 3Ac above to select the final solution(s). 3Bb. Describe what data were analyzed, and how they were analyzed to help identify the final solution(s). NOTE: The team described the methods and tools they used to determine the final solution(s) in 3Ba; 3Bb is now asking the team to describe what data were analyzed and how they were analyzed, to support the selection of their final solution(s). 3Bc. Describe how stakeholders were involved in determining the final solution(s). NOTE: If the stakeholders were directly involved in the selection of the final solution(s), the team should describe how they were involved in the process; if they were not directly involved, the team should describe how the stakeholders’ interests and concerns were considered.
2012_CTEA_Judges_Training

39. 3Cb. List the benefits that may be realized from the solution(s).
3C. Guiding questions: How did the team validate the final solution(s)? What benefits does the team expect the organization to realize once the solution(s) is/are implemented? How did the team use data to justify the implementation? “Validation,” is not to be confused with “justification.” “Validation” is simply verifying that the solution(s), if implemented, will solve the problem(s)/address the root cause(s) found in 2Bc. “Justification” assumes that the solutions are valid, and that, in addition, they are a good “value” to the organization. Both validation and justification need to occur before the implementation of the final solution. They should not be based on data generated after implementation.
3Ca. (1) Describe the solution(s) the team felt would be most appropriate, and then
(2) Explain how they validated that the solution(s) would solve the problem prior to actually proceeding with the implementation of the solution(s).
3Cb. List the benefits that may be realized from the solution(s).
(1)Tangible benefits include those benefits that lend themselves to being represented by improvements in specific measures or metrics–cycle time, number of defects, production, etc.- -and
(2) Intangible benefits include those that may not easily be linked to a specific measure or measurement, but might require demonstration through improvements in a variety of related indicators–morale, employee satisfaction, etc. Teams should attempt to describe what indicators they are looking at when identifying their intangible benefits– what did they see that led them to believe there could be an improvement in a given intangible benefit?
2012_CTEA_Judges_Training

40. 3Cc. Explain how the team used data to justify the implementation of the solution(s) they determined to be best in 3B. The team should present the data and information they used to demonstrate the value to the organization in implementing their solution(s).
2012_CTEA_Judges_Training

41. Section 4. Project Implementation and Results (9 3 points = 27 points possible) Project Implementation and Results addresses how the team sought and secured buy-in. In addition, this section gives the team the chance to discuss the approaches they used to plan for and implement their solution(s). Finally, this is where the team gets to present the results that were achieved. This should be based on data generated after implementation of the solution(s).
Guiding questions: How were internal and external stakeholders involved in
implementation? How did the team go about securing buy-in, identifying resistance,
and convincing themselves (and others) that buy-in did, in fact, exist?
4Aa. Indicate how internal and external (as applicable) stakeholders were involved
in implementing the solution(s).
4Ab. (1) Describe how the team identified the various types of resistance
encountered (to their project, solution(s), and/or implementation plan), what
steps were taken to anticipate resistance, and (2) Describe how resistance from
any of the stakeholder groups was addressed.
4Ac. Explain how the team ensured that they had stakeholder buy-in. What proof
did the team have that stakeholder buy-in existed prior to implementation? If it was
not practical to obtain buy-in from all stakeholders, the team should explain why.
2012_CTEA_Judges_Training

42. 4B. Guiding questions: How did the team make the changes
4B. Guiding questions: How did the team make the changes? What changes did the team make? How did they assure that the new approach would be sustained? A successful project not only addresses an immediate need, it also leaves behind evidence that changes occurred and leaves in place a way to verify that changes were both made and sustained.
4Ba. Describe the methods used by the team to implement their solution(s). How did the team plan for the implementation of their solution(s), including action plan development, and such items as allocation of resources, time management activities, etc.?
4Bb. Describe the procedure, system, or other changes that were made to:
(1) implement the solution(s), and
(2) sustain the results over time.
4Bc. Describe the creation and installation of a system to:
(1) measure and manage the performance of the team’s solution(s), and
NOTE: If existing system(s) were able to be used, the team should describe how they were used.
2012_CTEA_Judges_Training

43. 4C. Guiding question: Was the project successful and worth celebrating
4C. Guiding question: Was the project successful and worth celebrating? The team should share the project’s benefits, whether those benefits were as expected relative to original goals, and how the team shared their success with all the stakeholders.
4Ca. Identify the actual results attributable to the implementation of the team’s solution(s). Tangible results include those results represented by improvements in specific measures or metrics–cycle time, number of defects, production, etc., and
Intangible results include those that may not easily be linked to a specific measure or measurement, but might be demonstrated through improvements in a variety of related indicators–morale, employee satisfaction, etc. Teams should describe the indicators they used to identify their intangible results and what led them to believe there was an improvement in a given intangible result.
4Cb. Describe how the team’s results impacted the organization’s goals, performance measures, and/or strategies as identified in 1B. Trail back through the criteria to the organization’s goals, performance measures, and strategies identified early in the problem solving process, and describe how the team’s results supported the achievement of those organizational goals, measures, and/or strategies.
4Cc. Describe the ways in which the team shared their results with the various stakeholder groups. Throughout the problem solving process, the team has described how they identified, communicated, and interacted with internal and external (as applicable) stakeholders to understand their concerns and gain their inputs. Teams should describe how they closed the feedback loop by sharing the project’s results with the stakeholders.
2012_CTEA_Judges_Training

44. Section 5. Team/Project Management and Project Presentation (4 4.5 points = 18 points possible) Team Management and Project Presentation cover the people-resource side of the project. This ranges from how the most appropriate team members were chosen to how they were prepared for this special assignment. Organizational assistance to assure project success is covered here. Also covered is how effectively the team communicated with themselves during the life of the project. Finally, the judges will provide feedback on the overall project presentation.
5A. Team Member Selection and Involvement:
(1) Explain how and/or why the various members of the team were selected including any specific skills, capabilities, knowledge, qualifications, and/or any other criteria used in selecting the team members, and
(2) Describe how the various team members were involved throughout the project including any specific tasks, roles, responsibilities, etc. they may have had during the project.
5B. Explain Team Member Preparation/Development. The team should detail any training orother preparation the team received prior to or during the project to help them operate more effectively as a team. Additionally, any training the team received related to the problem solving methodology/approach they used–including the various tools and techniques used throughout the project to collect, analyze, and/or present data and information–should be included.
5C. Explain Team Performance Management. Explain how the team members worked together effectively throughout the project. The team’s response should include an explanation of how the team capitalized on the skills of their individual members as they carried out their roles and responsibilities, how team members shared data and information throughout the project, how they ensured effective communication within the team, and how the team managed their performance with respect to project deadlines/deliverables/milestones.
NOTE: Responses might include a description of how meetings were conducted, any electronic means, or any other methods the team may have used, to share data and information. Regardless of the approach(es) the team used, an explanation of how they ensured effective communication within the team is appropriate.
2012_CTEA_Judges_Training

45. 5D. Project Presentation: Judges will score the team on the organization, clarity, and overall effectiveness of their presentation. Effective use of any audio/visual aids and any other presentation aids and/or techniques will also be considered in this item. It is important to note that the judges will assess how clearly and effectively the team communicated the story of their project.
Note: the scoring and feedback report will also reflect themes, both strengths and opportunities for improvement, which judges observed throughout the presentation. Consistent use of tools, analysis of data, stakeholder groups, etc. will be considered as integrating the project in a positive fashion.
2012_CTEA_Judges_Training

46. Criteria for Process Improvement Teams
2012_CTEA_Judges_Training

47. Project Selection/Purpose gives the team a chance to talk about the importance of the project to the organization, potential project stakeholders, and the team itself. Section 1 is divided into three sub-sections: (1A) allows the team to discuss how the project was selected. (1B) provides an opportunity for the team to demonstrate the importance of the particular project to the organization. (1C) gives the team a chance to discuss how potential stakeholders were selected and how those stakeholders might be impacted.
1A. Guiding Question: How was the project selected? Either the team or management selected the project. In either case, the process used to select the project must be clear and well stated. As with all team responses, providing specific examples of techniques and data used will strengthen the team’s presentation.
1Aa. (1) Describe how data and/or tools were used to support the selection, even if the project was assigned to the team, and
(2) Explain why these specific tools and/or data were used to select the project.
1Ab. (1) Explain how the project was selected, and
(2) Explain why the project was selected: what discerned gap or observed opportunity lead to the launching of this project?
NOTE: 1Aa addresses the tools and data used. The emphasis in 1Ab is on the decisions made because of the tool usage/data analysis in 1Aa.
1Ac. Describe the potential stakeholders for the project and how they participated in the selection process. If the stakeholders were not directly involved, then describe how their interests were known and taken into account.
NOTE: this should reflect the history of the project. The actual/final stakeholders may be different from the potential stakeholders at the time the project was being defined. The goal here is to be sure that the project was not selected in a vacuum without an understanding of the organizational environment.
2012_CTEA_Judges_Training

48. 1B. Guiding question: Why is this project important to the organization? A good project will support, or at least align with, one or more of the organization's goals, performance measures, and/or strategies. Organizational goals can vary in type and style, from broad vision or mission statements to very specific objectives related to their goals and/or strategies. These items are looking for the link between the team’s project and those goals and/or strategies. 1Ba. Identify the affected organizational goals, performance measures, and/or strategies. Note: Goals may be present at various levels in an organization and the team is being asked to identify the goals/performance measures impacted by their project. If the team’s project is linked to a lower-level goal, it may be helpful for the team to demonstrate the linkage between the project and the lower-level goal as well as the linkage between the lower-level goal and the organizational-level 1Bb. Identify types (WHAT) of impact the project will have on each goal, performance measure, and/or strategy presented in 1Ba. NOTE: “Type” implies a directional impact of some sort, such as increased/decreased costs, efficiencies, safety, associate satisfaction, etc. 1Bc. (1) Identify the degree of impact (HOW MUCH) the project will have on each goal, performance measure, and/or strategy, and (2) Explain how the team quantified the predicted degree of impact. If the team provided relative degrees of impact such as high, medium, or low, they should provide their rationale for deciding “high” vs. “medium” or “low.”
2012_CTEA_Judges_Training

49. 1C. Guiding questions: Who might be impacted by this project
1C. Guiding questions: Who might be impacted by this project? What type of impact might they expect to see? “Stakeholders” defines anyone who may be affected by the project. Stakeholders could typically include internal/external customers/clients, suppliers, employees, etc. Stakeholders may refer to a unique group (e.g., “the maintenance department”) or one or more individuals. 1Ca. (1) Identify the potential internal and external stakeholders who might be impacted by the project, and (2) Explain how the potential stakeholders were identified. 1Cb. (1) Identify the types of potential impact on stakeholders, and 2) Explain how these were determined. NOTE: Stakeholders may be impacted in many different ways by the project, from quality to timeliness. Demonstrating that the team understands how the project may have different impacts on the different stakeholder groups is important. As with 1Bb (type of goal impact), the impact is implied to be directional. This section could later tie to 4Ab, as correctly predicting potential negative impacts early in the project can help the team prepare to overcome resistance later. 1Cc. (1) Identify the degree of potential impact on stakeholders, and (2) Explain how this was determined. Not all stakeholders are impacted to the same degree. If the team provided relative degrees of impact such as high, medium, or low, they should provide their rationale for deciding “high” vs. “medium” or “low.”
2012_CTEA_Judges_Training

50. Section 2 Current Situation Analysis (6 items @ 4
Section 2 Current Situation Analysis (6 4.5 points = 27 points possible) Current Situation Analysis focuses on how the team analyzed the current situation. This includes the processes, data, and information. This section also covers how stakeholders were involved. The second half of this section covers how the team determined and validated their final improvement opportunity(ies). If the organization uses a specific methodology as part of the quality process, the team may want to consider sharing how that methodology was applied to identify the possible and final improvement opportunity(ies).
2 A. Guiding question: How did the team identify the potential improvement opportunity(ies)? A high- performing team will explore multiple explanations for their current situation, not assuming that the first explanation is correct. 2Aa. Describe the methods and tools used to identify possible improvement opportunity(ies). Teams may use a variety of approaches/processes to identify the possible improvement opportunity(ies) in a given situation. NOTE: For “opportunity for improvement teams,” describe the methods and tools used to generate a list of possible opportunities for improvement. 2Ab. Describe the team’s analysis of data to identify possible improvement opportunity(ies). Provide clear, specific examples of what data were analyzed (from the tools/methods described in 2Aa), and how they were analyzed, to help identify the possible improvement opportunity (ies). 2Ac. Describe how the stakeholders were involved in identifying the possible improvement opportunities. Provide supporting information. NOTE: “Involved” does not necessarily mean how stakeholders participated directly in the process of selecting the improvement opportunities. “Involved” can be addressed by describing how the team took the interests of the stakeholders into account in determining the possible improvement opportunities.
2012_CTEA_Judges_Training

51. Guiding question: How did the team analyze information to identify the final improvement opportunity(ies)? Having identified a variety of possible improvement opportunity(ies), in 2A, these items give the team a chance to share how they went about narrowing down the possibilities to identify the true improvement opportunity(ies) for this project. As is true elsewhere, by sharing any appropriate validation techniques, the team could strengthen their presentation. 
2Ba. Describe the methods and tools the team used to narrow down the list of potential improvement opportunity(ies) (identified in 2A), to the final improvement opportunity (ies).
2Bb. Describe the team’s analysis of data to select the improvement opportunity(ies). Provide clear, specific examples of what data were analyzed (from the tools/methods described in 2Ba), and how they were analyzed, to help identify the final improvement opportunity(ies).
2Bc. (1) Identify what the improvement opportunity(ies) was/were, and
(2) Explain how the team validated that the final improvement opportunity(ies) determined in 2Bb was/were in fact the final improvement opportunity(ies). What tools and/or methods did the team use to validate (prove) to themselves (and others if needed) that they identified the true improvement opportunity(ies)?
2012_CTEA_Judges_Training

52. Section 3. Solution Development (9 3 points = 27 points possible) Solution Development allows the team to demonstrate how they logically moved from the opportunities to their improvement action(s). Initially the team should share how they developed a set of possible improvement opportunities. From there, the team should share how they arrived at their specific improvement action(s) that would meet the project objective(s). Finally, the team should share how they convinced themselves and others that their proposals would be both practical and acceptable to the organization.
Guiding questions: How did the team identify the improvement actions? What criteria did the team decide on for selecting their final improvement actions? An excellent team is one that explores multiple options to arrive at the best one. They will also have specific, supportable criteria against which to judge those options. 3Aa. Describe the tools and methods the team used to identify the possible improvement actions for the improvement opportunity(ies) identified and verified in 2B. There may be a number of ways to address the improvement opportunity(ies). What tools and/or methods did the team use to identify these possible improvement actions? 3Ab. Describe the team’s analysis of data to develop possible improvement actions. Provide clear, specific examples of what data were analyzed (from the tools/methods described in 3Aa), and how they were analyzed, to help identify the possible improvement actions. 3Ac. Identify the criteria the team decided to use to assess the relative effectiveness of the various possible improvement actions they identified in 3Aa and 3Ab above. NOTE: The criteria should help the team narrow down the list of possible improvement actions to enable them to identify the best improvement action(s).
2012_CTEA_Judges_Training

53. Guiding questions: How did the team select/determine the final improvement action(s) from the possibilities generated in 3A above? How did they assure that their final choice(s) met the needs of the stakeholders? Having identified a variety of possible improvements, the team should share what lead them to decide on their final approach.
3Ba. Describe the methods and tools used by the team to select the final improvement action(s). Now that the team has identified the criteria they used in selecting the final improvement action(s), this item asks the team to describe the tools and methods they used along with the criteria identified in 3Ac above to select the final improvement action(s). 3Bb. Describe what data were analyzed, and how they were analyzed to help identify the final improvement action(s).
NOTE: The team described the methods and tools they used to determine the final improvement action(s) in 3Ba; 3Bb is now asking the team to describe what data were analyzed and how they were analyzed, to support the selection of their final improvement action(s).
3Bc. Describe how stakeholders were involved in determining the final improvement action(s). NOTE: If the stakeholders were directly involved in the selection of the final improvement action(s), the team should describe how they were involved in the process; if they were not directly involved, the team should describe how the stakeholders’ interests and concerns were considered.
2012_CTEA_Judges_Training

54. Guiding questions: How did the team validate the final improvement action(s)? What benefits does the team expect the organization to realize once the improvement action(s) is/are implemented? How did the team use data to justify the implementation? “Validation,” is not to be confused with “justification.” “Validation” is simply verifying that the improvement action(s), if implemented, will the improvement opportunity(ies) found in 2Bc. “Justification” assumes that the improvement actions are valid, and that, in addition, they are a good “value” to the organization. Both validation and justification need to occur before the implementation of the final improvement actions(s). They should not be based on data generated after implementation. 3Ca. (1) Describe the improvement action(s) the team felt would be most appropriate, and then (2) Explain how they validated that the improvement action(s) would solve the problem prior to actually proceeding with the implementation of the improvement action(s). 3Cb. List the benefits that may be realized from the improvement action(s). (1)Tangible benefits include those benefits that lend themselves to being represented by improvements in specific measures or metrics–cycle time, number of defects, production, etc.--and (2). Intangible benefits include those that may not easily be linked to a specific measure or measurement, but might require demonstration through improvements in a variety of related indicators–morale, employee satisfaction, etc. Teams should attempt to describe what indicators they are looking at when identifying their intangible benefits– what did they see that led them to believe there could be an improvement in a given intangible benefit? 3Cc. Explain how the team used data to justify the implementation of the improvement action(s) they determined to be best in 3B. The team should present the data and information they used to demonstrate the value to the organization in implementing their improvement action(s).
2012_CTEA_Judges_Training

55. Section 4. Project Implementation and Results (9 3 points = 27 points possible) Project Implementation and Results addresses how the team sought and secured buy-in. In addition, this section gives the team the chance to discuss the approaches they used to plan for and implement their improvement action(s). Finally, this is where the team gets to present the results that were achieved. This should be based on data generated after implementation of the improvement action(s).
4A. Guiding questions: How were internal and external stakeholders involved in implementation? How did the team go about securing buy-in, identifying resistance, and convincing themselves (and others) that buy-in did, in fact, exist? 4Aa. Indicate how internal and external (as applicable) stakeholders were involved in implementing the improvement action(s). 4Ab. (1) Describe how the team identified the various types of resistance encountered (to their project, improvement action(s), and/or implementation plan), what steps were taken to anticipate resistance, and (2) Describe how resistance from any of the stakeholder groups was addressed. 4Ac. Explain how the team ensured that they had stakeholder buy-in. What proof did the team have that stakeholder buy-in existed prior to implementation? If it was not practical to obtain buy-in from all stakeholders, the team should explain why.
2012_CTEA_Judges_Training

56. 4B. Guiding questions: How did the team make the changes
4B. Guiding questions: How did the team make the changes? What changes did the team make? How did they assure that the new approach would be sustained? A successful project not only addresses an immediate need, it also leaves behind evidence that changes occurred and leaves in place a way to verify that changes were both made and sustained. 4Ba. Describe the methods used by the team to implement their improvement action(s). How did the team plan for the implementation of their improvement action(s), including action plan development, and such items as allocation of resources, time management activities, etc.? 4Bb. Describe the procedure(s), system(s), or other change(s) that were made to: (1) implement the improvement action(s), and (2) sustain the results over time. 4Bc. Describe the creation and installation of a system to: (1) measure and manage the performance of the team’s improvement action(s), and NOTE: If existing system(s) were able to be used, the team should describe how they were used.
2012_CTEA_Judges_Training

57. Section 5. Team/Project Management and Project Presentation (4 4.5 points = 18 points possible) Team Management and Project Presentation cover the people-resource side of the project. This ranges from how the most appropriate team members were chosen to how they were prepared for this special assignment. Organizational assistance to assure project success is covered here. Also covered is how effectively the team communicated with themselves during the life of the project. Finally, the judges will provide feedback on the overall project presentation.
5A. Team Member Selection and Involvement: (1) Explain how and/or why the various members of the team were selected including any specific skills, capabilities, knowledge, qualifications, and/or any other criteria used in selecting the team members, and (2) Describe how the various team members were involved throughout the project including any specific tasks, roles, responsibilities, etc. they may have had during the project. 5B. Explain Team Member Preparation/Development. The team should detail any training or other preparation the team received prior to or during the project to help them operate more effectively as a team. Additionally, any training the team received related to the process improvement methodology/approach they used–including the various tools and techniques used throughout the project to collect, analyze, and/or present data and information–should be included. 5C. Explain Team Performance Management. Explain how the team members worked together effectively throughout the project. The team’s response should include an explanation of how the team capitalized on the skills of their individual members as they carried out their roles and responsibilities, how team members shared data and information throughout the project, how they ensured effective communication within the team, and how the team managed their performance with respect to project deadlines/deliverables/milestones. NOTE: Responses might include a description of how meetings were conducted, any electronic means, or any other methods the team may have used, to share data and information.
2012_CTEA_Judges_Training

58. 4C Guiding question: Was the project successful and worth celebrating
4C Guiding question: Was the project successful and worth celebrating? The team should share the project’s benefits, whether those benefits were as expected relative to original goals, and how the team shared their success with all the stakeholders.
4Ca. Identify the actual results attributable to the implementation of the team’s improvement action(s).
(1) Tangible results include those results represented by improvements in specific measures or metrics– cycle time, number of defects, production, etc., and
(2) Intangible results include those that may not be easily linked to a specific measure or measurement, but might be demonstrated through improvements in a variety of related indicators–morale, employee satisfaction, etc. Teams should describe the indicators they used to identify their intangible results and what led them to believe there was an improvement in a given intangible result.
4Cb. Describe how the team’s results impacted the organization’s goals, performance measures, and/or strategies as identified in 1B. Trail back through the criteria to the organization’s goals, performance measures, and strategies identified early in the improvement process, and describe how the team’s results supported the achievement of those organizational goals, measures, and/or strategies.
4Cc. Describe the ways in which the team shared their results with the various stakeholder groups. Throughout the improvement process, the team has described how they identified, communicated, and interacted with internal and external (as applicable) stakeholders to understand their concerns and gain their inputs. Teams should describe how they closed the feedback loop by sharing the project’s results with the stakeholders.
2012_CTEA_Judges_Training

59. 5D. Project Presentation: Judges will score the team on the organization, clarity, and overall effectiveness of their presentation. Effective use of any audio/visual aids and any other presentation aids and/or techniques will also be considered in this item. It is important to note that the judges will assess how clearly and effectively the team communicated the story of their project.
Note: the scoring and feedback report will also reflect themes, both strengths and opportunities for improvement, which judges observed throughout the presentation. Consistent use of tools, analysis of data, stakeholder groups, etc. will be considered as integrating the project in a positive fashion.
2012_CTEA_Judges_Training

60. Scoring Tips (1)
Note that “Exceeds” scoring level descriptions are cumulative.
What is looked for in “Exceeds” must include what is in “Meets”.
The scoring level of “Unclear” is when not enough information is provided to determine if the team’s approach met or could meet the criteria requirements.
The scoring level of “Not Covered” is not noted in the guidelines because it receives “0” points.
These are meant to be general guidelines for scoring to serve as a standard for evaluating team presentations.
All teams are still encouraged to be as creative and as innovative as possible in sharing their stories.
2012_CTEA_Judges_Training

61. Scoring Tips (2)
Criteria 2Aa, 2Ba, 3Aa, and 3Ba ask the team to describe their tools/methods.
Criteria 2Ab, 2Bb, 3Ab, and 3Bb ask the team to describe their analysis of data.
There are 14 criteria with two parts.
If you rate the two parts at different levels, you must score it at the lower level.
For example, if one part is “Meets” and the second part is “Exceeds”, the item is scored at “Meets”.
2012_CTEA_Judges_Training

62. Read the ASQ-ITEA Glossary of Terms
Scoring Tips (3)
Read the ASQ-ITEA Glossary of Terms
ASQ-ITEA has constructed a Glossary of Terms that is very important for judges.
In particular be sure to study the information given for “Describe” because each of the four words listed there requires a different level of information.
Also be sure to study the term “Integration” because teams that demonstrate this characteristic should score better and are more likely to attain the “Exceeds” level than those that do not.
Integration explained on the next two slides.
2012_CTEA_Judges_Training

63. Integration Explained (1)
The Criteria in Detail draw attention to where an item detail has a connection (cross-linkage) to other criterion item(s).
The Criteria show that addressing a criterion item detail in one area has an impact on one or more other criterion areas.
How the team addresses these cross-linkages indicates the degree of integration within the team project, stakeholder, and organization direction.
This is a significant factor for a team to score at the “Exceeds” level.
2012_CTEA_Judges_Training

64. Integration Explained (2)
“1Cb. Identify the types of potential impact on stakeholders and explain how these were determined.
Stakeholders may be impacted in many different ways by the project, from quality to timeliness.
Demonstrating that the team understands how the project may have different impacts on the different stakeholder groups is important.
As with 1Bb (type of goal impact), the impact is implied to be directional.
NOTE: This section could later tie to 4Ab, as correctly predicting potential negative impacts early in the project can help the team prepare to overcome resistance later.
The Cross-Linkage in 1Cb is with 1Bb and 4Ab, if addressed per the criteria, would indicate there is integration in the team’s approach across the project and/or organization.
2012_CTEA_Judges_Training

65. Review the CTEA Judges Workbook (all 3 tabs)
Scoring Guidelines
Review the CTEA Judges Workbook (all 3 tabs)
During the team presentation, each judge uses their Judges Worksheet to take notes and to evaluate the presentation.
You will have the forms in all 3 tabs given to you at the judging event.
2012_CTEA_Judges_Training

66. Judges Worksheet (Tab 1 in the Judges Workbook)
This is the Judges Worksheet to be used during the teams presentation.
Refer to the worksheet during the independent review and during consensus with your team.
2012_CTEA_Judges_Training

67. Reaching Consensus (1)
After doing your individual scoring, the judges must reach consensus on their scores across the panel.
A panel member collects and posts all scores to the Consensus Score Sheet.
No consensus discussion is necessary on an item if there is only one scoring level deviation among the judges’ scores.
2012_CTEA_Judges_Training

68. Review the Consensus Scoresheet (Tab 2 of the Judges Workbook)
Reaching Consensus (2)
Review the Consensus Scoresheet
(Tab 2 of the Judges Workbook)
If there is more than one scoring level deviation, the panel members must adjust their individual scores to be within one deviation of each other (example on next slide).
No judge can have a score that deviates two levels from another judge.
This is done through dialogue about reasons for the respective scoring.
Each judge supports their rational for the scoring and discuss as a group.
Then the judges’ scores are averaged on the Consensus Scoresheet.
2012_CTEA_Judges_Training

69. Consensus Scoresheet Sample
This is sample of an initial Consensus Scoresheet.
Notice 2 scoring level deviation on items 1B.b and 1C.b.
2012_CTEA_Judges_Training

70. Final Scoresheet Sample Tab 3 in Judges Workbook
Review the ASQ-ITEA Scoresheet Sample
The results of the Consensus Scoresheet are automatically populated in the Final Scoresheet.
Review the CTEA Final Scoresheet (Tab 3).
2012_CTEA_Judges_Training

71. Familiarizing Yourself with a Presentation
Review the ASQ-ITEA Presentation Sample
The final step in understanding scoring is to familiarize yourself with what a presentation looks like.
While each team presentation is different, one is provided as part of this training package and available on the CTEA website.
2012_CTEA_Judges_Training

72. How to Write Effective Feedback Comments
Lesson 3
How to Write Effective Feedback Comments
2012_CTEA_Judges_Training

73. Purpose of Feedback Reports
A valuable tool teams and their organizations can use to take action.
Celebrate Strengths
Consider Suggestions for Improvement (SFIs)
The feedback report has two components.
Total Score (from judges consensus)
Feedback Comments
Strengths
Suggestions for Improvement
2012_CTEA_Judges_Training

74. Qualities of ‘Well Written’ Feedback Comments
Read the CTEA Judges Guidelines
Succinct, specific narrative.
Constructive and actionable.
Clear to the team why included.
Complete sentences w/o grammar errors.
Non-prescriptive.
Address important elements.
Supports score and evaluation.
2012_CTEA_Judges_Training

75. Do’s & Don’ts of Comment Writing
Use Facts w/o judgment.
Be specific about what is missing (‘it is not clear’ comments).
Use Criteria language and link to examples in the team’s presentation.
Be impactful and answer the ‘so what’.
Use Opinions.
Be prescriptive (Avoid “could, should or would”).
Be overly technical in your writing so that the team is unable to understand feedback.
2012_CTEA_Judges_Training

76. Feedback Comments Should Have the Following Items
Number of the Criteria item.
Number on the presentation slide(s) that the comment addresses.
Language of the criteria item in the form of a brief quote or paraphrase from the item.
Language of the team in the form of a brief quote or paraphrase from the relevant slide(s).
2012_CTEA_Judges_Training

77. Adhere to Style Guidelines
Use a polite, professional, positive tone.
Use active voice, present tense.
Use the team's terminology.
Avoid jargon or acronyms except when used by the team.
Refer to “the team”, not name.
Reference figure numbers when appropriate.
2012_CTEA_Judges_Training

78. Determining What to Emphasize in Feedback Comments
Strength Comments should be the highest scoring items.
SFI Comments should be the lowest scoring.
Comments on integration should only be written if there are noticeable issues with respect of cross-linkages (either as Strengths or SFIs).
2012_CTEA_Judges_Training

79. Examples: Well Written Strength Comments
“The presentation effectively identified the internal and external stakeholders and clearly assigned rated measures to show how much they will be impacted by the project (Slide 17)(1Cb).”
“Excellent information was provided explaining why this project was selected (Slide 3). Six sigma tools were used to justify the project selection and were clearly explained and illustrated in slides 6 – 9. Additionally the use of data to substantiate this project’s selection was highly informative.”
2012_CTEA_Judges_Training

80. Example: Poorly Written Comment
“The team did a good job when they presented their project.”
Issues with Strength Comment:
No support with examples to let the team know what was good.
Does not reinforce practices.
Does not describe the impact.
Does not relate to the criteria.
Not in present tense, active voice.
2012_CTEA_Judges_Training

81. ‘Tips’ for SFI Opening Statements
Although the team addresses/states/indicates…
While the team demonstrated … it is not clear/apparent/evident how/what the team...
The team does not address/report how/what…
2012_CTEA_Judges_Training

82. Examples: Well Written SFI Comments
“The team clearly explained the types of impact and the degree of impact of the project on both internal and external stakeholders; however in order to strengthen this item the judges were searching for information on “how the scores and measurable data were obtained.” (1Cb & 1Cc).”
“While the team showed how they used six sigma tools DMADV and SIPOC to identify potential stakeholders; there was no evidence provided to describe how these potential stakeholders were involved in the selection of the project (Slide10).”
2012_CTEA_Judges_Training

83. Example: Poorly Written SFI Comment
“It was impossible for us to tell what the team achieved. Goals and measures were glaringly omitted. Without an action plan that includes goals it will be impossible for the team to achieve meaningful results. The organization should be using the balanced scorecard to help them.”
Issues with Poorly Written SFI Comment:
Critical tone “glaringly omitted”.
So what is opinion, not fact “Without an action plan that includes goals it will be impossible for the team…”.
Prescriptive.
No linkage to criteria.
2012_CTEA_Judges_Training

84. Refining Your Comments
What is the real strength or SFI?
Is it clear?
What team practices in the comment support the Strength/SFI?
Is the “so what” linked to the main point of the comment?
Is the “so what” factual?
2012_CTEA_Judges_Training

85. Feedback on Integration
Earlier the need to include integration (or lack thereof) in the scoring was mentioned
If the team showed a high score due to integration where the Criteria call for a cross- linkage, then you should write a Strength feedback comment to that effect
On the other hand, if the team showed a direct lack of integration where the Criteria call for a cross-linkage, then you should write an SFI comment to that effect
2012_CTEA_Judges_Training

86. Reaching Consensus on Feedback Comments
How the judging panel does the work of writing draft feedback comments is up to the panel, and the lead judge will provide suggestions and guidance.
There is a requirement for the judges to reach consensus on their feedback comments.
2012_CTEA_Judges_Training

87. Supporting Materials for Writing Feedback Comments
Review the CTEA Feedback Report Template
Read the ASQ-ITEA Feedback Sample
The two supporting documents named above are provided to you for writing feedback.
The first is an actual feedback report for the ASQ- ITEA presentation provided earlier (HD Supply).
The second is a template you will be provided at the judging event.
2012_CTEA_Judges_Training

88. Next Steps – Phase 2 Lesson 4 2012_CTEA_Judges_Training

89. Schedule (1)
Attend one of the following judge conference calls using the call-in information: x
Tuesday, September 10, 8: :30 am
Thursday, September 12, 12:30 - 2:30 pm
Tuesday, September 17, 8: :30 am
We will answer questions regarding the process that you may have at that time.
We will discuss logistical issues, locations, schedules, etc.
We will verify that you have all documentation.
2012_CTEA_Judges_Training

90. Schedule (2) Attend your assigned judging event(s):
October 8, 2013: Los Angeles
October 10, 2013: San Diego
October 15, 2013: Sacramento
Respond to your lead judge’s request for comments on the final draft of the feedback report within 1 week of receipt.
The Lead Judge will submit the final feedback report to the CTEA Council Chair by October 28, 2013.
2012_CTEA_Judges_Training

91. Schedule (3)
The CTEA Council Chair provides the feedback reports to the teams and their sponsoring organizations:
December 2, 2013
CTEA Recognition and Awards Ceremony:
2014 (To be announced)
ASQ ITEA Finalist Presentations and Awards Ceremony, ASQ World Conference:
May 5-7, 2014, in Dallas, TX
2012_CTEA_Judges_Training

92. Thank you for taking Phase 1 of the CTEA Judges Training
Conclusion
Thank you for taking Phase 1
of the CTEA Judges Training
See you in October!
2012_CTEA_Judges_Training

93. ASQ-ITEA Criteria Summary
Read the ASQ-ITEA Award Criteria
The following slides show the summary form of the ASQ-ITEA Award Criteria.
2012_CTEA_Judges_Training