1. CONSENSUS TRAINING
July 10, 2019

2. SEPTEMBER
TODAY

3. Independent Review Consensus Site Visit Independent Review
Review for Conflict with Applicant
Review the Framework
Read the Application
Identify Relevant Key Factors for each Item
Complete Results Matrix
Comments, KF, Item References in Navigator
Click “Done”!
Prepare Consensus Items
Participate in Consensus meeting
Prepare Site Visit issues
Complete Site Visit
Prepare Final Scorebook
12 process items
Criteria
Key Factors
Application: PROCESS
6-8 comments (ADLI)
5 results items
Application: RESULTS
5-7 comments (LeTCI)
Consensus
Site Visit
Independent Review
Review slide
The process can seem a bit “tedious”, by design

4. The Consensus Process This is the core skill of being an examiner.
If you can master these two dimensions you will have 80% of what you need to know to be an examiner.
Don’t leave today without knowing you have these two dimensions mastered.
This is also one of the key take-aways for you and your organization. It will help you figure out what next step is, where there are opportunities for improvement.
Our hope is that after today, you will be as good at this part of being an examiner as someone that has been an examiner for multiple years.

5. Overview
Purpose: Create high quality, actionable feedback for the Applicant
Action Plan: Synthesize individual scorebooks into one team scorebook which will become the basis for Site Visit Strategy and Final Feedback Report
Keys to success:
Strength comments identifying criteria nugget, key factor(s), application examples and process maturity (ADLI or LeTCI)
Next step OFI Concepts or Questions to guide site visit planning
Emphasize the overall objective of Stage 2 Consensus which is:
To condense individual scorebooks into one scorebook for each applicant
Review the list of things that are key to the achievement of that objective
Explain that for some applicants the Consensus Scorebook will be the only feedback they get.

6. Roles Team Lead – manages team & consensus process
Item Lead – responsible for synthesizing comments for specific assigned Item(s)
Item Backup – reviews and provides feedback to Item Lead to validate and improve the final output
Define Roles and Responsibilities

7. Steps 1. Consensus Preparation 2. Consensus Call 3. Post-Consensus
8/20/2019
Steps
1. Consensus
Preparation
2. Consensus Call
3. Post-Consensus
Item Lead Synthesizes Team IRs in Navigator
Item Lead Score
Prepare Script
Item Lead follows Script
Discuss comments
Team Score
Take notes to edit comments later
Edit comments per consensus
Report due to MEI Office w/in 7 days
Site Visit Planning
First we will focus on Consensus Planning
How long with each stage……
Prep… hours
Call….8 hours
Post….. Couple of hours

8. Preparing for Consensus

9. Preparing an Item for Consensus Meeting
1. Review the Criteria requirements
2. Read applicant’s response
3. Select 4-6 relevant key factors
4. Review team members’ Independent Reviews
SYNTHESIZE
Synthesize team members’ IR into 6-8 comment topics
Develop rationale
WRITE
Draft 6-8 feedback comments based on your chosen topics
Balance strengths and OFIs
SCORE
Score the item based on your consensus review
REVIEW/EDIT
Revise comments based on feedback from your backup
Provide feedback to your partner
PREPARE
Complete Presentation Script for each item you will present during the consensus meeting
Now that you’ve selected and written comments, scored your item, we need to get feedback from the back-up before the consensus meeting

10. Synthesize Comments Examiner 6 Item 1.1 Examiner 7 Item 1.1 Examiner 5

11. Navigator - Stage 2

12. Navigator – Review Stage 1 Comments
For each of your assigned items in the Stage 2 scorebook
Click on this icon to show the entries from all the Stage 1 IRs

13. Export from Navigator
Click on this icon to open the print preview/report for this item.
Click on this icon to export the report data (Key Factors, Strengths and OFIs) to excel or word.

14. Ways to Sort/Review: Paper OR Excel Method
Team Member IRs
Excel…
Export Nav to Word or Excel
Copy and paste all Strength comments into First Tab of W/S
Sort by item ref
Make a new tab for each Item ref (work on writing a comment)
Copy the final comments back to navigator
Team Pre-Consensus
See Cheat Sheets in Examiner Resource Center

15. Volunteer Resource Center

16. Synthesis: Excel tool Export the Item to Excel
Sort comments by Item Ref
Cut and paste “same item ref” comments into a new tab
Read/review the common comments
Create the “best” comment by selecting one from IR or combining elements from several IR comments
On consolidated comment…indicate which examiners (initials) agree with comment concept - not word for word
Repeat until you have the best 6-8 proposed Comments for the item
(AA, BB, CC)

17. EXAMPLES

18. EXAMPLES

19. How to write a Consensus Comment
Consensus Comment may be:
A single IR comment which best expresses a unique or a shared IR idea (for example: 3 examiners comment on the same criteria with similar ADLI evaluations  select the best written comment) OR
Combined elements and/or language from several IR comments
A new comment composed using the ideas expressed by one or more IR comments

20. Frequently Asked Questions
Q - Do I have to use all the IR Comments?
A – No, select the “best” IDEAS from the IR comments
Q – What should I do with IR Comments I didn’t use?
A – List them on the Comments Not Used form
Q – What are the “best” IR comments?
A – Comments which align with the Criteria and the applicant’s Key Factors; clearly communicate the organization’s next step toward process maturity
Q – What if the IR comments conflict (for example, a strength and an OFI addressing the same criteria)?
A – The Item leader should evaluate the comments based on the Criteria, the application responses, and Key Factors to determine whether the organization will be better served if the comment is a strength or an OFI, for example: a Strength for Deployment versus an OFI for Learning.

21. Comment Content Comments Include 4 elements Criteria
Nugget of criteria you are addressing
Key Factors
Relevance or importance to the applicant
Evaluation
Process maturity using ADLI or LeTCI
Application
Strength: 2-3 examples
OFI: missing evidence
Comments Include 4 elements
Remember High School Composition class
Introduce the Topic  a few Details or Supporting Evidence  Conclusion or Summary

22. Characteristics of GOOD Comments
Are clearly aligned with the Criteria
Contain only one main idea supported by 2-3 examples
Are evaluated on maturity of process (ADLI) or results (LeTCI)
Use the applicant’s terminology – not “Baldrigese”
Are actionable and relevant to the applicant
Add value - not just restate application
Are factual not judgmental

23. Entering Synthesized Comments in Navigator
The Consensus Scorebook’s format is slightly different than the Independent Review Scorebook: “As Evidenced by” is now called “Rationale”. And there are no checkboxes for ADLI or LeTCI.
Select 4-6 Key Factors for each Item at the top of the Scorebook
Paste all final comments into the Strength or OFI areas
Check the Key Factor Reference and Item Reference boxes for each comment.
Use “Rationale” for notes, ADLI/LeTCI, site visit issues
Enter your score
Let your team leaders and item backup know the item is ready for review.

24. Comments Used/Not Used
Comments Not Used Matrix.xls
Download form from Examiner Resource Center
Indicate why comment wasn’t used (examples include: not actionable, lower level/not next step, not criteria based, etc.)
These may be brought back later (either at consensus or site visit)

25. SCORING

26. Scoring Guidelines

27. Process Category Scoring Guidelines
Results Category Scoring Guidelines

28. Score the Item Think holistically at the ITEM Level
Score is determined by the Strengths (& SOFIs)
No strengths  score is 0
No OFIs  score is 100%
Process or Results (ADLI/LeTCI)
OFIs should describe what is needed to get to the
next higher box = applicant’s next steps
Score in 5% increments (50, 55, 60, etc.)
Repeat on Team Call
NOTE: This is your PROPOSED Score based on synthesized Item Comments

29. Example: Item 6.2 STRENGTHS OFIS
6.2a – LifeBridge Organ and Tissue Sharing (LOTS) has a well deployed approach to manage the cost, efficiency, and effectiveness of their operations ... (A, D, L, I)
6.2a – Although “LOTS employs comprehensive measures to maximize efficiency and ensure cost control, which are balanced with customer needs” it is unclear that this is accomplished using a systematic and effective process. (A)
6.2b – LOTS ensures the security and cybersecurity of sensitive or privileged data and information through multiple policies and procedures ... to verify appropriate access and current users. (A, L)
6.2b – It is not evident that LOTS has tested its cybersecurity response plan to ensure it can recover from cybersecurity incidents. (L)
6.2c(1) – LOTS has an established safety program, a safety committee and a safety officer that uses the LDS for ensuring a safe operating environment. (A, L, I)
6.2c(2) – LOTS deploys a systematic and effective Emergency Response Plan (ERP) to address prevention, continuity of operations, and recovery, using routine testing and analysis, remote access to key services, plus business interruption insurance and collaboration with its workforce, partners, suppliers, and other OPOs. (A, D)
6.2c(2) – It is not clear how LOTS utilizes its regular testing and analysis processes to improve the effectiveness of the ERP, such as ensuring IT systems will be available and secure in the event of a disaster or emergency. (L, I)
Discussion

30. Entering Scores in Navigator
1. Start in the % column.
2. Determine if 50-65% is too high, too low, or just right. Select best score range.
3. Then choose the best score within that range (in increments of 5%).

31. Consensus Meeting Preparations

32. Review YOUR Item(s) Review synthesized comments:
8/20/2019
Review YOUR Item(s)
Review synthesized comments:
Covers the main objectives of the Criteria (a1, a2, b1, …)
Contains: Criteria nugget, Key Factor, evaluation of process maturity, examples from application or what’s missing
Provides actionable and relevant feedback
No conflicts or contradictions between comments (Strengths vs OFIs or other items)
Verify that the score accurately reflects the Strength comments
Before you celebrate:
Review your comments to verify that they cover the main objectives of the criteria and conform to the Comment Guidelines
Think about the big picture to be sure there are no conflicts between Strengths and OFIs
Most importantly: do they provide actionable feedback for the Applicant?
Verify your proposed score fits with the ratio of strengths and comments for the Item
Then your back-up and team leader will review and provide feedback; make any revisions based on their input

33. Share your Item(s) & Make Edits from Feedback
8/20/2019
Share your Item(s) & Make Edits from Feedback
Enter your proposed Comments and Score in Navigator
Share with Item Back-up and Team Leaders with adequate time for feedback
Incorporate B/U and TL feedback into Navigator comments and rescore if needed
Before you celebrate:
Review your comments to verify that they cover the main objectives of the criteria and conform to the Comment Guidelines
Think about the big picture to be sure there are no conflicts between Strengths and OFIs
Most importantly: do they provide actionable feedback for the Applicant?
Verify your proposed score fits with the ratio of strengths and comments for the Item
Then your back-up and team leader will review and provide feedback; make any revisions based on their input

34. Item Backup - Role & Responsibilities
The role of the item Backup is to provide a critical review of the Consensus comments:
Requires use of: Criteria book, Application, Key Factors
Proofread for grammar, spelling, punctuation
Wordsmith – for clarity and accuracy
Do Strength Comments Conform to the Comment Guidelines?
Criteria, key factor(s), relevant examples and process maturity (ADLI/LeTCI)?
Are they actionable and appropriate for organization?
Adequate length (not too short or too long)
No conflicts or contradictions between comments (Strengths vs OFIs or other items)
Does the score seem appropriate for the Strength comments

35. Script Template (1 Item per form)
Presentation Script Template
See Consensus Forms on the Volunteer Resource Center website

36. Complete Scripts Presentation Script includes:
8/20/2019
Complete Scripts
Presentation Script includes:
Criteria Intent (Summary only)
Key Factors for this Item
Comment Summary…DO NOT paste the full comments  just the intent (ADLI/LTCI)
Proposed Score and Rationale
Which Box, why?
Why not lower band/Why not higher band?
Backup/Team Lead review/edit script
Once your comments/scores have been approved by your back-up and Team Lead, prepare a Presentation Script for Consensus Day for each assigned Item:
You will “read” from these scripts during Consensus – it saves much time on Consensus Day!
The Script will include a very brief summary of the criteria requirements, a brief review of the 4-6 Key Factors used, and a summary of the strengths as well as the OFIs
Lastly, you’ll report on your scoring Rationale
The script format is found on the Volunteer Resource Center and must be completed prior to the Consensus meeting

37. 1-2 Days Before Consensus

38. Familiarize yourself with Consensus Scorebook – Every Item
EVERYONE must be knowledgeable about the NEW Scorebook
Read the complete Consensus scorebook – it’s not the same as your IR Scorebook
Look for:
Conflicts - between Strengths and OFIs both within and across Items
Clear, Good grammar, spelling, etc.
Do you Agree? (evaluation of process maturity, balance of Strengths & OFIs, scoring, etc.)
Do you see cross-cutting ideas or themes? (Strengths or OFIs)
Accurate?...based on Application, uses Applicant’s terminology
Actionable? … clear and concise next steps
Aligned? … with criteria ...with scoring box
Appropriate? … professional tone, non-judgmental
Make notes of your questions or things to discuss during Consensus and potential questions for site visit
You know your items and your backups items – Before Consensus, read everyone else’s items.

39. Consensus Call

40. Consensus Call – Roles & Responsibilities
8/20/2019
Consensus Call – Roles & Responsibilities
Team Leader
Present Agenda/Ground Rules
Revisit Key Factors
Discuss Key Themes
Overseer
Ensure integrity of the process
Mediate Criteria questions
Item Lead
Read prepared script summarizing Comments (Str & OFIs), Proposed Score and rationale
Lead discussion of conflicts or concerns
B/U takes notes
Team Members
Discuss item (questions, conflicts, etc.)
Resolve issues
Agree on changes
Listen for conflicts and integration with other Items throughout Report
Offer wordsmithing suggestions in writing only, share with Item lead
Poll Individual Scores - Adopt average of scores if in “same box”, else discuss until consensus is reached
Identify Key Theme concepts
Repeat for all Items
Revisit any Parking Lot issues
Team leader determines sequence for reporting - listed on agenda
Ensure Item leads read their scripts – will save so much time.
Identify Key Theme Concept (review of Key Themes in a few slides)
Complete Key Theme Matrix Form

41. Tips and Ground Rules Allow time and follow Agenda
8/20/2019
Tips and Ground Rules
Allow time and follow Agenda
8 hours (or more) – one day or two half days?
Choose a timekeeper to stay on schedule
Schedule and take breaks
All other devices turned off while on call - no multi-tasking
Everyone in a private/quiet room and use Mute when not talking
Say your name before speaking
“Not used” comments are not discussed unless someone asks
If you disagree or don’t understand, ask questions - don’t cave. Everyone has an equal voice on every item
Go around “room” asking for input, scores, or questions (rotate sequence)
Overseers make good scorekeeper
Use a Parking Lot for longer unresolved discussion items; come back to them later using Consensus Resolution Form
It’s not over until all the work is complete
Discussion stops if anyone leaves the call
TEAM LEADERS – REITERATE THESE ON THE DAY OF CONSENSUS
Ground rules – a basic set are provided – share with team on your first call and see if they agree. May need to add specific rules based on their input.

42. What to have with you during Consensus Call
8/20/2019
What to have with you during Consensus Call
Quiet room, good chair, snacks, phone
If using cell phone, have charger cord
Headset/speaker and Mute function
Computer with Navigator or Printout of FULL Pre-Consensus Comments and Scores
List of your comments/questions/issues to discuss (All Categories)
Criteria Book
Application
Key Factors
Results Spreadsheet
Your Presentation Scripts, Comments Not Used, and any materials for reference
Paper, pens, highlighters, post-it-notes, etc.
The preferred method is to project the comments directly from the Consensus scorebook in Scorebook Navigator.
However, Team Leaders need to provide a paper method backup in the event of internet connection challenges.
It will be a long day for everyone, but it will be much longer if anyone shows up with a bad attitude or an unwillingness to consider the point of view of others.
A couple of people may also wish to bring a power strip – just in case there aren’t enough plug ins close to the table.

43. Team Scoring
If everyone’s scores are within the same band (i.e %) - use the average of the scores (rounded to nearest 5%)
Otherwise, the Overseer will lead a discussion to understand the reasons some examiners gave higher and/or lower scores
The goal is to move the team toward a “consensus” score where variance is <= 20 points and everyone is comfortable within a single band

44. Key Themes

45. What are Key Themes? “Executive Summary” of Feedback Report
Address important strengths or OFIs as reflected in Key Factors – usually 2- 3 Strengths, 2-3 OFIs, 2-3 Results (+/-)
Traceable to Comments/Key Processes
Are cross-cutting (appear in multiple Items) OR
Address a significant issue in one Item (role-model practice or threat to sustainability)
May address a Core Value of the Criteria
Summarize how well Criteria requirements are addressed
May address ADLI or LeTCI
Review slide as a reminder to what makes up a Key Theme
Do not copy and paste comments when creating Key Themes – it’s a summary of the comments
May address a core value, but not required

46. Key Themes Matrix
Core Value & Key Theme Matrix.xlsx

47. How to Fill Out the Key Themes Matrix

48. Where are groups of OFI comments?
Consensus Comment #s
Where are groups of OFI comments?
Where are groups of Strength comments?
Once identified, you can mark these key theme supporting comments as ++ or – in Navigator

49. Writing Key Themes
Writing a Key Theme is similar to writing a comment:
Comment Elements
Key Theme Elements
Criteria being addressed
Core Value being addressed
Key Factor(s)
2-3 Examples from Application
3-4 Examples from Feedback Comments
The team leader will ask for volunteers to write key themes as part of the post consensus process.

50. After the Consensus Meeting

51. Deliverables Edits identified during Consensus
Item Evaluation/Comments (Item Leader)
Score (Item Leader)
New Work created after Consensus
Key Themes (as assigned)
Team Leader will enter Key Themes into Navigator.

52. Post-Consensus Timeline
Category Leaders should refine Item Comments while Consensus discussions are fresh (Within 2 days)
Examiners will draft Key Themes as assigned and submit to the Team Leader (in Word); Team Leader will enter in Navigator
Item Leader/Overseer will review revised comments
Initial Consensus Report due to Awards Office within 7 days after Consensus
Refine your comments based on discussions during the consensus discussion
Team Leader, back- up or other senior examiners will write Key Themes – (discussed at Senior Examiner training)
Scorebook due to Awards Office 7 days following Consensus date

53. Next Steps Team Call(s) Discuss/summarize/clarify Consensus process
Schedule Due dates for Consensus tasks
Review Scoring

54. If team members missed this webinar (or you loved it so much you want to watch it again)
Consensus Training
Slides available on Examiner Resource Center
Recorded session

55. Q&A
Any questions?