1. Feedback Scale Assessment 1. Use a structured rating instrument 2. Norm your rating instrument 3. Train or instruct your raters 4. Monitor results and make adjustments

2. Testing an existing Scale a) “Back of the envelope” agreement coefficient: # agreements / # agreements + # disagreements b) Item variability index:   rater 1 – rater 2  2 C) Response Frequencies

3. Fixing Problems Spend time on anchors and assumptions Change mushy wording Discuss items that produce high variability Eliminate bad items Are some items “global in sheep’s clothes”?

4. Rater Support is Key Provide clear directions Do direct training if possible Employ physical examples if possible Discuss disagreements

5. In Essence… Anchor effectively Document process Train raters a little Assess your instrument