1. 1 Power of Reviewing Standards Consortia Conference Call March 15, 2005 Susan Pimentel Consortia Conference Call March 15, 2005 Susan Pimentel

2. 2 Welcome Evaluation! Benefits: Ensures standards are appropriate, clear, accurate and complete Ensures standards are appropriate, clear, accurate and complete Builds ownership and awareness Builds ownership and awareness Provides ideas about rollout of standards. professional development, etc. Provides ideas about rollout of standards. professional development, etc.

3. 3 Make it Multi-Faceted! 1. Internal Review: Determine how the standards drafts align to the research and other documents as required (e.g., GED, K-12 standards, NCTM) {Alignment} 2. Field Review: Ensure standards are understandable and acceptable to field who will be charged with standards’ implementation. {Validity} 3. External Expert Review: Focus here is primarily on the sufficiency and understandability of the content. {Validity} {Validity}

4. 4 Tips for Internal Review 1. Ask members of writing teams to conduct. 2. Get specific about how standards align to research or other docs. Are they: Congruent?Congruent? Aligned?Aligned? Contradictory?Contradictory? 3. Document extent of alignment (develop written rationale or correlation chart).

5. 5 Tips for Field Review: Focus Groups 1. Focus Group: Involve instructors, program directors, etc.Involve instructors, program directors, etc. Keep groups small (10-15)Keep groups small (10-15) Plan on 3 hoursPlan on 3 hours Be prepared to walk through the docsBe prepared to walk through the docs Employ members of writing teams to runEmploy members of writing teams to run Send invitations and drafts well in advance of sessionSend invitations and drafts well in advance of session Hold sessions around key regions of the stateHold sessions around key regions of the state

6. 6 Focus Groups, cont’d. No defensiveness allowed!! Time to listen and inquireNo defensiveness allowed!! Time to listen and inquire Develop feedback forms/surveysDevelop feedback forms/surveys Document conversations/feedbackDocument conversations/feedback Report back to focus groups (meet again if possible)Report back to focus groups (meet again if possible)

7. 7 Tips for Field Review: Survey Test survey with members of your writing team Test survey with members of your writing team Mail out to programs w/personal appeal to directors Mail out to programs w/personal appeal to directors Make it web-based as well Make it web-based as well Collect quantitative and qualitative information Collect quantitative and qualitative information Ask about more than just the quality of the standards (e.g., roll out ideas, prof. dev.) Ask about more than just the quality of the standards (e.g., roll out ideas, prof. dev.) Consider using a general question survey and standards specific survey Consider using a general question survey and standards specific survey See attached draft survey.

8. 8 Combo is Best! 1. Pros of Focus Group: Personal and in-depth inquiry possible (but time consuming and can only accommodate small numbers) 2. Pros of Surveys: Able to reach large numbers and time and cost efficient (but can’t inquire, explain, discuss and promote greater understanding). Combo allows gives you depth and breadth; allows you to gather rich qualitative and quantitative information.

9. 9 Tips for External Expert Review 1. Develop a set of questions for the expert reviewer. Include key aspects of the academic domain? Any gaps? Include key aspects of the academic domain? Any gaps? Clear and specific enough to inform instruction? Clear and specific enough to inform instruction? Adherence to research? Adherence to research? Appropriate rigor? Appropriate rigor? Reasonable and manageable? Reasonable and manageable? Proper sequence or progression of skills? Proper sequence or progression of skills? 2. Be clear about how specific you want the reviewer to be (e.g., general evaluation and recommendations or specific suggestions for wording changes and refinements). 3. Need to give “experts” a month or more lead-time.

10. 10 Pros of Expert Review They are: Current on research Current on research Familiar with expectations across the nation (and perhaps the world!) so they offer a broad perspective Familiar with expectations across the nation (and perhaps the world!) so they offer a broad perspective Independent Independent Able to recognize common pitfalls Able to recognize common pitfalls Offer specific recommendations about how to strengthen your standards Offer specific recommendations about how to strengthen your standards Able to provide standards with a “stamp of approval” that you can present to legislators, policy makers, etc. Able to provide standards with a “stamp of approval” that you can present to legislators, policy makers, etc.

11. 11 Prepare Teams for Review 1. Reviewers tend to tell you what they don’t like rather than what they do like. 2. Read between the lines: Standards mean change and change can be scary so important to hold on to your principles (e.g., must align to the research, prepare adults for success). 3. Prepare for conflicts in the reviews and determine means to grapple with the contradictions (go back to why you asked a group or individual to give you feedback).

12. 12 Prepare Teams, cont’d. 4. Let teams know that feedback offers a set of suggestions, not mandates. Having said that, let teams know when it is just a matter of style or the change won’t diminish the draft, make the change even if it isn’t their preferred choice. It promotes ownership. Having said that, let teams know when it is just a matter of style or the change won’t diminish the draft, make the change even if it isn’t their preferred choice. It promotes ownership. 6. 6. Set the expectation that teams will review every suggestion carefully and either adjust the standards or note a rationale for not accepting the feedback. The End