1. Directions for Expert Review Panel
Directions for Expert Review - Establishing Content Validity for COED rubrics
Establishing Content Validity for COED rubrics

2. Thank you! Thank you for this work!
This work is integral to our accreditation. We sincerely appreciate your time and willingness to participate.
If you have any questions, please ask.
Let’s begin …
First of all, Thank you for this work! This work is integral to our accreditation. We sincerely appreciate your time and willingness to participate. We cannot move forward in this effort without the assistance of our expert panel partners. Finally, If you have any questions as you complete this task, please don’t hesitate to ask. Let’s begin …

3. Process Reviewing Materials Completing Tasks
Submitting the Information
Should not take too much of your time (we hope!)
We will begin by talking through each phase of the expert panel review process. The three phases of this work include: reviewing the materials, completing the tasks, and then winding up with the final steps by submitting the information back to UNC charlotte. Finally, while we want your feedback to be thorough, we don’t anticipate that this process will take great amounts of time. It’s fairly straightforward and we’ve tried to make it as painless as possible.

4. Reviewing Materials Hard copies or electronic versions
Letter of purpose (may be in the body of )
A copy of the assessment instructions provided to candidates.
A copy of the rubric used to evaluate the assessment.
The response form aligned with the assessment/rubric for the panel member to rate each item.
We begin by reviewing the materials sent to you. These materials may be hard copies or more likely, electronic versions of document. These documents should include 1) A letter of purpose, explaining why you were selected for this work. This may be in the body of an or it may be a separate document; 2) A copy of the assessment instructions provided to candidates; 3) A copy of the rubric used to evaluate the assessment; and 4) The response form aligned with the assessment/rubric for the panel member to rate each item. While the first three items are informative, the last item, the response form, is where you will record your responses.

5. Completing your Tasks
Goal = establishing content validity for each rubric
“Validity” - to what extent the item measures the key construct it purports to measure.
Key construct, or “overarching idea,” should be identified on your response form, along with an operational definition of the key construct.
Example …
The primary objective of this process is to establish content validity for each rubric. As an expert reviewer, your feedback ratings on the response form help to determine the rubric’s validity. For clarification, we are using the term “validity” to determine to what extent the rubric measures what we (as the authors of the rubric) claim it does. Your primary task is to rate each of the items on the rubric in terms of to what extent the item measures the key construct it purports to measure. Key construct, or “overarching idea,” should be identified on your response form, along with an operational definition of the key construct, so you understand exactly how UNC charlotte faculty are using the term. As an example, we will use a piece from one of our currently used rubrics.

6. Example:
We use a rubric measuring multiple constructs. One of those constructs is CONTENT KNOWLEDGE.
The operational definition we are using for CONTENT KNOWLEDGE is “Knowledge about actual subject matter that is to be learned or taught.”
Notice this definition changes depending on the grade level/content area. This is why reviewers are selected based on their expertise.
For example, in the College of Education, we use a rubric measuring multiple constructs. One of those constructs is CANDIDATE CONTENT KNOWLEDGE. The operational definition we are using for CANDIDATE CONTENT KNOWLEDGE is “A candidate’s knowledge about actual subject matter that is to be learned or taught.” Notice this definition changes depending on the grade level/content area. This is why reviewers are selected based on their expertise.

7. Example
On our rubric, there are currently three items that are collectively used to measure CANDIDATE CONTENT KNOWLEDGE:
A: Demonstrates knowledge of content
B: Demonstrates awareness of literacy instruction across all content areas
C: Makes content relevant for all learners
On our rubric, there are currently three items on the rubric that are collectively used to measure CANDIDATE CONTENT KNOWLEDGE: A: Demonstrates knowledge of content; B: Demonstrates awareness of literacy instruction across all content areas; and C: Makes content relevant for all learners. In considering this example, most reviewers would probably agree that A, demonstrates knowledge of content is important in considering the overall construct of CONTENT KNOWLEDGE. But what about B, demonstrating awareness of literacy instruction? While definitely something that all educators should be knowledgeable about, does “demonstrating awareness of literacy instruction” measure CANDIDATE CONTENT KNOWLEDGE, as it is defined here? Some educators may consider B to be less important in measuring CANDIDATE CONTENT KNOWLEDGE, depending on their perspective and experience. This is why we ask multiple content experts to provide their professional opinions in considering each of these key concepts. Going back to consider item A again, we may consider it to be vague in measuring CONTENT KNOWLEDGE. Item A, in essence, simply repeats the key construct. An expert reviewer may determine that Item A is too broad and needs clarification to be effective. This is the kind of feedback we seek from our expert panel of review.

8. What am I rating? How Representative the Item is of Key Construct
How Important the Item is in measuring Key Construct
How Clear the Item is
Rate each item on scale of 1-4
Each item on the response form is rated in terms of how Representative the item is in measuring the key construct, how important the item is, and how Clear the item is. Each item is rated on a scale of 1-4.

9. Rate from 1-4
Representativeness of item in measuring the overarching construct
1 = item is not representative
2 = item needs major revisions to be representative
3 = item needs minor revisions to be representative
4 = item is representative
The first thing to assess will be Representativeness. Using a scale from 1-4, reviewers will rate the Representativeness of each item in measuring the overarching construct. 1 = item is not representative, 2 = item needs major revisions to be representative, 3 = item needs minor revisions to be representative, or 4 = item is representative.

10. Rate from 1-4
Importance of item in measuring the overarching construct
1 = item is not necessary to measure the construct
2 = item provides some information but is not essential to measure the construct
3 = item is useful not but essential to measure the construct
4 = item is essential to measure the construct
Next, using a scale from 1-4, reviewers will rate the Importance of item in measuring the overarching construct.
1 = item is not necessary to measure the construct
2 = item is provides some information but is not essential to measure the construct
3 = item is useful not but essential to measure the construct
4 = item is essential to measure the construct

11. Rate from 1-4 Clarity of item 1 = item is not clear
2 = item needs some major revisions to be clear
3 = item needs some minor revisions to be clear
4 = item is clear
Finally, reviewers will rate the Clarity of item
1 = item is not clear
2 = item needs major revisions to be clear
3 = item needs minor revisions to be clear
4 = item is clear

12. Recording your Responses
Response form or link to it in materials
Rate each item  how well it measures key constructs
Are there enough items to measure the construct?
Other feedback
Key Construct
Item 1
Item 2
Item 3
In your materials from UNC charlotte, you should have received either a copy of your response form or an electronic link to it. Following the directions as explained here, please go through and rate each of the items on the rubric as to how well they measure the identified key constructs. Remember: one construct may have multiple items to measure it, or it may only have one item. Part of your feedback includes open ended responses: are there enough items to measure the construct? Too many? What else would you like to share with us about measuring this item? Please give your feedback in the open-ended responses that should be provided on your response form.

13. One last thing …
You’ve been sent copies of the rubric and the original student assignment in your packet of materials. These materials are for your information … you are not assessing the assignment or the entire rubric – you’re only looking at how well the rubric measures what it purports to measure.
Rubric indicators are not part of this (inter-rater reliability next year!)
One last thing: You’ve been sent copies of the rubric and the original student assignment in your packet of materials. These materials are for your information … you are not assessing the assignment or the entire rubric – you’re only looking at how well the rubric measures what it purports to measure. Rubric indicators are not part of this … we will look at the rubric indicators when we assess inter-rater reliability next year.

14. When finished …
Please return the response form to your UNC Charlotte contact.
If you have any questions, please contact us.
THANK YOU for your time and willingness to assist us in this important accreditation work.
When you’re finished, please return the response form to your UNC Charlotte contact. If you have any questions, please contact us. And once again, THANK YOU for your time and willingness to assist us in this important accreditation work.