Assessment Literacy Series 1 -Module 6- Quality Assurance & Form Reviews

SAM-L Development participants will: 1. Apply a quality assurance checklist to the developed materials to ensure all documents meet the requirements. 2. Verify the item is measuring the intended content and standard. 3. Ensure the scoring key/rubrics are correct. 2 Objectives

Participants may wish to reference: Guides Handout #10 – Quality Assurance Checklist QA Screening Tool Performance Measure Rubric-Teacher Other “Stuff” “Smart Book” 3 Helpful Tools

4 Outline of Module 6 Module 6: Form Reviews Content Review Fairness, Sensitivity, Bias Apply Checklist to all forms Alignment Rigor

Form Review Reviewing forms involves the following components:  Content Review  Bias  Fairness  Sensitivity  Accessibility  Alignment Note: Refer to “Smart Book” for detailed explanations of these components. 5

Content Review  Determine if each item/task clearly aligns to the targeted content standard.  Evaluate all items for accuracy.  Judge if each item is age and grade appropriate for students. Reading level is appropriate. Vocabulary is appropriate. Required reasoning skills are appropriate.  Review each item stem and answer options to be sure they are clear and understandable. Student response expectations are clear. 6

Item Bias  Bias is the presence of some characteristic of an item that results in the differential performance of two individuals with the same ability but from different subgroups.  Bias is not the same as stereotyping.  Bias-free items/tasks provide an equal opportunity for all students to demonstrate their knowledge and skills. 7

Fairness and Sensitivity Review to ensure:  Items are sensitive to different cultures, religions, ethnic and socio-economic groups, and disabilities.  There is a balance of gender roles.  Use of positive language, situations, and images.  Items avoid text that may elicit strong emotions in specific groups of students and prevent those students from accurately demonstrating their skills and knowledge. 8

Sensitive Topics to Avoid Abortion Birth control Child abuse/neglect Creationism Divorce Incest Illegal activities Occult/witchcraft Questioning parental authority Rape Religious doctrine Sex/sexuality Sexual orientation Weight Suicide Sexually transmitted diseases *Note: The above list does not contain every sensitive topic that item developers should avoid and does not consider test-taker’s age/maturity. 8

What is Accessibility?  Test accessibility is defined as the extent to which a test eliminates barriers and permits the test-taker to demonstrate his or her knowledge of the tested content.  All items should be reviewed to ensure they are accessible to the entire population of students.  Item reviews must consider: Readability Syntax complexity Item presentation Font size Clarity of images, graphs, tables Amount of work space 10

Alignment Categorical Concurrence ◦ The same categories of the content standards are included in the assessment ◦ Items might be aligned to more than one standard Source of Challenge ◦ Students give a correct or incorrect response for the wrong reason (bias) Balance of Representation ◦ Ensures there is an even distribution of the standards across the test Range of Knowledge ◦ The extent of knowledge required to answer correctly parallels the knowledge the standard requires Depth of Knowledge ◦ The cognitive demands in the standard must align to the cognitive demands in the test item Source: Webb, N. L. (1997). Research Monograph No. 6: Criteria for alignment of expectations and assessments in mathematics and science education. Washington, DC: Council of Chief State School Officers 11

Quality Assurance QA Screening Tool ◦ Use to examine first draft materials ◦ Outlines key criteria Quality Assurance Checklist [Handout #10] ◦ Use with final drafts ◦ Examines each developed component of the performance measure. Performance Measure Rubric-Teachers ◦ Comprehensive self-assessment of teacher- made performance measures. 12

Process Steps 1. Apply the Quality Assurance Checklist [Handout #10] to the performance measure and supporting documents. Apply the checklist to each individual item /task, and then evaluate the measure in its totality. 2. Make notes directly on the paper copies of the documents. 3. Review areas that may need additional focus. 4. Print and conduct a final team review of all documents, particularly the operational form and scoring keys. 5. Turn your packet in to the facilitator, and destroy draft material. 13

14 Summary Module 6: Form Reviews Evaluated the documents created by the performance measure development process, and ensured the measures were of high-quality, rigorous, and aligned to the targeted content standards. All documents went through a quality assurance process prior to finalization.