CARLY GUINN COLLEGE OF WILLIAM AND MARY CRIN 550, DR. HINDMAN Assessment Analysis and Critique

Purpose of the Test Administered: March 7, 2013  Matoaka Elementary School  3 rd grade  21 students A review of second grade concepts: Ancient Egypt and China  Test designed to meet the new rigor of SOL tests  Multiple correct responses  Sorting through multiple layers Administered during the 40-minute block they are used to using for Social Studies

Table of Specifications (Select-Response Questions) Question # (% correct)

Table of Specifications (Supply-Response Questions) 1. Student writes a paragraph using complete sentences, accurate punctuation, indentation, and appropriate spelling (2 points) (90%) 2. Student identifies one plausible way in which the culture of Ancient China was affected by its environment/land (4 points) (43%) 3. Student identifies one plausible way in which the culture of Ancient Egypt was affected by its environment/land (4 points) (62%) 4. Student accurately identifies an invention of Ancient Egypt or Ancient China that was built due to the environment of the respective region (3 points) (86%) *(If more than one invention is provided, points will only be given for one correct answer) 5. Student provides the correct usage of the invention identified (2 points) (67%) Rubric Item (point value) (% correct)

A Final Note on Scoring… Questions 1-10,  Select-response  One point each  13 points total Questions  Multiple-response questions  One point per correctly-selected item; points deducted from collected points for incorrect response  8 points total Question 28  Supply-response: short answer question  5 rubric items; “all or nothing” for each item  15 points total 46 points total

Students in the Aggregate Class Scores High score45 = 95% Low score24 = 52% Mean36.95/46 = 80% Mode32, 41, 45 = 70%, 89% 98% (3 responses each) Median39 = 85% Results by Gender Female (11) Mean: 36.6 = 80% Mode: 32, 41, 42 = 70%, 89%, 91% (2 responses each) Median: 39 = 85% Male (10) Mean: 37.3 = 81% Mode: 35, 40, 45 = 76%, 87%, 98% (2 responses each) Median: 37.5 = 82% Results by Ethnicity White (18) Mean: 37.6 = 82% Mode: 41, 45 = 89%, 98% (2 responses each) Median: 40 = 87% Hispanic (3) Mean: 33 = 72% Mode: n/a (one response for each score) Median: 33 = 72%

Students in the Aggregate Class Scores High score45 = 95% Low score24 = 52% Mean36.95/46 = 80% Mode32, 41, 45 = 70%, 89% 98% (3 responses each) Median39 = 85% Results by Reading Level “High” readers (above grade level) (9) Mean: 40.3 = 88% Mode: 45 = 98% Median: 41 = 89% “Average” readers (on grade level) (9) Mean: 36.8 = 80% Mode: 32, 35, 40 = 70%, 76%, 87% (2 responses each) Median: 35 = 76% “Low” readers (below grade level) (3) Mean: 27.3 = 59% Mode: n/a (one response for each score) Median: 27 = 59%

Student Profiles: Student A Student Description  Caucasian female  Low reader (below grade level), works with reading specialist  Low-SES (receives free/reduced lunch)  Very little family support: rarely returns notes signed, does not receive help with homework Student A’s Scores Questions 1-108/10 Questions /13 Questions /8 Short Answer7/15 TOTAL31 = 67% Student missed questions: 5, 9, 13, 22, 24 Student missed short answer items 2 and 3 Student missed questions: 5, 9, 13, 22, 24 Student missed short answer items 2 and 3 All questions missed were at the comprehension/application level Student was significantly impacted by having to write a short answer response

Student Profiles: Student B Student Description  Caucasian male  Average reader (reads just below mid-year third grade level)  Does not receive free/reduced lunch; active in sports and extracurricular activities  High level of family support; parents often participate in school-related functions  “Fence sitter” Student B’s Scores Questions 1-109/10 Questions /13 Questions /8 Short Answer13/15 TOTAL40 = 87% Student missed questions: 6, 13, 14, 22 Student missed short answer item 5 Student missed questions: 6, 13, 14, 22 Student missed short answer item 5 Student missed most questions from SOL 2.4b; no questions were missed from the “knowledge” level Student was affected by questions that were designed to reflect the rigor of the new SOLs

PRINCIPLES OF ASSESSMENT QUESTION FORMATS SCORING DECISIONS BENEFITS FOR STUDENTS: SHORT- AND LONG-TERM CREATING A VALID AND RELIABLE ASSESSMENT Instructional Decisions

Principles of Assessment Judge student learning  Provided with detailed information regarding students’:  Knowledge of content  Ability to respond to various question formats  Ability to apply their knowledge of content to written responses Improve student learning  Teacher understands what material must be reviewed  Finds “gaps” in learning and understanding  Opportunity to discuss new question formats with students  Prepares students for SOLs  Emphasizing reading directions!  Review strategies and techniques for addressing writing prompts  Teacher-lead “think aloud”

Question Formats Multiple choice Multiple response  Students required to select more than one answer on three of four questions  Students needed to read directions carefully Matching  Multi-level matching  Student needed to first understand what a “contribution” was  Determine if it was from Egypt or China Short-Answer  Take content knowledge and apply it to a writing prompt  Analysis-level thinking  Also graded on “third-grade” writing

Scoring Decisions Multiple response items  Needed to give points to accurately address question format  Students accumulate points throughout the test – inability to “lose” points for each question Short answer items  Rubric was clearly defined for students in the question itself (expectations were clear)  Points per rubric item were proportionally related to the intended difficulty level and thought-time requirement  Assessment of student writing – expected to “help” students

Short-Term Long-Term Reviewing second- grade knowledge relevant for SOL Assessment tells students what they know and what they should review prior to the SOL Provides practice with new question formats and expectations “Think aloud” following test review provides information about ways to address writing prompts Benefits for Students

Validity Reliability Test items were aligned with:  Bloom’s Taxonomy  VA standards (SOL Blueprints)  Instructional time Looked like a test (especially like practice SOL tests) Test was administered directly following instruction of material Question design reflected rules specified by Gareis and Grant Questions accurately evaluated students’ understanding of content unless student guessed correctly Test was reviewed by another W&M CRIN 550 student Possible threats  Random error  Students’ discomfort with question format Taking Precautions

Edits and Modifications for the Future Ensure ample instruction is given on most difficult topics Short answer questions Give students instruction regarding question formats prior to test

CARLY GUINN Assessment Analysis and Critique