Self-Assessing Locally-Designed Assessments Jennifer Borgioli Learner-Centered Initiatives, Ltd.

Handouts qualityrubrics.pbworks.com/DATAG

Organizational Focus Assessment to produce learning… and not just measure learning.

“Less than 20% of teacher preparation programs contain higher level or advanced courses in psychometrics (assessment design) or instructional data analysis.” Inside Higher Education, April 2009

To be assessment savvy….

1999 APA Testing Standards

“The higher the stakes of an assessment’s results, the higher the expectation for the documentation supporting the assessment design and the decisions made based on the assessment results.”

Performance-Based Assessments (PBAs) A performance task is an assessment that requires students to demonstrate achievement by producing an extended written or spoken answer, by engaging in group or individual activities, or by creating a specific product. (Nitko, 2001)

Three Types of Measurement Error Subject effect Test effect Environmental effects

Subject Effects

Testing Fatigue Test Familiarity Bias Score

Test Effects

Final Eyes isn’t about editing rather “is this what you want the students to see/read?”

Test from Period 1 Test from Period 2

Compare with...

Environmental Effects

Reliability = Consistency

Reliability Indication of how consistently an assessment measures its intended target and the extent to which scores are relatively free of error. Low reliability means that scores cannot be trusted for decision making. Necessary but not sufficient condition to ensure validity.

three general ways to collect evidence of reliability Stability: How consistent are the results of an assessment when given at two time-separated occasions? Alternate Form: How consistent are the results of an assessment when given in two different forms?; Internal Consistency: How consistently do the test’s items function?

Cronbach’s Alpha “In statistics, Cronbach's (alpha) is a coefficient of reliability. It is commonly used as a measure of the internal consistency or reliability of a psychometric test score for a sample of examinees. Alpha is not robust against missing data.”

Item Analysis “This isn’t familiar to me”

Percent of Students Selecting Choice “E”

Validity = Accuracy

How do we ensure alignment and validity in assessment? Degrees of Alignment Strong The assessment/learning activity clearly aligns to the target; the assessment/activity and the target are almost one in the same. The language of the standard is explicit. You could confidently infer or conclude the level of student learning/understanding for the target. Moderate The assessment/learning activity addresses the target; the target is included in the learning experience but is not the primary focus. The language of the standard is only partially used. You would need an additional data point to confidently infer the level of student learning/understanding for the target. Weak The assessment/activity misses the target; it might prepare kids for the target, but doesn’t address it. You could not assess level of student learning/understanding for the target.

If you want to assess your students’ ability to perform, design, apply, interpret then assess them with a performance or product task that requires them to perform, design, apply, or interpret.

How many? – 5 standards in a PBA (reflected in rows in the rubric) 3 – 5 items per standard on a traditional test

Minimum

Basic

Articulated

One assessment does not an assessment system make.

Fairness and Bias Fair tests are accessible and enable all students to show what they know. Bias emerges when features of the assessment itself impede students’ ability to demonstrate their knowledge or skills.

In 1876, General George Custer and his troops fought Lakota and Cheyenne warriors at the Battle of the Little Big Horn. In there had been a scoreboard on hand, at the end of that battle which of the following score-board representatives would have been most accurate? A.Soldiers > Indians B.Soldiers = Indians C.Soldiers < Indians D.All of the above scoreboards are equally accurate

What are other attributes of quality assessments?

Standard Error of Measurement An estimate of the consistency of a student’s score if the student had retaken the test innumerable times

How is the SEM calculated?: The SEM is calculated by dividing the SD by the square root of N. This relationship is worth remembering, as it can help you interpret published data. Calculating the SEM with Excel Excel does not have a function to compute the standard error of a mean. It is easy enough to compute the SEM from the SD, using this formula. =STDEV()/SQRT(COUNT()) For example, if you want to compute the SEM of values in cells B1 through B10, use this formula: =STDEV(B1:B10)/SQRT(COUNT(B1:B10)) The COUNT() function counts the number of numbers in the range. If you are not worried about missing values, you can just enter N directly. In that case, the formula becomes: =STDEV(B1:B10)/SQRT(10)

WHEN DESIGNING A PRE/POST PERFORMANCE TASK the standards and thinking demands must stay the same. the modality that students express their thinking through must also stay the same. the content of the baseline and post must be different. the rubrics for the pre/post will be the same in terms of thinking and modality, but the content dimension will be different.

Jennifer