Test Design & Construction RSCH 6109: Assessment & Evaluation Methods Test Design & Construction Purpose & Framework Test Specifications or Blueprint Item Construction Field Testing Evaluation & Revision

Evaluating the Value of a Test RSCH 6109: Assessment & Evaluation Methods Evaluating the Value of a Test The value of a test can only be determined relative to its purpose and intended use Value = Reliability, Validity, Cultural Sensitivity However, “value” is not a property of the test itself Reliability, Validity, and Cultural Sensitivity are properties of the information the test provides in the context of the administration, scoring, interpretation, and application to a particular population Contrast Proper and Improper use of the BDI

The Pilot Study RSCH 6109: Assessment & Evaluation Methods Give the test to a small group from the target audience Ask them to complete the test and to give feedback about the test How long did it take you to complete the test? Anything ambiguous / confusing items or wording? Anything not covered / suggestions for additional questions? Talk Out Loud Method / Focus Groups Open-ended questions about the construct

Descriptive Statistics – Item Analysis RSCH 6109: Assessment & Evaluation Methods Descriptive Statistics – Item Analysis Item difficulty Distribution of item responses Item discrimination Correlations between items Begin the process of eliminating or modifying item content

Reliability Studies RSCH 6109: Assessment & Evaluation Methods Internal consistency (Consistency) Split-half reliability (Consistency) Test-retest reliability (Stability) Inter-rater reliability (Agreement) Further the process of eliminating or modifying items

Construct Validity RSCH 6109: Assessment & Evaluation Methods What is the test really measuring? How many sub-scales should there be? Exploratory factor analysis Define the subscales that emerge Consider eliminating or adding items Higher order factors

Content Validity RSCH 6109: Assessment & Evaluation Methods Does the test really cover the content domain? Expert review Comparison to theory Consider additional items

Further Validation RSCH 6109: Assessment & Evaluation Methods Criterion Validity Concurrent Validity / MTMM matrix Predictive Validity Known Group Differences Validity Confirmatory Factor Analysis

Cultural Sensitivity RSCH 6109: Assessment & Evaluation Methods Expert review DTF – Differential Test Functioning DIF – Differential Item Functioning

Final Stages RSCH 6109: Assessment & Evaluation Methods Create Norm Tables Create Norm Tables for Sub-groups Prepare the technical manual Guidelines for administration Scoring Interpretation Technical properties Compare the reliability, validity, and cultural sensitivity evidence and technical manual to the Standards