Classical Test Theory Psych DeShon

Big Picture To make good decisions, you must know how much error is in the data upon which the decisions are based To make good decisions, you must know how much error is in the data upon which the decisions are based Classical test theory is a model that can be used to estimate the magnitude of error present in the data (reliability) Classical test theory is a model that can be used to estimate the magnitude of error present in the data (reliability) Based on strong assumptions Based on strong assumptions

Sources of Error Psychologically interesting variables that matter, but were not included in the model Psychologically interesting variables that matter, but were not included in the model Idiosyncratic Error Idiosyncratic Error Mood, fatigue, boredom, language difficulties, attention Mood, fatigue, boredom, language difficulties, attention Generic Error Generic Error Poor instructions, confusing categories or anchors, setting, variations in administration Poor instructions, confusing categories or anchors, setting, variations in administration Additive Error Additive Error acquiescent, leniency, severity response sets acquiescent, leniency, severity response sets

Sources of Error Systematic errors Systematic errors Social desirability Social desirability Demand characteristics Demand characteristics Experimenter expectancies Experimenter expectancies Halo error Halo error Interviewer bias Interviewer bias Rater dispersion bias Rater dispersion bias Midpoint response sets Midpoint response sets

Classical Test Theory Single Indicator Model Single Indicator Model Most common measurement model Most common measurement model Two assumptions Two assumptions Observed score is a linear combination of true score and error Observed score is a linear combination of true score and error Error is a normally distributed random variable Error is a normally distributed random variable

Classical Test Theory Recall the variance of a linear composite is: Recall the variance of a linear composite is: If you assume that error and true scores are independent, then If you assume that error and true scores are independent, then

Classical Test Theory Signal to noise ratio: Signal to noise ratio: Reliability Reliability

CTT – Estimating Reliability Can't see true score so can't form the reliability ratio directly Can't see true score so can't form the reliability ratio directly Must find some way to estimate it Must find some way to estimate it Parallel Forms!! Parallel Forms!! Original approach to reliability estimation Original approach to reliability estimation Assume you have two exactly equivalent measures of the same latent variable (X & X') Assume you have two exactly equivalent measures of the same latent variable (X & X')

CTT- Parallel Forms If two measures are parallel, then: If two measures are parallel, then: Same true scores Same true scores So, same true score variance So, same true score variance Same error variance Same error variance Therefore, same observed score variance Therefore, same observed score variance

CTT – Parallel Forms te1e1 e2e2 X1X1 X2X2 X i = t + e i

CTT – Parallel Forms Given these relations, the correlation between two parallel forms is an estimate of the reliability Given these relations, the correlation between two parallel forms is an estimate of the reliability

CTT – Standard Error of Measurement Reliability is a metric free measure of the amount of error in a measurement system Reliability is a metric free measure of the amount of error in a measurement system Standard error of measurement translates the reliability estimate into the metric of the measure Standard error of measurement translates the reliability estimate into the metric of the measure

CTT – Standard Error of Measurement Used to place a confidence interval around a particulal observed score Used to place a confidence interval around a particulal observed score Standard error of measurement and cut scores Standard error of measurement and cut scores

CTT- Multiple Indicator Model Also called common factor model Also called common factor model Spearman (1906) Spearman (1906) CTT defines a person's true score as the mean response to an infinite set of observations CTT defines a person's true score as the mean response to an infinite set of observations Operational definition Operational definition Therefore, the best way to get close to the person's true score (less error!) is to obtain responses to a large # of equivalent measures or indicators Therefore, the best way to get close to the person's true score (less error!) is to obtain responses to a large # of equivalent measures or indicators

CTT- Multiple Indicator Model Now, reliability of a linear composite of indicators instead of a correlation between two indicators Now, reliability of a linear composite of indicators instead of a correlation between two indicators X j =an examinees test score on the jth indicator X j =an examinees test score on the jth indicator F is the standardized examinees true score (t ) F is the standardized examinees true score (t ) e j =exminees random error on the jth indicator e j =exminees random error on the jth indicator μ j =the item mean (e.g., difficulty) μ j =the item mean (e.g., difficulty) λ j =factor loading, item sensitivity,item discrimination λ j =factor loading, item sensitivity,item discrimination

t X1X1 X2X2 X3X3 XkXk e1e1 e2e2 e3e3 ekek Unobserved Observed Unobserved A very simple measurement model... r t,e = 0 r e i e j = 0 r x i x j.t = 0 Key assumptions

CTT – Multiple Indicators This represents a set of simple regressions This represents a set of simple regressions One for each indicator One for each indicator Regresses the observed scores onto the latent scores Regresses the observed scores onto the latent scores Only difficulty is that the latent scores aren't observable Only difficulty is that the latent scores aren't observable BUT!, we can observe the effects of the latent scores (assuming a homogeneous model) BUT!, we can observe the effects of the latent scores (assuming a homogeneous model)

CTT – Multiple Indicators Some neat consequences... Some neat consequences... Covariance of scores on any 2 indicators is: Covariance of scores on any 2 indicators is: Variance of the j th indicator Variance of the j th indicator Variance due to true and error Variance due to true and error So, if we know the parameters we can compute the variances and covariances So, if we know the parameters we can compute the variances and covariances

CTT – Multiple Indicators Can also reverse this process Can also reverse this process If we know variances and covariances, we can compute the parameter estimates If we know variances and covariances, we can compute the parameter estimates Factor loadings may be obtained from the covariance of any three items Factor loadings may be obtained from the covariance of any three items

CTT – Multiple Indicators Can also reverse this process Can also reverse this process Error variances (aka uniquenesses) may be obtained via subtraction as: Error variances (aka uniquenesses) may be obtained via subtraction as:

CTT – Multiple Indicators Take 5 items with the following parameters Take 5 items with the following parameters

CTT – Multiple Indicators Imply the following covariance matrix Imply the following covariance matrix

CTT – Multiple Indicators You can compute the parameters from the variances and covariances You can compute the parameters from the variances and covariances You can compute the variance and covariances from the parameters You can compute the variance and covariances from the parameters Ex: covariance of items 3 and 5 Ex: covariance of items 3 and 5 Ex: factor loading for item 4 Ex: factor loading for item 4

CTT – Multiple Indicators This is an ideal example This is an ideal example Model fits perfectly Model fits perfectly No sampling error – we know the parameters No sampling error – we know the parameters In reality, all the various ways of estimating the parameters yield slightly different estimates In reality, all the various ways of estimating the parameters yield slightly different estimates Need a way to cope with this (disrepancy function) Need a way to cope with this (disrepancy function) Provides a conceptual intro to model identification Provides a conceptual intro to model identification

CTT – Multiple Indicators Parallel Indicators Parallel Indicators Equal lambdas (loadings) Equal lambdas (loadings) Equal error variances Equal error variances Tau-Equivalent Indicators Tau-Equivalent Indicators Equal lambdas Equal lambdas Congeneric Indicators Congeneric Indicators Nothing needs to be equal Nothing needs to be equal Can compare model fit to determine Can compare model fit to determine

CTT – Multiple Indicators Reliability of a homogeneous set of indicators Reliability of a homogeneous set of indicators Reliability is the ratio of true score variance to total variance Reliability is the ratio of true score variance to total variance Can be estimated directly from the common factor model Can be estimated directly from the common factor model

CTT – Multiple Indicators Omega is: Omega is: Ratio of variance due to the common attribute to the total variance in X Ratio of variance due to the common attribute to the total variance in X Square of the correlation between X (observed score) and the the common factor Square of the correlation between X (observed score) and the the common factor Correlation between two parallel test scores Correlation between two parallel test scores Square of the correlation between the total score of m indicators and the total score of an infinite set of indicators Square of the correlation between the total score of m indicators and the total score of an infinite set of indicators

CTT – Multiple Indicators Coefficient alpha Coefficient alpha If the tau-equivalent model holds! If the tau-equivalent model holds! This quantity can be estimated directly from the variance covariance matrix This quantity can be estimated directly from the variance covariance matrix

CTT – Multiple Indicators If the tau-equivalent model holds... If the tau-equivalent model holds... If not tau-equivalent, alpha will be less than omega If not tau-equivalent, alpha will be less than omega Notice m! - leads to the spearman-Brown Notice m! - leads to the spearman-Brown The number of indicators The number of indicators

CTT – Multiple Indicators Standards for alpha Standards for alpha.7 has become the standard.7 has become the standard Nunnaly was the origin Nunnaly was the origin.7 for research.7 for research.9 for application.9 for application Why? - The SEM is huge for.7 Why? - The SEM is huge for.7