Skills Diagnosis with Latent Variable Models

Topic 1: A New Diagnostic Paradigm

Assessments should aim to improve, and not merely ascertain the status of student learning For test scores to facilitate learning, they need to be interpretative, diagnostic, highly informative, and potentially prescriptive Most large-scale assessments are based on traditional unidimensional IRT models that only provide single overall scores These scores are useful primarily for ordering students along a continuum Introduction

Alternative psychometric models that can provide inferences more relevant to instruction and learning currently exist These models are called cognitive diagnosis models (CDMs) Alternatively, they are referred to as diagnostic classification models (DCMs) CDMs are multiple discrete latent variable models They are developed specifically for diagnosing the presence or absence of multiple fine- grained attributes (e.g. skills, cognitive processes or problem-solving strategies)

Fundamental difference between IRT and CDM: A fraction subtraction example IRT: performance is based on a unidimensional continuous latent trait Students with higher latent traits have higher probability of answering the question correctly

Fundamental difference between IRT and CDM: A fraction subtraction example IRT: performance is based on a unidimensional continuous latent trait Students with higher latent traits have higher probability of answering the question correctly CDM: performance is based on binary latent attribute vector Successful performance on the task requires a series of successful implementations of the attributes specified for the task

Required attributes: (1) Borrowing from whole (2) Basic fraction subtraction (3) Reducing Other attributes: (5) Converting whole to fraction (4) Separating whole from fraction

The response vector of examinee i will be denoted by, The response vector contains J items, as in, The attribute vector of examinee i will be denoted by Each attribute vector or pattern defines a unique latent class Thus, K attributes define latent classes Basic Elements and Notations of CDM

Example: When, the total number of latent classes is Although arbitrary, we can associate the following attribute vectors with the following latent classes:

Like IRT, CDM requires an binary response matrix as input Unlike IRT, CDM in addition requires a binary matrix called the Q-matrix as input The rows of the Q-matrix pertain to the items, whereas the columns the attributes The 1s in the jth row of the Q-matrix identifies the attributes required for item j Basic CDM Input

Examples of Attribute Specification

The goal of CDM is to make inference about the attribute vector The basic CDM output gives the (posterior) probability the examinee has mastered each of the attributes That is, we get For example,, indicates that we are quite certain that examinee has already mastered attribute 1 Basic CDM Output

Each examinee gets a vector of posterior probabilities For reporting purposes, we may want to convert the probabilities into 0s and 1s We can use different rules for this conversion If ; Otherwise,

Example:

Each examinee gets a vector of posterior probabilities For reporting purposes, we may want to convert the probabilities into 0s and 1s We can use different rules for this conversion If ; Otherwise, If ; or If ; Otherwise,

Example: ? – means we do not have sufficient evidence to conclude one way or the other