1 Causal Rasch Models and Individual Growth Trajectories National Center for the Improvement of Educational Assessment January 18, 2011 A.Jackson Stenner Chairman & CEO, MetaMetrics

2 “Although adopting a probabilistic model for describing responses to an intelligence test, we have taken no sides in a possible argument about responses being ultimately explainable in causal terms.” (Rasch, 1960, p.90)

3 Three well researched constructs  Reader ability  Text Complexity  Comprehension

4 Reader Ability Temperature

5 Reading is a process in which information from the text and the knowledge possessed by the reader act together to produce meaning. Anderson, R.C., Hiebert, E.H., Scott, J.A., & Wilkinson, I.A.G. (1985) Becoming a nation of readers: The report of the Commission on Reading Urbana, IL: University of Illinois

6 An Equation = Reader Ability Text Complexity Comprehension - Conceptual Statistical Raw Score = i e (RA – TC ) i 1 + e (RA – TC i ) RA = Reading Ability TC = Text Calibrations

7 Each of these thermometers is engineered to use the same correspondence table Each of these reading tests is engineered to use the same correspondence table

8 Correspondence Table: C o and Lexile Raw Score CoCo Lexile Raw Score CoCo Lexile Raw Score CoCo Lexile Raw Score CoCo Lexile L L L L L L L L L L L L L L L L L L L L L L L L L L L L L L L L L L L L L L L L L L L L

9 Aspect/ConstructTemperatureReader Ability Object of measurementPerson InstrumentThermometerReading test Measurement outcomeNumber of theory calibrated cavities (0-45) that fail to reflect green light Count correct on a collection of 45 theory calibrated test items Substantive theoryThermodynamic theoryLexile Theory Unit of measurementDegree Fahrenheit ( o F)Lexile (L) Correspondence table/ calibration equation Exploits a chemical reaction and light absorption to table temperature as a function (Guttman Model) of a sufficient statistic Exploits semantic and syntactic features of test items to table reader ability as a function (Rasch model) of a sufficient statistic Measure/QuantityMeasurement outcome converted into a quantity via the substantive theory Measurement outcome converted into a quantity via the substantive theory Readable technologyNexTemp Thermometer™Oasis™ General objectivityPoint estimates of temperature are independent of the thermometer Point estimates of reader ability are independent of the reading test Anatomy of Two Measurement Procedures

10 Ten Features of Causal Response Models – whether Guttman or Rasch 1. Both measurement procedures depend on within-person causal interpretations of how these two instruments work. NexTemp uses a causal Guttman Model, The Lexile Framework for Reading uses a causal Rasch Model. 2. In both cases the measurement mechanism is well specified and can be manipulated to produce predictable changes in measurement outcomes (e.g. percent correct or percent of cavities turning black). 3. Item parameters are supplied by substantive theory and, thus, person parameter estimates are generated without reference to or use of any data on other persons or populations. Therefore, effects of the examinee population have been completely eliminated from consideration in the estimation of person parameters for reader ability and temperature.

11 4. In both cases the quantitivity hypothesis can be experimentally tested by evaluating the trade-off property. A change in the person parameter can be off-set or traded-off for a compensating change in the measurement mechanism to hold constant the measurement outcome. 5. When uncertainty in item difficulties is too large to ignore, individual item difficulties may be a poor choice to use as calibration parameters in causal models. As an alternative we recommend, when feasible, averaging over individual item difficulties to produce “ensemble” means. These means can be excellent dependent variables for testing causal theories. 6. Index models are not causal because manipulation of neither the indicators nor the person parameter produces a predictable change in the measurement outcome. Ten Features of Causal Response Models – whether Guttman or Rasch cont’d.

12 7. Causal Rasch models are individual centered and are explanatory at both within-subject and between-subject levels. The attribute on which I differ from myself a decade ago is the same attribute on which I differ from my brother today. 8. When data fit a Rasch model differences between person measures are objective. When data fit a causal Rasch model absolute person measures are objective (i.e. independent of instrument). 9. The case against an individual causal account, although popular, has been poorly made. Investigators need only experiment to isolate the causal mechanism in their instruments, test for the trade-off property and confirm invariance over individuals. This has been accomplished for a construct, reader ability, that has been described by scholars as the most complex cognitive activity that humans regularly engage in. Given the success with reading, we think it likely that other behavioral constructs can be similarly measured. 10. Causal Rasch models make possible the construction of generally objective growth trajectories. Each trajectory can be completely separated from the instruments used in its construction and from the performance of any other persons whatsoever. Ten Features of Causal Response Models – whether Guttman or Rasch, cont’d.

13 To causally explain a phenomenon [a measurement outcome] is to provide information about the factors [person processes and instrument mechanisms] on which it depends and to exhibit how it depends on those factors. This is exactly what the provision of counterfactual information…accomplishes: we see what factors some explanandum M [measurement outcome, raw score] depends on (and how it depends on those factors) when we have identified one or more variables such that changes in these (when produced by interventions) are associated with changes in M (Woodward, 2003, p.204).

14 How Many Ways Can We Say X Causes Y? X “elicited a greater” YX “impacts” Y X “accounts for” YX “has been linked to” Y Y “is the result of” XX “didn’t diminish” Y Y “because of” XY “depends on” X X “has led to” YX “largely motivates” Y Y “stemmed from” XX “proved critical to” Y X “fosters” YX “changes” Y X “triggers” YX “affects” Y

15 Psychometrics vs. Metrology Aspect Interpretation of Probability Group Centered Interpretation involves 100 people with the same ability answering a single item Individual Centered Interpretation involves administering 100 items with the same calibration to a single person Person MeasuresA person’s response record is embedded in different samples and each group specific Rasch analysis produces a different measure A person’s response record is evaluated against theory-referenced calibrations Measurement ErrorTraditional test theory uses a sample standard deviation and a sample correlation to compute an SEM which is intended to characterize the individual ISEM is the within person standard deviation over replications of the measurement procedure Data Fit to the ModelVaries with the locally constructed frame of reference, sample dependent Fit is to a theory, thus, sample independent ValidityCorrelational, thus, sample dependent Causal within person, thus, sample independent

16

17

18 r = r” = R 2” = RMSE” = 99.8L Figure 1: Plot of Theoretical Text Complexity versus Empirical Text Complexity for 475 articles “Pizza Problems”

19 What could account for the 8% unexplained variance?  Missing Variables  Improved Proxies/Operationalizations  Expanded Error Model  Rounding Error  Interaction between Individual and Text  Psychometric Uncertainty Principle

20

21 May 2016 (12 th Grade) Text Demands for College and Career May 2007 – Dec Encounters 117,484 Words 2,894 Items 848 Minutes Student th Grade Male Hispanic Paid Lunch

22 Item-Based vs. Ensemble-Based Psychometrics

23 Reading Task-Complexity Plane for Dichotomous Items Native Lexile Added Hardness Added Easiness Production Cloze Auto-Generated Cloze Unit Size Adjustment Applied to Logits

24 Comparing Item-Based vs. Ensemble- Based Psychometrics  Item-Based – Item statistics – Item characteristic curves – DIF for items  Ensemble-Based – Ensemble statistics – Ensemble characteristic curves – DIF for ensembles

25 The Ensemble  Objective: Correspondence Table – Raw score to Lexile measure  What we think we know – Mean and spread of item distributions for a passage  What is assumed to be unknown – Individual item difficulties 1300L (132L)

26 The Process – Iteration 1 STEP 1 Sample 45 Item Difficulties from Ensemble STEP 2 Compute Lexile Measures for Each Raw Score (1 to 44) STEP 3 Table Results Raw Score Lexile Measure 362L 514L 584L L Sample 1

27 The Process – Iteration 2 STEP 1 Sample 45 Item Difficulties from Ensemble STEP 2 Compute Lexile Measures for Each Raw Score (1 to 44) STEP 3 Table Results Raw Score Lexile Measure 362L 514L 584L L Lexile Measure 354L 506L 575L L Sample 1 Sample 2

28 The Process – Iteration 1,000 STEP 1 Sample 45 Item Difficulties from Ensemble STEP 2 Compute Lexile Measures for Each Raw Score (1 to 44) STEP 3 Table Results Raw Score Lexile Measure 362L 514L 584L L Lexile Measure 354L 506L 575L L Sample 1 … Sample 1,000 Mean Lexile Measure 378L 509L 589L L Mean of 1,000

29 Closing No matter how it is sliced and diced, analyses of joint and conditional probability distributions yield no more than patterns of association. Nothing in the response data nor Rasch analyses of these data exposes the processes (features of the object of measurement) or mechanisms (features of the instrument) that are hypothesized to be conjointly causal on the measurement outcomes.

30 A. Jackson Stenner CEO, MetaMetrics Contact Info: