1. 1 How to Model and Test for the Mechanisms that make Measurement Systems Tick IMEKO Jena, Germany Wednesday, August 31, 2011 A.Jackson Stenner Chairman & CEO, MetaMetrics jstenner@lexile.com

2. 2 Reader Ability Temperature

3. 3 Four well researched constructs  Reader ability  Text Complexity  Task Difficulty  Comprehension

4. 4 Reading is a process in which information from the text and the knowledge possessed by the reader act together to produce meaning. Anderson, R.C., Hiebert, E.H., Scott, J.A., & Wilkinson, I.A.G. (1985) Becoming a nation of readers: The report of the Commission on Reading Urbana, IL: University of Illinois

5. 5 Reading is a process in which information from the text [Complexity] and the knowledge possessed by the reader [Ability]act together to produce meaning [Comprehension] as indexed by a specific task type [Difficulty].

6. 6 An Equation = Reader Ability Text Complexity Comprehension - Conceptual Statistical Raw Score = i e (RA – TC - TD) i 1 + e (RA – TC i - TD) RA = Reading Ability TC = Text Calibrations TD = Task Difficulty - Task Difficulty

7. 7  A causal model relating reader ability, text complexity, task difficulty and comprehension  Measures reader ability, text complexity, and task difficulty on a common scale—the Lexile scale  Allows educators to forecast the level of success a reader is likely to experience with a particular text when responding to a specific task requirement The Lexile Framework for Reading

8. 8 Eight Features of the Causal Model Relating Text Complexity, Reader Ability, Task Difficulty and Comprehension 1. The model is individual centered. The focus is on explaining variation within person over time. 2. In this framework the measurement mechanism is well specified and can be manipulated to produce predictable changes in measurement outcomes (e.g. percent correct). 3. Item parameters are supplied by substantive theory and, thus, person parameter estimates are generated without reference to or use of any data on other persons or populations. Therefore, effects of the examinee population have been completely eliminated from consideration in the estimation of person parameters for reader ability.

9. 9 4. The quantitivity hypothesis can be experimentally tested by evaluating the trade-off property for the individual case. A change in the person parameter can be off-set or traded-off for a compensating change in text complexity to hold comprehension constant. The trade-off is not just about the algebra. 5. When uncertainty in item difficulties is too large to ignore, individual item difficulties may be a poor choice to use as calibration parameters in causal models. As an alternative we recommend, when feasible, averaging over individual item difficulties to produce “ensemble” means. For example empirical text complexities can be excellent dependent variables for testing causal theories. Eight Features of the Causal Model cont’d.

10. 10 6. Causal Rasch models are individual centered and are explanatory at both within-subject and between-subject levels. The attribute on which I differ from myself a decade ago is the same attribute on which I differ from my brother today. 7. When data fit a Rasch model, differences between person measures are objective. When data fit a causal Rasch model absolute person measures (reader abilities) are objective (i.e. independent of instrument). 8. Causal Rasch models make possible the construction of generally objective growth trajectories. Each trajectory can be completely separated from the instruments used in its construction and from the performance of any other persons, whatsoever. Eight Features of the Causal Model cont’d.

11. 11 Let’s examine in detail each of the four constructs that make up the Lexile Framework for Reading  Text Complexity  Reader Ability  Task Difficulty  Comprehension

12. 12 May 2016 (12 th Grade) 1200 1000 1400 1600 Text Demands for College and Career May 2007 – April 2011 347 Encounters 138,695 Words 3,342 Items 983 Minutes Student 1528 7 th Grade Male Hispanic Paid Lunch Expected: 73.5% Observed: 71.7%

13. 13

14. 14 Mythology  Text Complexity Theoretical: 1300L Empirical: 1357L Adapted from Oasis Article courtesy of EBSCO Publishing The study and interpretation of myth and the body of myths of a particular culture. Myth is a complex cultural phenomenon that can be approached from a number of viewpoints. As generally understood, a myth is a story or narrative that is traditional in a certain culture, having been passed down from early times and regarded as true. It may be said to 1 symbolically the origin of the basic elements and assumptions of a culture. Mythic narratives frequently revolve around the doings of gods or heroes, and may relate, for example, how the world began, how humans and animals came into being, or how certain customs, gestures, or forms of human activities 2. Almost all cultures possess or at one time possessed and lived in terms of myths. 1 immerse belittle portray contradict 2 originated adorned handicapped entwined

15. 15 r = 0.952 r” = 0.960 R 2” = 0.921 RMSE” = 99.8L Figure 1: Plot of Theoretical Text Complexity versus Empirical Text Complexity for 475 articles “Mythology” Reliability =.996 SEM = 12L

16. 16 What could account for the 8% unexplained variance?  Missing Variables or Theory misspecification  Better Criterion Variable – task specificity hypothesis  Improved Proxies/Operationalizations  Expanded Error Model – Treat Item Type as Random  Rounding Error  Imperfections in Theory Implementation

17. 17

18. 18 Connect reading to postsecondary demands— www.lexile.com/toefl

19. 19 Examine growth overtime—

20. 20 Task Plane Different task types may vary in average difficulty and in unit size. Adjustments for added easiness or hardness and adjustments for unit size (Humphry, 2011) can be made to bring new task types into the Lexile Frame of Reference.

21. 21 Task Plane 0Added HardnessAdded Easiness Reference Value (Native Item) Unit Size

22. 22 How Temperature and Pressure Relate Under Constant Volume TemperatureVolumePressure 2000°K20 Liters20.0 atm 1000°K20 Liters10.0 atm 500°K20 Liters5.0 atm 250°K20 Liters2.5 atm 125°K20 Liters1.25 atm

23. 23 Comprehension Rates for Readers of the Same Ability with Texts of Different Complexity or How Reader Ability and Comprehension Rate Relate Under Varying Text Complexity Reader Ability Text Complexity Text Titles Comprehension Rates 1000L 500L 750L 1000L 1250L 1500L The Magic School Bus, Inside the Earth (Cole) The Martian Chronicles (Bradbury) The Reader’s Digest The Call of the Wild (London) On Equality Among Mankind (Rousseau) 96% 90% 75% 50% 25% Comprehension Rates for Fixed Reader Ability

24. 24 Comprehension Rates for Readers of Different Ability with Texts of the Same Complexity or How Reader Ability and Comprehension Rate Relate Under Constant Text Complexity Reader AbilityClassroom TextbookComprehension Rates 500L 750L 1000L 1250L 1500L 1000L 25% 50% 75% 90% 96% Comprehension Rates for Fixed Text Complexity

25. 25 Testing the Lexile Theory  How closely does Observed Comprehension (success rate) correspond to what the Lexile theory predicts?

26. 26

27. 27 Oasis: Usage Report by Reader Lexile

28. 28

29. 29 Oasis: Usage Report By Category of Article

30. 30 To causally explain a phenomenon [a measurement outcome] is to provide information about the factors [person processes and instrument mechanisms] on which it depends and to exhibit how it depends on those factors. This is exactly what the provision of counterfactual information…accomplishes: we see what factors some explanandum M [measurement outcome, raw score] depends on (and how it depends on those factors) when we have identified one or more variables such that changes in these (when produced by interventions) are associated with changes in M (Woodward, 2003, p.204).

31. 31 How Many Ways Can We Say X Causes Y? X “elicited a greater” YX “impacts” Y X “accounts for” YX “has been linked to” Y Y “is the result of” XX “didn’t diminish” Y Y “because of” XY “depends on” X X “has led to” YX “largely motivates” Y Y “stemmed from” XX “proved critical to” Y X “fosters” YX “changes” Y X “triggers” YX “affects” Y

32. 32 A. Jackson Stenner CEO, MetaMetrics jstenner@Lexile.com Contact Info: