1. PARA Project Overview, Results, Next Steps Martha Thurlow and Deborah Dillon
National Accessible Reading Assessment Projects

2. Overview: PARA Partners
Partnership for Accessible Reading Assessment (PARA) consists of:
University of Minnesota - NCEO (National Center on Educational Outcomes) and Department of Curriculum and Instruction
CRESST (Center for Research on Evaluation, Standards, and Student Testing, University of California, Los Angeles) and University of California, Davis
Westat
National Accessible Reading Assessment Projects

3. PARA Research Program Assumptions
Assumption #1: We don’t know everything about what goes into accessible reading assessment yet.
Assumption #2: Preliminary research must inform design of “testlets.”
Assumption #3: Both preliminary and experimental research will inform our knowledge of Principles and Guidelines (Goal 3) for future assessments.
National Accessible Reading Assessment Projects

4. Three Major Research Question Areas
What characteristics of students require more accessible assessment of reading?
What characteristics of current assessment practices hinder accessibility?
What characteristics would accessible assessment have?
National Accessible Reading Assessment Projects

5. PARA Activities and Findings
Disability Reports
Differential Item Functioning Analysis
Differential Distractor Functioning Analysis
Test Characteristics Study
Student Characteristics Pilot Study
National Accessible Reading Assessment Projects

6. Disability Reports*
Addressing disabilities that seem to affect reading some of these are the focus of our projects and some are not
Visual Impairments
Deaf or Hard of Hearing
Autism
Learning Disabilities
Mental Retardation
Speech or Language Impairments
Emotional or Behavioral Disabilities
* GAC members were instrumental in reviewing/writing these reports.
National Accessible Reading Assessment Projects

7. Differential Item Functioning (DIF) Analysis
Used to examine differences between students with and without disabilities – to see whether items functioned differently.
Found that many items exhibited DIF, especially for grade 9 students with disabilities, most often in the second half of the assessment.
Several issues with the data limited findings – NRT, no access to category of disability or accommodations information; concern about high omission rate.
National Accessible Reading Assessment Projects

8. Differential Distractor Functioning (DDF) Analysis
Used to examine patterns of incorrect answers to distractors – to see whether items functioned differently.
Found that many items exhibited DDF for students with disabilities in grade 9, especially in the second half of the assessment.
Several issues with the data also may limit these findings – NRT, no access to category of disability, accommodations information; concern about high omission rate.
National Accessible Reading Assessment Projects

9. Test Characteristics Study
Examined test specifications for purposes and constructs, number of items assigned to constructs, and types of items typically found.
Found that not all standards in states are assessed; yet, more than one-third of states assessed some form of foundational skills; further, in most states, comprehension and analysis/interpretation are considered an extension of foundational skills.
Not all states posted their test specs/blueprints on Websites; these states are being pursued now to complete the analysis.
National Accessible Reading Assessment Projects

10. Student Characteristics Pilot Study
Identifying students whose teachers believe have reading skills that are not accurately measured by state reading assessments.
Found that teachers are able to identify students, but did not identify many with comprehension limitations that obscured other reading skills and those who have strengths outside of what most reading tests cover.
Issues with follow-up measures made it difficult to verify teachers’ beliefs about skills.
National Accessible Reading Assessment Projects

11. PARA’s Next Steps Chunking Study Student Characteristics Study
Motivation Study
National Accessible Reading Assessment Projects

12. Chunking Study
National Accessible Reading Assessment Projects

13. Purpose of Study
To examine the effects of chunking reading passages on the performance of students with disabilities, and to compare this effect to the effect on the performance of students without disabilities.
National Accessible Reading Assessment Projects

14. Definition
Chunking refers to the concept of building breaks directly into the test booklet by separating the passage into more manageable “chunks,” and adding relevant test questions directly beneath the chunk.
National Accessible Reading Assessment Projects

15. Participants and Passages
Participants: Grade 8 students with and without disabilities
Passages: Three passages with multiple choice items. All were released state assessment items, for which permission to use was obtained
National Accessible Reading Assessment Projects

16. Design
Students with and without disabilities will be randomly assigned to either Version A (standard version) or Version B (chunked version). All students will be given background questions, feedback question related to fatigue and mood, and a student motivation scale.
National Accessible Reading Assessment Projects

17. Student Characteristics Study
National Accessible Reading Assessment Projects

18. Purpose of Study
To identify students whose teachers believe have reading skills that are not accurately measured by state reading assessments, and follow-up with students.
To check on the prevalence of “less accurately measured students” (LAMS) with various characteristics.
National Accessible Reading Assessment Projects

19. Types of Students Fluency limitations obscure comprehension skills
Comprehension limitations obscure other reading skills
Strengths exist outside of what most reading tests cover
Responds poorly to standardized testing circumstances or materials
National Accessible Reading Assessment Projects

20. Research Questions How well can teachers identify LAMS?
How well can teachers document their rationale?
How well can a brief examination confirm or disconfirm teacher judgments?
National Accessible Reading Assessment Projects

21. Procedures Teachers complete nomination questionnaire
Researchers interview teachers
Researchers interview students
Researchers examine students
Students are 4th and 8th grade students with and without disabilities
National Accessible Reading Assessment Projects

22. Examination Protocols
After interview, student examined with:
CBM, listen to text with immediate retelling
CBM, read text with immediate retelling
CBM, choice to read or listen to text, and do standard multiple choice questions
CBM, think aloud
National Accessible Reading Assessment Projects

23. Prevalence Study
Teachers complete survey about all of their LAMS, indicating which of the types they have in their classroom and their characteristics
National Accessible Reading Assessment Projects

24. Designing Large Scale Reading Assessments that Increase Students’ Motivation during the Assessment
National Accessible Reading Assessment Projects

25. Purpose of Study
To examine whether improving the motivational characteristics of a large-scale reading assessment increases its accessibility for students with disabilities, and in so doing provides a more valid assessment of these students’ reading proficiency due to their increased engagement.
National Accessible Reading Assessment Projects

26. Research Questions
1. Does the option of exerting choice in the selection of passages on a large-scale reading assessment improve the accessibility of the assessment and lead to increased comprehension due to improved motivation and engagement for students with disabilities and general education students?
2. When students with disabilities and general education students are given the option to choose literary and informational passages that are included on large-scale reading assessments, do they perform better on one text type or the other?
National Accessible Reading Assessment Projects

27. Research Questions – cont.
3. Are students who are identified as being more motivated to read on general motivation to read measures (e.g., scales and questionnaires) more proficient in reading when given a choice of reading passages on large-scale assessments?
4. Are students who are identified as being more situationally motivated to read, as reported on questions following each passage, more proficient in reading those passages?
National Accessible Reading Assessment Projects

28. Participants & Setting
Participants: Students will be selected from grades 4 and 8. Students with disabilities as well as general education students will be in the experimental condition; comparable students will be in the control condition.
Setting: Computer lab/classroom, untimed testing situation, appropriate accommodations for students with various disabilities.
National Accessible Reading Assessment Projects

29. Design
In the counterbalanced design of the study we are looking to see if there is increased performance between students (both general education and those with disabilities) who are given choice (C) in selecting the passages they read and those who are not (NC). A stratified random assignment of students will be necessary to ensure that students representing particular disabilities are randomly assigned to the experimental and control condition. Students will take the version of the test they are assigned.
National Accessible Reading Assessment Projects

30. Design—cont.
Students in the experimental conditions will be allowed to “design your own assessment.”
The assessments will include 4 total passages (two literacy-fiction and two informational-expository—based on the NAEP 2009 Reading Framework).
Separate pools of passages will be provided for 4th and 8th grades to select from; each passage will be followed by 5-6 multiple choice items.
National Accessible Reading Assessment Projects

31. “Choice Condition”
Students in the “choice” condition will be presented with short (1-2 sentence) descriptions/summaries for 6 literary-fiction passages. Students choose 2 of the 6 passages to include in their individually designed assessment.
Students will be presented with short (1-2 sentence) descriptions or summaries for 6 informational-expository passages. Students choose 2 of these 6 passages to include in their individually designed assessment. Students will be given time to read the descriptions prior to the initiation of the test.
After selecting the 4 total passages, students will be asked to briefly state why they chose these passages.
National Accessible Reading Assessment Projects

32. “Control Condition”
“Control” condition students will read 2 literacy-fiction and 2 informational-expository passages from the same high interest pool of passages, but they will not be allowed to choose the particular passages that will be included in their assessment.
“Control” condition students will be randomly assigned passages (Note. These passages will have been ranked as low interest by 4th and 8th grade students during a pre-calibration study).
National Accessible Reading Assessment Projects

33. “Control Condition”
“Control condition” students will follow the same procedures as the experimental conditions students: Students will be presented with short (1-2 sentence) descriptions or summaries for 6 literary-fiction passages and 6 informational-expository passages. Students will be asked to rate these passages according to level of interest.
National Accessible Reading Assessment Projects

34. Passage Selection Criteria
Passages used in this assessment will be selected from existing literature and textbooks based on broad appeal to students at the 4th and 8th grades, respectively. There will be separate pools of passages for 4th and 8th grades.
Children’s literature scholars and the research literature will be consulted to select high interest passages.
National Accessible Reading Assessment Projects

35. Passage Selection Criteria—cont.
Reading passages will be chosen to represent two text types and one specific form of text within the text types: informational—exposition; & literary—fiction.
18-24 high interest passages at the 4th grade level and the same number at the 8th grade level will be secured; half of the passages will be literary and half will be informational texts.
Short (1-2 sentence summaries will be written for each passage).
National Accessible Reading Assessment Projects

36. Passage Selection Criteria—cont.
To determine the difficulty of the passages for 4th and 8th grade readers, each passage will be rated as equivalent on factors such as difficulty and accessibility (using multiple reading ability measures) and structure (e.g., text grammar or top-level structure).
Permissions will be procured for passages; more passages will be secured than needed on the actual test and more items for each passage will be written (e.g., due to the calibration process and removal of weak items).
National Accessible Reading Assessment Projects

37. Skills Assessed: The Use of Cognitive Targets
Item writers will use the 2009 NAEP cognitive targets to create 10 items to follow each of the passages for each grade level.
The tasks or questions students answer after reading passages will match the content and key ideas of individual passages rather than being generic questions.
National Accessible Reading Assessment Projects

38. Attending to Issues of Motivation
Texts and items will be ordered from easy to difficult to foster motivation via improved self efficacy.
Situated motivation questions will be woven into the test booklets for the choice condition only and will be placed after the items that go with each question; these questions will indicate students’ perceptions of the texts they read (e.g., difficulty; interest) and their sense of self-efficacy in reading and completing the items following the passage (the task).
A pre-assessment general motivation survey will be given to all students to provide information on student feelings about “self as reader” (e.g., Motivation for Reading Questionnaire-MRQ).
National Accessible Reading Assessment Projects

39. Demographic data on all students will be collected, including whether they are students with a disability and/or students who are English language learners.
Observational data and verbal protocols will be obtained from students to determine their perspectives on assessment instruments, tasks, and contexts.
National Accessible Reading Assessment Projects