1. Minneapolis, MN June 27, 2012
Closing the Opportunity Gap for Students with Disabilities: Demonstrating and Testing a Model for Aligning Instruction, Standards, and Assessments
Presented by:
John Smithson, University of Wisconsin-Madison
Deborah Matthews, Kansas Department of Education
Chris Woolard, Ohio Department of Education
Wendy Soica, Ohio Department of Education
Claire Greer, North Carolina Dept. of Public Instruction
Moderator: Rolf Blank, CCSSO
Discussant: Anne Chartrand, Southeast Regional Resource Center

2. SEC-SPED Research questions
What is the fidelity of the enacted curriculum (i.e., instruction in classrooms) for students with disabilities in relation to current state standards and assessments?
What unique implications exist for students participating in the Alternate Assessment – Modified Academic Achievement Standards (AA-MAAS) or the Alternate Assessment–Alternate Academic Achievement Standards (AA-AAAS)?
How does the enacted curriculum for students with disabilities differ by type of instructional setting/class type, in comparison to general education students, and by school and teacher characteristics?
To what extent are student achievement scores in Mathematics and English Language Arts (ELA) for students with disabilities explained by instructional alignment to standards? What are the results with different assessment measures?
What are the implications of the alignment and achievement analysis results for assessments based on the Common Core State Standards and/or developed under RTTT state consortia?
Descriptive
Descriptive
Descriptive
Analytic
This research project is focused on the five research questions noted here.
Some of these questions I refer to as being descriptive in nature… they are concerned with knowing what opportunities students with disabilities have to learn standards-based academic content.
The other questions I describe as analytic… they have to do with the impact or role of opportunity to learn on student achievement, and the implications of alternate ‘targets’ or program goals on OTL for students with disabilities. My comments will focus more generally on how we operationalize/measure OTL, and how that information can be used to improve student achievement. The other presenters in this session will provide more of the descriptive information to fill out this picture of what kinds of opportunities to learn that students with disabilities receive, and how that may be impacted by one or another program or strategy used by schools and districts to meet the needs of students with disabilities.
Analytic
Opportunity-To-Learn (OTL)

3. SEC-SPED Research questions
Research questions re-cast relative to OTL:
1. Does opportunity to learn standards-based content (OTL) differ among students with disabilities and the general education population? Is there an OTL gap?
2. What are the implications of OTL for students that participate in an alternate or modified standards-based assessment program?
3. How does OTL differ by instructional setting and programmatic resources?
4. Does OTL contribute to explanations of variation in student achievement (i.e., does OTL help to predict achievement scores)?
5. What are the implications for OTL as standards-based assessments become increasingly aligned to Common Core State Standards?
Each of the research questions is in one way or another concerned with this notion of opportunity to learn, and in fact each research question can be re-stated in ways that emphasize this relationship to OTL.
Of particular interest to me is research question #4, which asks about the link between our measure of OTL and student performance on high stakes assessments.

4. Descriptions of Academic Content (Instruction/Assessments/Standards)
Measuring OTL: Step 1 Describe instruction and standards using the SEC taxonomy
Perform Procedures
Demonstrate
Understanding
Conjecture/Analyze
Integrate/Synthesize
Recall
(What students should know)
Number Sense
EMPHASIS
Operations
Measurement
Knowledge
The SEC uses a two-dimensional taxonomy, based on the work of Andrew C. Porter, now the Dean for the Graduate School of Education at the University of Pennsylvania. The methodology allows us to use a systematic and neutral framework with which to describe any curriculum related document or description of classroom content coverages. This use of a systematic language allows us to collect multiple descriptions of standards, assessments and practice and make detailed comparisons between various descriptions which can ultimately be reduced to a single quantitative measure that describes the extent of agreement or alignment between any two content descriptions.
Algebra
Geometry
Skills
(…and be able to do)
Descriptions of Academic Content (Instruction/Assessments/Standards)

5. Content Map Tile Chart CCSS Gr. 8 Document Number Sense Operations
All Content Areas
Number Sense
Operations
Measurement
Consumer Applications
Basic Algebra
Advanced Algebra
Geometric Concepts
Advanced Geometry
Data Displays
Statistics
Probability
Analysis
Trigonometry
Special Topics
Functions
Instructional Technology
These content descriptions can then be displayed graphically in order to provide the reader a concise, yet comprehensive description of the curriculum embedded in one or another document (e.g. standards, assessments, curriculum materials, etc.) The graphic displays on this page show two ways in which these descriptions can be presented. The example shown here shows the Common Core Standards for 8th grade math, using both a ‘content map’ and a ‘tile chart’. Both graphics shown here report the same data, and simply use a different type of graphic to present that information. The SEC website allows users to select whichever display they prefer to use.
Memorize
Procedures
Demonstrate
Conjecture
Non-routine

6. Comparing Descriptions of Instructional Content
Descriptions of classroom content coverage can also be displayed using the same types of charts. The content maps used here show the content coverage for two groups of 5th grade teachers. The map on the left reports results from 4 special education teachers regarding the content coverage they report for a specific group of students they work with. The map on the right reports results from 18 5th grade teachers. Both groups of teachers represented here are from the same state. The numbers reported in the upper left and upper right indicate summary measures of ‘alignment’. The upper left reports ‘fine grain’ alignment… that is how well do the two descriptions align at the detailed level of topic. The number reported in the upper right reports on ‘coarse grain’ alignment… that is how well do the two descriptions align with one another at the level of content area… the 16 content areas displayed in these charts. (Note it is possible to drill-down within any of these content areas to get a view or description of content at the detailed topic level within that content area. These charts, for mathematics in one state, show a very high degree of alignment between both groups of teachers. They all teacher pretty much the same content, with some slight shifts in emphasis. This speaks to one of our descriptive research questions (3), regarding how similar instruction is between special education and general education teachers, though this particular example is only looking at one small piece (5th grade instruction in one state) of the larger picture.

7. Descriptions of Opportunity To Learn Standards-based Academic Content
(Alignment to Standards)
When we talk about OTL from an analytic perspective we are primarily interested in the alignment of instruction to the standards… that is, students’ opportunity to learn standards-based content. The charts displayed here show results from the 4 special ed. teachers reported on the previous slide compared to the common core standards for grade 5.

8. Descriptions of Opportunity To Learn Standards-based Academic Content
(Alignment to Standards)
The charts displayed here show results from the 18 general ed. teachers reported earlier compared to the common core standards for grade 5. So here the alignment indices reported at the top indicate the degree of alignment to the relevant common core standards is very close for both groups, and indicate relatively good alignment to the common core standards for both groups.

9. Alignment Comparisons By Position & Subject
Language Arts & Reading
Mathematics * * * * * *
These bar graphs focus on the results for alignment of instruction to one or another ‘target’ for both groups of teachers in mathematics and language arts. These results are across the full sample of teachers from all 3 states participating in the study. As can be seen by these charts, alignment between general ed. and special ed. teachers is fairly similar in mathematics, and quite different for language arts and reading.
* Statistically significant at p < 0.05

10. “Students perform better on assessments that measure content they have had the opportunity to learn.”
Theory Or
Truism?
The results beg the question though, do these measures of opportunity to learn have any connection with achievement. We ask this question though not so much to provide evidence to support the theory that OTL does impact achievement, but rather to see if the measures meet our expectation that OTL should impact achievement. If it does, we do not take it so much as evidence for the importance of OTL, but rather as a form of validation for our instruments. However, answering the question of the relationship of OTL to achievement is easier stated than done, because it requires having data on student achievement that can be linked to specific teachers and specific classrooms. Such data can be hard to come by. While most states are beginning to move in that direction, many are still unable to make this connection. For the 3 states participating in this study, only one has such data readily available, one cannot make the connection between teacher and student, and the third can only by linking otherwise disparate data sets. Thus as of this presentation I only have the necessary data to investigate this question for one of the three states in the study. But those results are nonetheless informative, and worth consideration.
Does OTL as measured by SEC assist in explaining (predicting) variations in student achievement?

11. Explaining Student Achievement & Learning Gains
The Sample:
2129 Students across
57 math Courses
taught by 47 teachers
(some teacher/courses may include multiple classes)
The Assessments:
Regular Assessments administered to students in grades 4-9
2 NCEXTEND1 Assessments
56 NCEXTEND2 Assessments
Known predictors of student achievement (Control Variables):
Prior Achievement
Economic Disadvantage
Disability Status
Here is the sample of data I had available to draw on for the analyses I am going to report for mathematics.
Using a simple multivariate regression model, I control for most of the known predictors of student achievement in order to examine the additional (if any) role that OTL plays when controlling for these known factors.
New Predictor (Independent Variable):
OTL measured as the fine grain Alignment Index (AI)

12. Model #3 .535 Predicting: mathScore_2011 for Regular Assessment
Based on Prior Achievement (math_score2010)
Economically Disadvantaged Status (eds)
Student With Disability (swd)
State Content Standards (stateTrgt)
Adj. R2 :
.535
For purposes of time and brevity, I am skipping results for the first two models, which simply look at prior achievement alone, and then prior achievement in combination with poverty and disability before including alignment into the target. As model 3 reported here shows, each factor plays a significant role in predicting student achievment, with prior achievement having the strongest impact. Review of the standardized coefficients column gives a sense of the relative impact each factor has on achievement for this statistical model, with this group of teachers (n=47).

13. Model #4 .551 Predicting: mathScore_2011 for Regular Assessment
Based on Prior Achievement (math_score2010)
Economically Disadvantaged Status (eds)
Student With Disability (swd)
State Content Standards (stateTrgt)
District LRE Category A
District LRE Category B
District LRE Category C
Adj. R2 :
.551
Further extending the model, I used district reported measures on ‘Least Restrictive Environment’ (LRE) to investigate the role (if any) that how access to the curriculum is provided to students. Interestingly, all 3 categories play a significant role in the model, though of the 3, category A (SWD spend at least 80% of their time in the general ed. classroom) has the strongest impact. Category B (students spend 40% or less of their time in general ed. classrooms) has a negative impact on student achievement. One must be careful in interpreting these results though, since it is not clear if the lower performance is due to limited access to the general ed. classroom or because students with more severe types of disabilities tend to be placed in this category, and it is the disability that is affecting performance.

14. Caveats Prior Research Class-level and HLM modeling
Results are less compelling for ELAR
Why?
While the results reported for math provide a classic example of the types of results we would hope to see for OTL, and confirms findings from prior work, the results for language arts are not as compelling with respect to OTL and student achievement. This leads to questions about ‘why’ OTL measures in language arts do not show a significant relationship to achievement.

15. WHY? Instrumentation Observer/Reporter The Nature of ELAR Content
Based on Reading Scores Only

16. NC Mathematics Scatterplot of Alignment to Common Core & Student Performance
Above Avg. Performance
Above Avg. Alignment
Above Avg. Performance
Below Avg. Alignment
Below Avg. Performance
Above Avg. Alignment
What potentially useful information is provided to a teacher by knowing which of these four quadrants they fall into based upon their reporting of instructional content coverage?
Below Avg. Performance
Below Avg. Alignment

17. ELAR Alignment to Common Core By Reading Score Scatterplot

18. END