2011 OSEP Leadership Mega Conference Collaboration to Achieve Success from Cradle to Career 2.0 State Monitoring Under IDEA A Snapshot of Past Practices and a Framework for the Future Julie Bollmer and Marsha Brauen, Westat Presentation EB 1

Study objectives and historical context Overview of framework for state monitoring systems Methodology for site visit data collection Discussion of each framework component Summary Questions/discussion 2 Overview of Presentation

Study O bjectives and Historical Context 3

Provide a description of the nature and scope of states’ monitoring systems Describe states’ monitoring systems at two points in time Create a framework to describe state monitoring systems 4 Study Objectives

State monitoring systems in and were the focus of the study. Systems were influenced by: – Federal legislation – OSEP’s monitoring systems – SPP and APR implementation. 5 Historical Context of the Study

F ramework for State Monitoring Systems 6

Development of Framework7 No requirement that states use a particular monitoring approach Framework used to help describe the variation in state monitoring systems Framework incorporated important activities of state monitoring systems Used various sources to create the framework Multiple reviews by Advisory Panel

8 Framework for State Monitoring

Methodology for Site Visit Data Collection 9

Random sample of 20 states  Two rounds of data collection First round  Second round   Conducted semi-structured interviews with state staff, local staff, stakeholders  Reviewed states’ documentation on monitoring activities 10 Site Visit Data Collection

Single or Multiple Processes Number of Monitoring Processes and States with Multiple Monitoring Processes Number Part BPart C (N=20) (N=20) (N=20) (N=20) Monitoring processes across states States with more than one process

Discussion of Each Framework Component 12

Problem Identification Problem Identification: Comparing performance (e.g., on a specific indicator) to an expectation (e.g., a target established for that indicator) and detecting possible deficiencies. – Indicator and Target Setting – Indicator Data Collection and Analysis – Problem Detection 13

Problem Identification14

Examples for Problem Identification: – Stakeholder committee includes parents of children with disabilities or representatives from advocacy groups. – Target setting is accomplished through a systematic process. – Written documentation exists describing the data collection methodologies, including site visit. – Procedures are in place to monitor the quality of the data collected. – Findings reflect performance in relation to specific targets. – State-level problem identification reports are disseminated to stakeholders. 15

Problem Identification Percentage of Part B and Part C Monitoring Processes That Had Problem Identification 16

17 How indicators selected Part BPart C (N=34) (N=32) (N=24) (N=28) By state, with stakeholder input21854 By state, without stakeholder input By state, based on SPP/APR†16†19 Problem Identification Number of Monitoring Processes That Used Various Approaches to Select Indicators † Not applicable. Because the 2004 amendments to IDEA, which require states to submit SPPs/APRs, were not enacted until December 2004, this approach was not used by states when selecting their indicators for

Problem Investigation Problem Investigation: Exploring why an identified problem exists and whether it is systemic or localized/isolated 18

Problem Investigation Examples for Problem Investigation: – Materials used to seek input from stakeholders are tailored to improve understanding and facilitate their contribution to the problem investigation process. – Individuals conducting problem investigations have appropriate training. – Procedures are in place to minimize interruptions/ disruptions to school, program, or district routines when collecting data. – Data are verified for accuracy. – Findings focus directly on the identified problems under investigation. 19

Problem Investigation Percentage of Part B and Part C Monitoring Processes That Had Problem Investigation 20

21 Approach Part BPart C (N=16) (N=17) (N=8) (N=8) Same approach for all problems10 75 Tailored, based on nature of problem6713 Problem Investigation Number of Monitoring Processes in Which Various Problem Investigation Approaches Were Used

Corrective Action and Enforcement Corrective Action and Enforcement: Correcting noncompliance by making immediate changes to documentation, procedures, or practices 22

Corrective Action and Enforcement Examples for Corrective Action and Enforcement: – CAPs specify all problem areas. – Steps/components of the CAPs are clearly delineated for each identified problem. – Timelines are included for each step/component of the CAPs. – CAPs are disseminated to local stakeholders beyond the LEA/EIS program administrators. – Opportunities are provided for LEA/EIS program personnel to discuss CAPs with SEA/lead agency staff. – The SEA/lead agency has written general enforcement procedures. 23

Corrective Action and Enforcement Percentage of Part B and Part C Monitoring Processes That Had Corrective Action and Enforcement 24

25 Follow-up approach Part BPart C (N=32) (N=32) (N=21) (N=23) General communication10941 Submit evidence of implementation51119 Progress reports Site visits, meetings, data reviews None (did not follow up)8431 Corrective Action and Enforcement Number of Monitoring Processes in Which Various Follow-up Approaches Were Used to Ensure Corrective Action Plan Implementation

Improvement Planning and Implementation Improvement Planning and Implementation: Developing and implementing strategies for improving systemic performance or reducing the potential for noncompliance – Improvement Planning – Improvement Plan Implementation 26

Improvement Planning and Implementation27

Examples for Improvement Planning and Implementation: – Input is sought from parents of children with disabilities or representatives from advocacy groups. – Written improvement plans address each problem area. – Plans describe the ways that implementation will be monitored by the SEA/lead agency. – Documentation confirms that plans were followed or changes in plans justified. – Meetings are held with SEA/lead agency to discuss local implementation. 28

Improvement Planning and Implementation Percentage of Part B and Part C Monitoring Processes That Had Improvement Planning and Implementation 29

30 Follow-up approach Part BPart C (N=23) (N=21) (N=11) (N=12) General communication6233 Submit evidence of implementation1302 Progress reports51467 Site visits, meetings, data reviews81066 None (did not follow up)8332 Improvement Planning and Implementation Number of Monitoring Processes in Which Various Follow-up Approaches Were Used to Ensure Improvement Plan Implementation

Reassessment Reassessment: Checking to see whether a corrective action plan or an improvement plan has been effective 31

Reassessment Examples for Reassessment: – Reports describe the data used to reassess performance/compliance on specified indicators and targets. – Opportunities are provided for LEA/EIS program personnel to discuss reassessment reports with SEA/lead agency staff. – Findings from reassessments are disseminated to state-level stakeholders. 32

Reassessment Percentage of Part B and Part C Monitoring Processes That Had Reassessment 33

34 Reassessment Number of Monitoring Processes in Which Reassessments Were Conducted by LEA/EIS Program Staff and/or State Agency Staff Who conducted reassessments Part BPart C (N=11) (N=8) (N=12) (N=9) LEA/EIS program staff only3231 State agency staff only6657 Both LEA/EIS program staff and state agency staff 2041

Summary 35

Framework has not been evaluated to look at whether its adoption will improve compliance with IDEA and outcomes for students with disabilities; limits the conclusions that could be drawn from the study (e.g., did not look at change over time or trends). The framework is not the only one that could be developed to represent state monitoring systems. Retrospective information was collected 36 Limitations of the Study

State monitoring systems were not static; in both years, a percentage of them were in transition Many factors influenced states’ monitoring systems Substantial variability in the nature and design of states’ monitoring systems In both years, few Part B or C monitoring processes included all five framework components 37 Key Findings

In both years, Problem Identification and Corrective Action and Enforcement were the most common framework components included in state monitoring processes States varied in the way they carried out each component and the degree to which elements were present or absent 38 Key Findings

Reports from this study are available on the IES website: The database is restricted-use and will need to be obtained from IES Marsha Brauen Julie Bollmer Rob Ochsendorf 39 Want More Information?