New York State Professional Development Grant Taking Advantage of Capacity: Salvaging Evaluations and Providing Models of Effective Practice Presenters: Matt Giugno – SIG/SPDG Project Director Wilma Jozwiak – SIG Statewide Coordinator Laura Payne-Bourcy – SIG/SPDG Project Evaluator

Queens Bronx SL FE CL WAHA AS JL ST TS BD DC OD RC SU UL DU OU PW RO WS WE ES NA Manhattan Brooklyn West (including Staten Island) MOMO M1 M2 E1 ON LG E2CE OW OC CO OS Brooklyn East OMOM HE Mid-State Mid-West Hudson Valley East Long Island New York City 7/14/06 West SIG & SPDG Service Regions

One Type of New York

Another Type of New York

Our Purpose Today:  Talk about the realities that intrude on best plans, and what we did to address our realities in the SIG Grant  Talk about how we relied on existing capacity to salvage outcomes  Talk about how our SIG experience informed our decisions in developing our SPDG project

The NYS SIG Initiative goals were to:  Reduce achievement gap between special and general education students in high and low need schools.  Reduce or eliminated the disproportionality of language and ethnic minority students in classification and placement practices.

LEA Regional School Support Centers (RSSCs) Special Education Training and Resource Centers (SETRCs) Institutions of Higher Education (IHEs) Special Education Quality Assurance (SEQA) State Improvement Grant Higher Education Support Center (HESC) VESID: Resources, TA, Oversight New York State SIG Organizational Model

Two Areas of Intended Impact: NYS SIG resources and partnerships were designed to address needs of:  Inservice Teachers and  Pre-Service higher education faculty

Changing Landscape During the five years of SIG, evaluation design and methodologies had to respond to:  External shifts/expectations/needs (OSEP) and  Concurrent internal programmatic changes/shifts (VESID) Some of these shifts were anticipated and were worked into the original design, and some were not….

Challenges...  Practice:  Large geographic distances between targeted schools served by SIG Teams.  Roles:  Shifts in thinking about responsibilities and roles of SIG Teams and RSSC/SETRC partners.  Partnerships:  Difficulty engaging parent organizations.  Programming:  Introduction of new program components.

 Time needed to embed change:  Grants initially intended for two years were extended to three and four.  Logistics of grant awards:  Changes to NYC district administrative structures.  Reporting:  Institution of new achievement reporting mechanisms. Challenges...

 Alignment:  Degree of ‘match’ of project goals to State Performance Plan indicators.  Accountability/Rigor Part 1:  Development of Federal Performance Measures.  Accountability/Rigor Part 2:  ‘Collective call’ to utilize scientific and evidence based practice. Challenges...

Changing expectations require a response that is both programmatic and evaluative in nature.

The NY SIG Responsive Model The NY SIG Responsive Model Life in a changing landscape...

Post-SIG analysis: Where did we still struggle? New York State used the experience and challenges of five years of SIG to develop retrospective SIG program and evaluation design activities. Some of those challenges and experiences included:

–Limited opportunities for programmatic response to changing OSEP requirements due to stage of program implementation (year 4 of 5). –‘Newness’ of SPP and subsequent lack of data (particularly involving IEPs and student outcomes, transition etc). –Identifying impact on schools and students in NYC amidst changes in NYC educational structure. –Identifying the evidence base of said activities “after the fact”.

Responding to Changing Expectations: OSEP requirements for performance measures

New requirements for evidence  Identify and collect evidence of:  replication of scientific and evidence based practices and proportion of personnel using these practices,  sustainability of efforts, and  alignment with State Performance Plan.

Program Responses: Performance Measures SPDG Mind Map 1. Effective Practice: more on that later…. 2. Replication: professional development, matching, regional & state facilitation. 3. Sustainability: mentoring, symposia, clearinghouse. 4. SPP: more on that soon…….

Evaluation Responses: Performance Measures One solution was incorporation of data collection into program design and activities  co-design.  Exploration of Evidence Based Practices  Regional focus groups.  Analysis of worksheet products.

 Replication  Nomination forms: description of practice, evidence of effectiveness.  Validation protocols: practice exploration, collection of evidentiary data for student outcomes, educator practice, school outcomes.  Regional and state facilitator Documentation & Reporting criteria.  Implementation tracking system (to be developed).  Participant interviews (to be conducted).

More Evaluation Responses  Sustainability  Targeted site pre and post survey (to be developed).  Effective Practice site post survey (to be developed).

Program Responses: SPP All of the objectives and activities undertaken as part of SPDG will be carried out under the framework of the SPP. The relevant project phases (as per the developed program logic model and mind map) include:

 selection  target site selection based on lack of SPP attainment)  site matching  effective practice sites matched based on expertise in specific SPDG goal areas and SPP indicators for which the target sites have demonstrated need)

 implementation  professional development and ongoing technical assistance  individualized to target site needs  as per validated scientific and evidence based practice aligned to the SPP), and  documentation :  more to come…..

Evaluation Responses: SPP Documentation: Capturing outcomes via NYSED Quality Improvement Process (QIP) reporting using SPP as the core framework for analysis and reporting.

SPP Indicators  SPP indicators considered for initial data collection:  graduation and dropout,  achievement,  suspension and expulsion, and  Placement: Least Restrictive Environment  (SPP Indicators 1-6 of 20).

Not all of these indicators were markers in the original SIG Project design: Why these six SPP indicators?  Because NYS:  Is currently collecting baseline information for all districts for SPP 1-6,  Is able to make a determination as to which schools are not meeting standards,  Has identified with schools need further assistance.

Evaluation Procedures  SPP data analyzed annually for participating targeted sites.  Provisional outcome data plan:  Year One: baseline and treatment data for SPP 1-6.  Year Two: baseline SPP 1 – 20, treatment 1-6.  Year Three and beyond: baseline and treatment SPP 1 – 20.

Capturing Impact of SIG in New York City: A Retrospective We needed to better understand SIG work in NYC, including:  technical assistance conducted,  collaboration between UFTTC and targeted school  impact on systematic reform of schools,  parental involvement, and  student outcomes.

System Complexity number of schools involved + SIG work woven into broader UFTTC improvement efforts = how to tease out impact?

Evaluation Design Effort:  Review UFTTC SIG documentation,  Review NYBOE Quality Review Reports for select sites,  Interview UFTTC SIG Coordinator,  Interview key UFTTC Field Liaisons,  Interview key UFTTC site staff. Effect:  Interview key UFTTC site staff,  Interview select school staff at some sites, and  Review of available performance data.

Evaluation Framework  Implementation  Describe the work.  Explore involvement of school leaders.  Analyze teacher responses.  Consider salience and prioritization of approaches.  Describe continuous improvement and mid-course corrections.

 Analysis of Process and Planning  Communication: teachers and administrators.  Curriculum and instruction.  School policies and/or school functioning.  Barriers.  Analysis of Impacts  Student outcomes.  School outcomes.

Evolution of Effort: Identifying and Implementing Effective Practices New Directions from state and federal levels require:  Building educator capacity (skills and knowledge)  Needed to implement scientifically or evidence-based practices for children with disabilities.

Some Advantages:  Evaluation redesign activities provided reflection for schools resulting in increased capacity  Needs for redesign in SIG emphasized the need for “contingency planning” at the outset of the SPDG

The Segue to SPDG: How we responded to our lessons in developing our SPDG Supporting Successful Strategies to Achieve Improved Results: The S 3 TAIR Project

Taking Advantage of Lessons Learned Lesson: Difficulty in identifying data supporting impact  Response: Build identification of effective practice requirements into grant applications and site selection processes. Lesson: Availability (or lack) of data resulted in changes to cohort size  Response: Embed strategies for collection of data directly from schools and/or regions, and strengthen district reporting requirements.

Lesson: Demands for greater accountability and research to practice implementation  Response: Build capacity identification and utilization into the project plan, taking advantage of existing mandates for data collection and analysis  NCLB  SPP  NYS Contract for Excellence

Lesson: Experience from our Special Education Quality Assurance field work. High need, low performing districts consistently lacked effective practices in one or more of the following areas:  Reading instruction/literacy acquisition,  Positive academic and behavior interventions and supports, and  Implementation of effective special education programs and services  Response: These areas will be targeted for intervention in the NYS SPDG S 3 TAIR Project

Lesson: Even with skilled coaching, districts don’t always identify the most effective strategies  Response: support will be provided in:  Analyzing data and identifying key issue  Identified school improvement activities will be channeled to evidence based interventions  Implementation and evaluation of evidence based practice, including effective implementation practices (National Implementation Research Network)

 Lesson: Our high risk districts need models they can identify with to move from research to practice. We knew that:  H igh need districts want an implementation model in the state and in communities with similar characteristics.  Good examples exist of districts whose school improvement efforts have resulted in sustainable outcomes.  Districts doing good things are often too busy making it happen to talk much about it  Response: Design of NYS SPDG S 3 TAIR Project

Elements of S 3 TAIR  Identify and provide small grants to districts whose evidence based effective practices have resulted in good outcomes for students with disabilities:  District funding will support collaboration with S 3 TAIR Regional Field Facilitators to document the practices  District funding will also support mentor relationships with targeted high need districts  Quality Indicator tools for implementation are currently under development by the VESID Special Education Training and Resource Center network

 Fund school improvement efforts for targeted high need districts  Support field based regional staff (Regional Field Facilitators) who will:  Support relationship development with Effective Practice Mentor Schools;  Collaborate with other VESID funded TA networks to provide professional development as appropriate; and  Document the school improvement experience of the funded districts for the Clearinghouse.

 Continue our partnership with preservice preparation programs through the Higher Education Support Center/Task Force for Quality Inclusive Schooling:  Regional Task Force groups will identify promising practices regionally and nominate them for statewide validation.  IHEs will focus efforts on increasing preservice and inservice capacity in the identified areas.  The HESC will continue to support Partnership Grants for IHE/LEA collaboration on school improvement.