Building Bridges: Embedding outcome evaluation in national and state TA delivery Ella Taylor Diane Haynes John Killoran Sarah Beaird August 1, 2006

Session goals Situate classroom based technical assistance within the broader framework of teacher education Provide a rationale for the development of outcomes and performance indicators for children who are deaf- blind and their service providers Share how classroom based TA can use the OPIs, in conjunction with the IEP, to maximize student outcomes Demonstrate a relational database that facilitates the management and utility of the process Address participant questions

It’s all about the outcomes! Similarities TA providers & education professionals are being required to demonstrate the impact of our efforts TA providers & education professionals must do this within a standards based environment that demands learning for all students Differences Education professionals usually operate within only one venue (i.e., one state, one school district) TA providers operate across multiple venues Issue for TA providers is the use of a common language across the multiple venues

ESTABLISH LEARNING OUTCOMES ASSESSMENT PLAN INSTRUCTIONAL PLAN SHARE LEARNING OUTCOMES IMPLEMENT & ADAPT INSTRUCTION MONITOR STUDENT PROGRESS ASSESS LEARNING PROGRESS CONTEXT Organize, manage & maintain learning environment Variables at core of teacher work in a standards-based classroom (Schalock, Schalock & Girod, 2006).

Measuring impact across and within states Outcome evaluation system comprised of 1.Targeted outcome statements for each group of stakeholders (i.e., children, families, service providers, and systems). 2.Performance indicators associated with each outcome that help quantify how the expected result or change will be measured. 3.A TA process that embeds the targeted outcomes and performance indicators at the beginning of the TA cycle. 4.A computerized relational database system that ties the planning, delivery and evaluation of TA together.

Measuring impact within the classroom Extend outcome evaluation system to include –IFSP outcomes –IEP benchmarks –State academic standards –Alternate assessment components –Transition targets Clearly align and connect these components within the outcome evaluation system to facilitate their utility for the education professional

NTAC’s OPI System Provides a consistent coding system used in the –development and implementation of technical assistance –for ongoing collection, analysis & reporting of data. The system was developed across four domains: –Service Providers –Families –Systems –Children

How to demonstrate impact across trainings, individuals, students? Outcome –A statement of a measurable condition or an expected result or change. (i.e. increase, improvement, progress toward). Performance Indicator –A statement that helps quantify the outcome and indicates whether the outcome has been achieved. Often, multiple indicators may provide better evidence of the achievement of an outcome.

A Little History of the NTAC OPI System…

Federal emphasis on collecting and reporting outcome results has been challenging for many TA & D projects … including NTAC! The need for a systematic process of collecting and reporting outcome information (effect data) was identified. NTAC responded through a comprehensive four year process of –identifying measurable outcomes for service providers, children/youth, families, and systems; –developing instruments for collecting the data; –field-testing and redesigning the instruments multiple times; and, –analyzing the outcome data. Multiple stakeholders were involved in this endeavor including; –NTAC staff, –NTAC advisory board members, –State deaf-blind project personnel, –Families and consumers, and –Higher education faculty and researchers in the field.

NTAC’s OPIs Comprehensive outcomes for… –children (16) –families (12) –service providers (22) –systems (8) Multiple performance indicators for each outcome Embedded in all aspects of planning, delivery and evaluation Aligns needs assessment, delivery of TA and measurement of impact On web ( FileMaker Pro templates

Putting it all together and beginning a system… The NTAC Outcomes & Performance Indicator (OPI) System… A process created to assist with… –planning –collecting –aggregating, and –reporting effort, effect and outcome data

How NTAC uses the OPIs… Needs assessment and planning –Identify the stakeholder group service provider child family, and/or systems –Identify the outcome(s) to be targeted –Identify the performance indicators that will help determine attainment of targeted outcome(s)

How NTAC uses the OPIs… Delivery of service –Implement technical assistance that is targeted to the outcomes selected through… needs assessment, and planning process

Evaluation –Tailor assessment/evaluation measures to… the intensity of the TA targeted outcomes, and performance indicators How NTAC uses the OPIs…

More on evaluation… Align the intensity of the evaluation with the intensity of the activity –One time activity results in less intense evaluation than sustained professional development. Align needs (OPIs) of the activity with evaluation

Needs Assessment Activity (Effort) PROJECT GOALS Evaluation (Effect) Beginning: Identifying what needs to be done and selecting OPIs related to the needs. Middle: (1) Planning and delivering TA to meet the needs. (2) Planning on how you will evaluate effort. End: Documenting and evaluating your efforts to determine your effects. Incorporating outcomes into project activities…

Examples of incorporating OPIs into evaluation Change of knowledge –“As a result of the training, I have increased my knowledge about the receptive and expressive array of language and communication skills for children with deaf-blindness.” (SP.07.a) Follow-up (Service Providers) –“Based on the recent training provided on use of strategies to improve communication and/or language, please indicate your progress in performing the following tasks…” responding to the child’s expressive communication attempts (SP.07.d) implementing expressive communication strategies across environments (SP.07.e)

Examples of incorporating OPIs into evaluation Follow-up (Child change) –“Three months ago, you received technical assistance on understanding how a combined vision and hearing loss impacts learning and social/emotional development. As a result of that training, please indicate any progress the student has made in the following skills…” Rate of expressive communication (C.04.a) Use of multiple communication intents (C.04.b)

Embedding the OPIs into initial Change of Knowledge evaluation As a result of the training, I have increased my knowledge in … Rating SAANDSD 1. The receptive and expressive array of language and communication skills (SP7a) Providing ongoing opportunities for receptive and expressive communication attempts (SP7b) Strategies to expand the student’s receptive & expressive communication (SP7c) 54321

Embedding the OPIs into Follow-up evaluation As a result of the trainings on communication, I have made progress in implementing skills in… Rating Substantial Progress Some Progress Emerging Progress A little Progress No Progress 1. Providing ongoing opportunities for receptive and expressive communication attempts (SP7b) Implementing strategies to expand the student’s receptive & expressive communication (SP7c) 54321

My student has made progress in… Rating Substantial Progress Some Progress Emerging Progress A little Progress No Progress 1. Responding to others (C03c) Increasing rate of expressive communication (C04a) Embedding child change OPIs into the follow-up evaluation

Reporting Example Effect Language Percentage of service providers reporting an increase in knowledge about strategies to improve communication and language for students who are deaf- blind (8 out of 10 = 80%) Percentage of service providers reporting an increase in skills to implement strategies to improve communication and language for students who are deaf- blind (5 out of 6 = 83%)

APR Example Effect Language Mean rating of students demonstrating a change in receptive communication (4.0 on a 5.0 scale) Mean rating of students demonstrating a change in expressive communication (3.5 on a 5.0 scale)

Using the OPIs to develop the “big picture” within a state

Building the “big picture” Needs assessment –Uses outcome statements from the child OPIs to focus on the student first Action plan –Uses performance indicators to target specific impact on child –Incorporates IFSP/IEP, state standards, alternate assessment to clearly align with state mandates Action plan for service provider –Uses service provider outcomes and performance indicators to link impact on student learning to effort of service provider Evaluation –Clearly articulates how student learning/progress will be measured

Linkages Needs assessment & action plan provide clear direction for service provider and/or family Needs assessment helps TA providers determine common needs across the state Action plan identifies common strategies and measures of impact on student learning (outcomes) Evaluation can be used by both the service provider and the TA provider

Keeping track FileMaker Pro database

Questions?