Aggregating Outcomes for Effort and Effect: What NTAC Learned from its Site Review Ella L. Taylor, Ph.D. NTAC Teaching Research Institute Western Oregon University

Agenda Site review response to evaluation Revisions to evaluation plan Collecting effort and effect data Aggregating data Questions

Site Review Concerns about Evaluation Very complex Less likely to succeed than a simpler plan Needs to be simplified Needs to be made more realistic and appropriate

Constraints of Evaluation Everyone wants evaluation to be seamless and transparent Reticence to see the direct benefit for improving the project Evaluation is seen as cumbersome and confusing

Primary purpose of evaluation 1. Did we do what we said we were going to do? (effort) 2. What was the impact of what we did? (effect)

Effort vs. Effect Effort – actions carried out by the project Satisfaction data Numbers of participants, events, etc. Effect – impact of the actions What outcome resulted from the activity? Change of awareness Change of knowledge Change in skill/implementation (service provider, family, systems) Change in child (child change data)

Alignment What needs to be done? (Needs Assessment) How to meet the need? (Activity) (Effort data) Did we meet the need? (Evaluation) (Effort & Effect data)

Did we do what we said we were going to do? (EFFORT) Grant objectives Met = explain how we met them Not met = explain why not Data Number of events Number of participants Satisfaction with effort

What was the impact of what we did? (EFFECT) What to measure? How to measure? How to report succinctly? How to aggregate?

What was the impact of what we did? (EFFECT) Outcome and Performance Indicators (NTAC’s OPIs) Outcome A statement of a measurable condition or an expected result or change. (i.e. increase, improvement, progress toward). Performance Indicator A statement that helps quantify the outcome and indicates whether the outcome has been achieved. Often, multiple indicators may provide better evidence of the achievement of an outcome.

Kudos to… John Killoran Kathy McNulty Paddi Davies Many, many hours of development and refinement of NTAC’s OPIs

OPIs Comprehensive outcomes for children (15), families (9), service providers (19) and systems (6) Embed in all aspects of planning, delivery and evaluation Aligns needs assessment, project activities and measurement of impact On web (

How we are using OPIs Planning Identify the stakeholder group (service provider, child, family and/or systems) Identify the outcomes you will target Identify the performance indicators that will help you determine attainment of targeted outcome(s) Delivery of service Implement TA that is targeted to the outcomes selected Evaluation Tailor assessment/evaluation measures to targeted outcomes and performance indicators

An Example

Webinar Example: Planning Needs Assessment Comments at Project Directors’ Meeting during self- evaluation breakout sessions Conversations with state project directors, coordinators and staff NTAC Advisory committee meeting Outcome Goal: The use of formative and summative evaluation of the systems change and/or capacity building has increased (S4) Performance Indicator: Uses outcomes measures (S4g)

Webinar Example: Delivery of Service Activity: Webinar Align the intensity of our evaluation with the intensity of the activity One time activity results in less intense evaluation than sustained professional development. Align needs with activity with evaluation

Webinar Example Evaluation: Change of awareness Item Rating SAA N A Nor D DSDNA 1. The presenter had the necessary expertise NA 2. I increased my awareness of using outcome measures in evaluation (S4g) NA 3. I felt this activity was a good use of my time NA

A slightly more complicated example…

Example 2 Planning/Needs: Ongoing training and support Outcome: Use of formative and summative evaluation (Systems 4) Performance Indicators: Uses satisfaction measures (S4d) Uses awareness, knowledge or skills measures (S4e) Uses outcomes measures (S4g) Activity: Series of webinars for one region Evaluation: Change of knowledge & skill

I learned how to develop, implement and analyze… Rating SAA N A Nor D DSDNA 1. Satisfaction measures (S4d) 54321NA 2. Awareness, knowledge and skills measures (S4e) NA 3. Outcomes measures (S4g) NA

Another more complicated example…

Example 3 Planning/Needs: Development, implementation, analysis & support in the use of formative and summative evaluation Outcome: Use of formative and summative evaluation (Systems 4) Performance indicators: Uses participant demographic data (S4c) Uses satisfaction measures (S4d) Uses change in awareness, knowledge or skills measures (S4e) Uses outcomes measures (S4g) Uses formative and summative evaluation information for ongoing feedback and continuous improvement (S4j) Disseminates evaluation results of the systems change or capacity building activities (S4k)

Sustained Professional Development Activity: Multiple visits by TAS and evaluation specialist to 2 states Evaluation: Change of knowledge & skill Follow-up

As a result of the training, my ability to develop, implement and analyze the following evaluation measures has seen … Rating Substantial progress Some progress No progress 1. Use of demographic data (S4c) Use of satisfaction data (S4d) Use of A, K or S… (S4e) The remainder of PI… 321

Aggregating the Data

Data Collection What data did we collect? –Number of events/activities (effort) –Satisfaction (effort) –Change of awareness (effect) –Change of knowledge & skill (effect) –Follow-up evaluation (effect) On what, did we collect data? –Outcome: Use of formative & summative evaluation (Systems 4) –Performance indicators: demographic data (S4c) satisfaction data (S4d) awareness, etc. (S4e) outcomes measures (S4g) ongoing feedback (S4j) dissemination (S4k)

Data Webinar (5 point) Series (5 point) Visits (5 point) Follow-up (3 point) Demo. datam = 4m = 3 Satisfaction data m = 4 m = 3 A, K or S data m = 4 m = 3 Outcomes data m = 4m = 3m = 4m = 3 Ongoing feedback m = 3m = 2.5 Dissemina- tion m = 3m = 2

Aggregate the data Each event carries equal weight Assign different weight to each event because events have different levels of importance Follow-up evaluation should carry more weight because the intensity of the effort was greater

Conversion to 4 point scale Convert 5 point scale to 4 point 5 (strongly agree) = 4 (achieved) 4 (agree) = 3 (nearly) 3 (neither) = 2 (emerging) 2 & 1 (disagree) = 1 (non-existent) Convert 3 point scale to 4 point 3 (substantial) = 4 (achieved) 2 (some) = 3 (nearly) 1 (no) = 1 (non-existent)

Webinar (4 point) Series (4 point) Visits (4 point) Follow-up (4 point) Demo. datam = 3m = 4 Satisfactionm = 3 m = 4 A, K or Sm = 3 m = 4 Outcomesm = 3m = 2m = 3m = 4 Ongoingm = 2m = 3.5 Dissem.m = 2m = 3 Subtotal Weight.25 Total Mean of the means = ( = ) divided by 4 events = 3.03 (nearly achieved)

What does this mean? Effort NTAC conducted one national webinar, a series of webinars for one region, and several onsite consultations with two states to increase the states’ capacity to use formative and summative evaluation systems. Across the trainings, participants indicated 90% satisfaction with the skill of the consultants and the content of the activities.

What does this mean? Effect Across the trainings and consultations, participants report that they are very near achieving the ability to develop, implement, and analyze formative and summative evaluation measures to increase capacity and systems change (m = 3.03/4.0 scale). Could elaborate by listing performance indicators if needed.

Response to NTAC Site Review: 2004 – 2005 Field test 1)Embed Outcomes and Performance Indicators (OPIs) in planning and delivery of service. 2)Embed OPIs in all evaluation measures. 3)Share our evaluation systems and data with our state/multi-state partners

Addressing the constraints Aligning needs, delivery and evaluation through the OPIs yields a more seamless system (Constraint 1) Sharing data facilitates the use of data (Constraint 2) Consistency helps diminish confusion (Constraint 3)

Questions?

Contact Region 1 Shawn Barnard Paddi Davies Region 2 Jon Harding Barb Purvis Region 3 Nancy Donta Amy Parker Region 4 Kathy McNulty Therese Madden Rose

Additional Examples The following information will not be shared during the discussion, but is being provided as additional material.

Weighting the events differently Using the previous examples, let’s say that we believe the follow-up data should carry more weight since it indicates more long-term implementation and attainment of the outcome. We want the follow-up evaluation to carry 40% of the weight.

Webinar (3 point) Series (3 point) Visits (3 point) Follow-up (3 point) Demo. datam = 3m = 4 Satisfactionm = 3 m = 4 A, K or Sm = 3 m = 4 Outcomesm = 3m = 2m = 3m = 4 Ongoingm = 2m = 3.5 Dissem.m = 2m = 3 Subtotal Weight Total Mean of the means = ( = 14.20) divided by 4 events = 3.55 (with rounding – achieved!)

Embed in basic evaluation (Service Provider 1a) Satisfaction data “I was satisfied with my opportunity to learn about the impact of deaf-blindness on an individual’s overall development (i.e. social, emotional, cognitive).” Change of awareness “I have increased my awareness about the impact of deaf-blindness on an individual’s overall development (i.e. social, emotional, cognitive).” Change of knowledge/skill “As a result of the training, I can use my knowledge about the impact of deaf-blindness on an individual’s overall development (i.e. social, emotional, cognitive) to plan instruction.”

Embed in Follow-up Service Providers “Based on the recent training provided on understanding how a combined vision and hearing loss impacts learning and social/emotional development, please indicate your progress in performing the following tasks…” Child change “Three months ago, you received technical assistance on understanding how a combined vision and hearing loss impacts learning and social/emotional development. As a result of that training, please indicate any progress the student has made in the following skills…”