CAPP Evaluation August 2011 Update Jane Powers, Amanda Purington, Jenny Parise ACT for Youth Center of Excellence

Overview Review Evaluation Plan: Documenting EBP Implementation Ways to improve documentation Revisions to the Tools Next Steps

CAPP Initiative Goal 1: Promote healthy sexual behaviors and reduce the practice of risky sexual behaviors among adolescents Provide comprehensive, age appropriate, evidence-based and medically accurate sexuality education to promote healthy sexual behaviors Core Strategy 1: Provide comprehensive, age appropriate, evidence-based and medically accurate sexuality education to promote healthy sexual behaviors

We know a lot about what works to prevent teen pregnancy EBP Increase age of first intercourse Increase use of condoms Decrease # sexual partners Decrease frequency of sex Decrease Teen Pregnancy Promote Adol Sexual Health

What does it take to implement EBPs in real world settings???

Research to Practice While many EBPs have yielded positive outcomes in research settings, the record at the local level of “practice” is mixed (Wandersman, 2009; Lesesne et al, 2008; Fixsen, 2005). While many EBPs have yielded positive outcomes in research settings, the record at the local level of “practice” is mixed (Wandersman, 2009; Lesesne et al, 2008; Fixsen, 2005).

What do we know about Implementation? Implementation influences program outcomes Implementation influences program outcomes If EBPs are not implemented with fidelity and quality, not likely to result in outcomes observed in research If EBPs are not implemented with fidelity and quality, not likely to result in outcomes observed in research Achieving good implementation increases chances of program success and stronger benefits for participants Achieving good implementation increases chances of program success and stronger benefits for participants (Durlak and DuPre, 2008)

Factors Affecting Implementation Community Level Community Level Organizational Capacity Organizational Capacity Program Characteristics Program Characteristics Training and TA Training and TA Facilitator Characteristics Facilitator Characteristics

CAPP An Opportunity to Learn!!!

Opportunity to learn: How do EBPs work in different settings? How do EBPs work in different settings? How do EBPs work with diverse youth populations? How do EBPs work with diverse youth populations? What adaptations are needed to make EBPs work more effectively? What adaptations are needed to make EBPs work more effectively? Are EBPs successful in getting to outcomes? Are EBPs successful in getting to outcomes?

Documenting Implementation is critical! Helps us learn how the intervention was delivered Helps us learn how the intervention was delivered Helps us understand challenges faced Helps us understand challenges faced Helps us learn about what does and does not work re implementing EBPs Helps us learn about what does and does not work re implementing EBPs

Tools to evaluate Implementation Fidelity Checklist: document what you did, adaptations made, successes/challenges Fidelity Checklist: document what you did, adaptations made, successes/challenges Attendance Record: who, where, when, how much Attendance Record: who, where, when, how much

Role of the Fidelity Checklists

Document: Document: – What you did – Timing – Adaptations made and why – Challenges and successes

Role of the Fidelity Checklists Help you Prepare!! Help you Prepare!!

Using Fidelity Checklist to Plan Review activities Review activities What equipment is needed? What equipment is needed? Timing Timing Anticipating issues Anticipating issues Questions that might come up Questions that might come up Prepare for next cycle Prepare for next cycle

What have we learned?

Ways to Improve Documentation 1. Provide More Details! –WHAT was changed? –WHY was it changed?

GREAT example “We took the entire class period to implement and process this activity. Although it is timed at 15 minutes, it took ten minutes alone to break the class of 32 students into groups, hand out cards and explain the activity. We also needed more time to allow groups to respond as we physically had more groups than if we were working in a small group setting.” “We took the entire class period to implement and process this activity. Although it is timed at 15 minutes, it took ten minutes alone to break the class of 32 students into groups, hand out cards and explain the activity. We also needed more time to allow groups to respond as we physically had more groups than if we were working in a small group setting.”

Good example “ Due to the age of the participants and the material in the Robert Townsend video clip, the teacher did not find it to be appropriate for the classroom. Instead we discussed the information in the clip and used a short role-playing exercise to make the information clear.” –Why wasn’t it appropriate? –What was the role-playing exercise?

Need more details “ran out of time” Tells us WHY but not WHAT was changed Tells us WHY but not WHAT was changed Was the activity shortened (if so, how?) or skipped altogether? Was the activity shortened (if so, how?) or skipped altogether? “switched the order of these activities” Tells us WHAT the change was but not WHY it was made Tells us WHAT the change was but not WHY it was made

2. Document ALL adaptations Was Activity Carried Out According to Direction in the Facilitator’s Curriculum? Leave something out Leave something out Change order Change order Add new content Add new content Shorten activity Shorten activity Substitute activity Substitute activity Modify activity Modify activity

2. Document ALL adaptations cont. –Include info on pre-approved adaptations –Were NO adaptations at all made?? –Example NO adaptations made NO adaptations made General Comment: “It would have been nice to have more time for each activity/module” General Comment: “It would have been nice to have more time for each activity/module” – Were activities completed according to the curriculum? – Were some activities omitted or shortened?

3. Complete the Fidelity Checklist right away! –Capture this information when the experience of the module and the activities are fresh in your mind

4. Complete the right number of tools One set of tools per cycle per set of participants

Class 1 Class 3 Class 4 Class 5 Fidelity ChecklistAttendance Record Class 2 Fidelity ChecklistAttendance Record Fidelity ChecklistAttendance Record Fidelity ChecklistAttendance Record Fidelity ChecklistAttendance Record

Teacher 1 Teacher 2 Teacher 3 Fidelity ChecklistAttendance Record

Common Adaptations Common adaptations include: Common adaptations include: –Updating videos, or using something more relevant in place of videos –Changes to accommodate timing issues –Changing language/names to be more inclusive How can the COE support these common adaptations? How can the COE support these common adaptations? –Providing/linking to updates on medical information –Identifying alternative videos (updated) –Developing alternatives to the condom demonstration

Documentation Details Block names of individual participants Block names of individual participants Do not use acronyms – spell these out Do not use acronyms – spell these out Write legibly Write legibly Review your documentation for typos beforehand Review your documentation for typos beforehand Bundle all evaluation materials for each cycle (fidelity checklists and attendance records) and mail to: Bundle all evaluation materials for each cycle (fidelity checklists and attendance records) and mail to: Amy Breese Cornell University ACT for Youth Center of Excellence Beebe Hall Ithaca, NY 14853

Submit tools electronically Save the fidelity check list and attendance record under a different name and input the information Save the fidelity check list and attendance record under a different name and input the information Preserve the template for future use Preserve the template for future use Send the documents to Amanda Purington Send the documents to Amanda Purington

Revisions to the Fidelity Checklists 1. Clarified the adaptations description field: “If Changed, WHAT was changed and WHY? Please be specific: describe things you left out, added, or changed and WHY.” 2. Added “Were changes (if any) pre-approved?” 3. Added a comments section after each module: “Please use this space if you have comments on this module or any of its activities:”

Revisions to the Attendance Record 1.Added “Age” column for participants

Next Steps Additional tools: Additional tools: –Participant Feedback Form –Facilitator Feedback Tools –Pre/Post Tests –Other needs? Tools and presentations on - Search for “CAPP” in the site search box at the top of the home page. Tools and presentations on - Search for “CAPP” in the site search box at the top of the home page. Revise Tools for Year 2 Revise Tools for Year 2

Contact Information Amanda Purington: or Jane Powers: