1. July Webinar: OFA Healthy Marriage Programs: Learning from Evaluations to Date July 29, 2011  1:00 pm-2:30pm (EST) Presenters: Alan J. Hawkins, Ph.D., Professor of Family Life, Brigham Young University Eiko Komuro Venovic, Psy.D., Clinical Psychologist and Assessment Coordinator of the Family Bridges Program Alicia La Hoz, Psy.D., Clinical Psychologist and Family Bridges Program Director Ted Strader, COPES Program Director Craig McGuire, COPES Evaluator National Healthy Marriage Resource Center Webinar

2. How to ask a question:

3. Findings from the Field: Results from the NHMRC Meta-Analysis of OFA Healthy Marriage Grantee Programs Alan J. Hawkins, Ph.D. Kaylene J. Fellows, M.S. Candidate Brigham Young University Provo, Utah (hawkinsa@byu.edu)

4. OFA HM Grantee Programs Meta-Analysis (“Bird’s Eye” View: Aggregates into One Study) All OFA HMI Grantees invited to participate by NHMRC About 50 Grantees had pre-post outcome and indicated interest 44 grantees volunteered data 32 grantees provided usable data 51 programs (some Grantees had multiple programs) 53,329 total evaluated participants (from 51 programs)

5. Meta-Analysis Findings: Overall Overall Program Impact: (all grantee programs—all outcomes) ▫Pre-post: d =.40*** (k = 46) ▫Pre-follow-up: d =.51*** (k = 8) ▫No gender difference An Interpretation: ▫about 66% of program participants at post-test scored above the mean of the pre-test (by definition, 50% of program participants scored above the mean at pre-test) ▫about 70% of program participants at follow-up-test scored above the mean of the pre-test *** p <.001

6. Meta-Analysis Findings: By AA Allowable ActivityEffect Size (d) Number of Programs (k) Youth/HS Students.45**7 Unwed Expectant.33***8 Premarital.25***4 Premarital+Married.39***10 Married.50***13 Divorce Reduction.50***1 Marriage Mentoring--0 ** p <.01, *** p <.001

7. Meta-Analysis Findings: By Outcome Relationship OutcomeTimingEffect dpk Relationship QualityPre-post.40***34 Relationship QualityPre-follow-up.518 Communications SkillsPre-post.41***44 Communication SkillsPre-follow-up.64***8 Relationship ConfidencePre-post.21***25 Relationship ConfidencePre-follow-up-- Relationship AggressionPre-post.26***19 Relationship AggressionPre-follow-up-- Unhealthy Relationship KnowledgePre-post.44***15 Unhealthy Relationship KnowledgePre-follow-up-- Co-parentingPre-post.18***13 Co-parentingPre-follow-up-- ***p <.001

8. Retrospective vs. Pre-Post *** p <.001 MethodologyEffect (d)Number of programs (k) Difference? Pre-Post.39***46 p =.07 Retrospective.64***6

9. Moderators of Program Effects Were there differences for program dosage? Maybe ▫Medium dosage had stronger effects (p =.09)  Low dosage (8 hours^): d =.35*** (k = 10)  Medium dosage (9-20 hrs): d =.42*** (k = 27)  High dosage (20+ hours): d =.22*** (k = 4) ▫Mean dosage = 12.3 hours *** p <.001 ^ Note: Minimum 8 hrs. offered

10. Moderators of Program Effects Were there differences for participant education level? Yes ▫Programs with greater proportions of less educated participants had stronger effects (p <.001)  Low (<80% HS grads): d =.45*** (k = 16)  High (>=80% HS grads): d =.19*** (k = 8) ▫Program Average: 21% of participants less than a HS education (Note: Youth/HS programs not included.)

11. Summary Overall, looking at 51 programs with more than 53,000 participants, there were moderate positive program outcome effects (in line with similar studies). Moderate, positive program outcome effects were obtained for programs in each allowable activity tested: (youth, unwed expectant, premarital, marital, divorce reduction). Positive program outcome effects were obtained for each measured outcome. No gender differences in outcomes (k = 22). Moderate-dosage programs tended to have higher effects than lower- & higher-dosage. Programs with larger proportions of participants who did not have a high school education had stronger positive outcome effects. But, significant limitations of the study...

12. Limitations of the Study Not all grantees participated (about 30%) Used non-controlled studies; may overestimate effects Few follow-up effects (high non-completion rates) Unstandardized outcome data collection Not all grantees participated (about 30%) Used non-controlled studies; may overestimate effects Few follow-up effects (high non-completion rates) Unstandardized outcome data collection

13. COPES Healthy Marriage Program Evaluation July 29, 2011 Ted Strader, Founder and Executive Director of COPES Craig McGuire, President of Craig McGuire and Associates, COPES Evaluator www.copes.org

14. About COPES COPES (The Council on Prevention and Education: Substances, Inc.) is a private, nonprofit 501 (c) (3) organization providing consultation, education and training services to youth and families. COPES’ CLFC program has demonstrated strong research results of increasing protective factors and personal and family behaviors. In 2006, COPES was awarded a five-year Healthy Marriage Initiative grant from ACF.

15. COPES’ Services & Target Population Creating Lasting Family Connections (CLFC) An adaptation of CLFC was implemented to improve relationship skills: either a 10-session program or a weekend retreat program. Target population: individuals recently released from prison and/or in substance abuse treatment and their spouses.

16. CLFC Program Objectives 1)Increase in knowledge and use of effective communication skills 2)Increase in knowledge and use of conflict resolution skills 3)Increase in knowledge and use of effective intra-personal skills 4)Increase in knowledge and use of emotional awareness skills 5)Increase in knowledge and use of emotional expression skills 6)Increase in knowledge and use of inter-personal skills 7)Increase in knowledge and use of relationship management skills 8)Increase in relationship satisfaction 9)Increase in relationship commitment

17. Research Design & Methods Participants: 401 individuals who either voluntarily participated in the implementation of CLFC (intervention) or another program typically offered for those being released from prison (comparison). Average age: 34. Majority African American and Caucasian. Surveys administered to asses relationship skills at pre-test, post- test, and follow-up (3-6 months after post-test) Multilevel modeling (e.g., HLM) examined whether change in intervention was more positive than change in comparison for relationship skills Communication SkillsConflict Resolution SkillsIntra-Personal Skills Emotional AwarenessEmotional ExpressionInter-Personal Skills Relationship Management Skills Relationship SatisfactionRelationship Commitment

18. Evaluation Results When comparing husbands to a comparable sample of men, relationship skills remained relatively constant for the comparison group; however, relationship skills improved for the CLFC group. Examining only husbands and wives exposed to the CLFC program, the pattern of changes in relationship skills was nearly identical for husbands and wives. Relationship skills increased between pre- and post-test and increasing slightly between post-test and follow-up. All findings were statistically significant

19. Summary and Conclusions Findings suggest a direct and positive impact of the CLFC intervention on relationship skills. The relationship skills targeted by the CLFC program reflect relationship maintenance strategies and social support, which have been shown in the research literature to be related to increased commitment to relationships and a lower likelihood of divorce. It is likely that this implementation of the CLFC that positively affected relationship skills accomplished the ultimate goals of the HMI initiative.

20. Evaluation Challenges & Strategies In Year 1, the number of reentry men who were married was less than 10%. Time scheduling challenges with reentry married couples. Led us to change/adapt the intervention (reduced the number of sessions). Original design = 20 session program During Year 1 = 10 sessions and/or weekend retreat format Anticipated challenge: Difficulty getting reentry men to complete the program; also, if you include their partner you will have high attrition. Was not an issue!

21. Lessons Learned & Recommendations Bring the Management Team, staff and evaluators together to design team approach to conduct program and evaluation in tandem. Together create or select the right instruments, surveys, questionnaires, etc. to measure Processes (staff actions) and Outcomes (Participant changes in knowledge, attitudes, skills or behaviors). Together agree on the who, what, when, where and how of survey administration and who will get what info when and how. Pre-, Post- and a Follow-up assessment allows evaluators to look at sustained impacts over time. Review evaluation results together and adapt program, project, or processes if necessary based on results.

22. Thank you! For more information about the COPES program, please contact: Ted N. Strader, MS, CPS (502) 583-6820 tstrader@sprynet.com www.copes.org

23. July 29, 2011 Alicia La Hoz, Psy.D., Program Director Eiko Venovic, Psy.D. Assessment Coordinator FAMILY BRIDGES 2100 Manchester Road Suite 1500 | Wheaton, IL 60187 877-41BRIDGE Using Evaluation Methods to Improve your Program and Help Tell Your Program’s Story. Chicagoland Healthy Marriage Initiative

24. 24 About Family Bridges Family Bridges is a healthy marriage initiative program in the Chicagoland area that focuses on relationship-building skills Our workshops are offered daily throughout the Chicagoland area for couples, families, individuals and high school students Our events are offered in both English and Spanish Our educational materials are backed with years of research and have proven to be effective over and over again We partner with 12 partnering organizations to deliver services to our community. Our partners are the bridges into our Chicagoland community that makes the Family Bridges program a success. Vision: Transforming communities by empowering families. Mission: Family Bridges promotes thriving communities by delivering services to strengthen family relationships, while embracing their cultural and spiritual values.

25. 25 About what we do: logic model Couples workshops (married & engaged) Singles workshops (Expected parents & Divorce Reduction) High School students Workshops Public awareness campaign/ advertising Mentoring No. of couples that participated in workshops; No. of workshops attended No. of Single adults that participated in workshops; No. of workshops attended No. of high school students that Participated in workshops; No. of workshops attended No. of materials disseminated; Audience reach, etc. Improved communication Improved conflict resolution skills Increased commitment to the relationship Improved safety Increased social support Activities/Services Outputs Outcomes Awareness of self behavior Budgeting & finance skills Safety & risks re: relationships Understanding relationship development No. of couples & singles that participated in Mentoring June 17, 2009

26. 26 About our evaluation plan

27. 27 Context: program fidelity 21 Critical Criteria Corresponding with Activity Areas & Approach in Grant Narrative NACF Healthy Marriage Initiative: Knowledge & Commitment HMI Program Implementation: Curriculum Follow-Up NACF Healthy Marriage Initiative: Engagement HMI Program Implementation: Ancillary Services: Referral & Tracking Outreach: Intended Target Population HMI Program Implementation: Ancillary Services: Screening Tools Outreach: To Critical Populations Cultural Competence: Frequency & Quality of Training Outreach: To Community Organizations & Networks Cultural Competence: Monitoring & Feedback HMI Program Implementation: Public Awareness Cultural Competence: Implementation & Outcomes HMI Program Implementation: Marriage Education Curriculum: Curriculum Selection & Implementation Domestic Violence: Training HMI Program Implementation: Marriage Education Curriculum: Curriculum- Participant Fit Domestic Violence: Incorporation Into Intake Process HMI Program Implementation: Marriage Education Curriculum: Assessment Domestic Violence: Incorporation Into Marriage Education Curriculum HMI Program Implementation: Peer Mentoring FBHMI Project Management: Communication & Connection Within the Provider Organization Collaboration with Wheaton College for Structural Organizational Interviews Conducted  Annually Baseline Assessment  Partner Organizational Assessment

28. 28 Input: program objectives Participant Outcomes Instruments Intake Pre/Post Satisfaction Surveys Mentoring Survey Exit Survey Focus Groups Facilitator Evaluations Instruments Program Summary Reports Curriculum Summary Reports Facilitator Evaluations Site Visits

29. 29 Process: flow chart Process Flow Chart Intake Conduct Class Submit required paperwork to Meier Clinics Foundation Input data in website Meier Clinics Foundation reviews all documents Process to accounting and payment is made If missing documents org is notified Outreach Complete Class

30. 30 AA2-AA7 results +30,000 Number of individuals served via workshops and events to-date Eight to sixteen hours of MRE to: 5,809 students in youth and community groups, and high schools (AA2) 252 expectant mothers or mothers with children one years old or younger (AA3) 407 engaged adults (AA4) 9,434 married adults (AA5) 6,218 single adults (AA6) 6,824 mentoring follow-up workshops (AA7)

31. 31 AA2-AA6: outcomes Couples Individuals High School Couples, individuals & students reported improvement… 67% communication skills 70% conflict resolution skills 70% sense of commitment 52% parenting skills 61% communication skills 62% conflict resolution skills 57% sense of commitment 14% parenting skills 73% understanding of relationship development 72% communication skills 69% understanding & detecting self-destructive behavior 72% understanding in finance & budgeting 77% understanding risk & safety in relationships Couple participants reported increasing their marital satisfaction by 42% as a result of our program

32. 32 Dissemination of outcomes Annual Feedback of Organizational Assessment Annual Outcome Summary Reports by Organizations Annual Reports Documentary & Testimonials Resource Development for Sustainability Shock Sheets Case Study Professional Presentations & Community Forum Website Ongoing (5) Dissertations w/Students

33. 33 Implementation of standardized measures Listening to partners on the ground when implementing evaluation plan Integrating both qualitative & quantitative Practicing qualitative assurance practices Multiplication of outreach & service with strong partnerships Summary of lessons learned