1. Technical Assistance on Evaluation to Child Welfare Discretionary Grantees 2012 Abandoned Infants Assistance Comprehensive Support Services for Families Affected by Substance Abuse and/or HIV/AIDS Overview November 2012 James Bell Associates

2. James Bell Associates (JBA): Who We Are… Small business located in Arlington, VA. Small business located in Arlington, VA. Conducting research and evaluation in public health and child welfare since 1979. Conducting research and evaluation in public health and child welfare since 1979. 30+ employees and nationwide pool of consultants. 30+ employees and nationwide pool of consultants. Experts in program evaluation, technical assistance and training, applied research. Experts in program evaluation, technical assistance and training, applied research. DHHS clients include ACF, HRSA, NIH, ASPE, and SAMHSA DHHS clients include ACF, HRSA, NIH, ASPE, and SAMHSA

3. JBA’s Evaluation TA Contract TA provider to CW discretionary grantees since 1999. TA provider to CW discretionary grantees since 1999. Number of grantees served annually varies (currently ~100). Number of grantees served annually varies (currently ~100). Other priority areas supported include Adoption Opportunities Programs, Child Abuse/Neglect Prevention, Integrating Trauma-Informed and Trauma-Focused Practice in CPS, Promoting Safe and Stable Families, Diligent Recruitment, Child Welfare and Early Education/Education Partnerships, Improving Service Delivery to Youth in CWS. Other priority areas supported include Adoption Opportunities Programs, Child Abuse/Neglect Prevention, Integrating Trauma-Informed and Trauma-Focused Practice in CPS, Promoting Safe and Stable Families, Diligent Recruitment, Child Welfare and Early Education/Education Partnerships, Improving Service Delivery to Youth in CWS. TA is also provided to Family Connection grantees under a separate cross-site evaluation contract. TA is also provided to Family Connection grantees under a separate cross-site evaluation contract.

4. Technical Assistance Goals Support the Children’s Bureau in disseminating knowledge and promoting the use of effective and evidence-based practices in the child welfare field. Support the Children’s Bureau in disseminating knowledge and promoting the use of effective and evidence-based practices in the child welfare field. Promote the implementation of rigorous, useful, and practical evaluations. Promote the implementation of rigorous, useful, and practical evaluations. Strengthen the quality of evaluation reports and products available to the child welfare field. Strengthen the quality of evaluation reports and products available to the child welfare field.

5. JBA’s Evaluation TA Contract: What We Provide Responsive TA to individual grantees upon request. Responsive TA to individual grantees upon request. Proactive TA to individual grantees upon request of FPO and/or JBA recommendation. Proactive TA to individual grantees upon request of FPO and/or JBA recommendation. Group TA to grantee clusters (e.g., facilitating conference calls & Webinars regarding research/evaluation topics of common interest). Group TA to grantee clusters (e.g., facilitating conference calls & Webinars regarding research/evaluation topics of common interest).

6. JBA’s Evaluation TA Contract: Important Facts JBA is not a program monitor or auditor; our role is to work in partnership with the Children’s Bureau to assist you in implementing a quality evaluation. JBA is not a program monitor or auditor; our role is to work in partnership with the Children’s Bureau to assist you in implementing a quality evaluation. We do not conduct evaluations under this contract. We do not conduct evaluations under this contract. JBA’s services are available at no cost to grantees. JBA’s services are available at no cost to grantees.

7. TA Approach ― Introductory mailings ― Evaluation TA packet ― Kickoff Meeting sessions/calls ― Initial conference calls ― Develop TA plans ― Ongoing communication ― Web-based portal/information sharing ― Ongoing evaluation document review

8. Cluster TA Grantee meeting presentations and facilitated discussions Grantee meeting presentations and facilitated discussions Webinars and teleconferences Webinars and teleconferences Summary documents Summary documents Web-based portal Web-based portal Site-Specific TA Conference calls Conference calls Document review Document review E-mail E-mail Site visits Site visits TA Modalities Proactive and Responsive

9. More About Site-Specific TA Tailored to the specific interests and needs of grantees. Tailored to the specific interests and needs of grantees. Responsive TA: Grantee contacts JBA for advice or assistance with a specific evaluation issue. Responsive TA: Grantee contacts JBA for advice or assistance with a specific evaluation issue. Proactive TA: FPO asks JBA to contact grantee if a specific concern or TA need is identified. Proactive TA: FPO asks JBA to contact grantee if a specific concern or TA need is identified.

10. More About Cluster TA Initiated based on interests and needs of FPO and grantees. Initiated based on interests and needs of FPO and grantees. Focus is on peer learning, networking and information sharing. Focus is on peer learning, networking and information sharing. Examples: Facilitate collection and analysis of data on common outcome measures. Facilitate collection and analysis of data on common outcome measures. Facilitate discussion and information sharing regarding research and evaluation issues or challenges of common interest. Facilitate discussion and information sharing regarding research and evaluation issues or challenges of common interest.

11. Now that you have won your grant … Enhancements to the evaluation described in your proposal are encouraged. Components to revisit include: Enhancements to the evaluation described in your proposal are encouraged. Components to revisit include: Logic Model Logic Model Research/Evaluation Design Research/Evaluation Design Outputs/Outcomes Outputs/Outcomes Output/Outcome Measures Output/Outcome Measures Data Collection Tools/Instruments Data Collection Tools/Instruments

12. Consider Revising Your Evaluation Design If: Expected data sources are not available. Expected data sources are not available. Inadequate referrals pose challenges for random assignment or comparison group designs. Inadequate referrals pose challenges for random assignment or comparison group designs. Original evaluation plan/logic model focused on the “wrong” outcomes or measures. Original evaluation plan/logic model focused on the “wrong” outcomes or measures. Assessment instruments are invalid, unreliable, or otherwise inappropriate. Assessment instruments are invalid, unreliable, or otherwise inappropriate. Some interventions/activities were never implemented and cannot be evaluated. Some interventions/activities were never implemented and cannot be evaluated. Original evaluation plan was too ambitious. Original evaluation plan was too ambitious.

13. Back to the Drawing Board: Steps to Take Before Changing Your Evaluation Discuss your proposed changes with JBA. Discuss your proposed changes with JBA. Contact your FPO to obtain approval before you make any changes. JBA cannot approve changes! Contact your FPO to obtain approval before you make any changes. JBA cannot approve changes! Document changes in subsequent progress reports to CB and JBA. Document changes in subsequent progress reports to CB and JBA.

14. Include JBA in the Distribution of Semi-Annual Progress Reports Include JBA in the Distribution of Semi-Annual Progress Reports Send a copy directly to your JBA TA liaison every six months (electronic copies preferred). Send a copy directly to your JBA TA liaison every six months (electronic copies preferred). Allows JBA to stay informed about your project and identify possible TA needs. Allows JBA to stay informed about your project and identify possible TA needs.

15. Questions? Kim Keating, MEd Sr. Research Associate/TA Liaison keating@jbassoc.com 703/528-3230 James Bell Associates 3033 Wilson Boulevard, Suite 650 Arlington, VA 22201