Workshop/Breakout Title Workshop/Breakout Speaker(s) Demonstrating, Replicating and Expanding Evidence-based Social Programs Geri Summerville, Public/Private Ventures Tammi Fleming, Public/Private Ventures Debra Delgado, The Annie E. Casey Foundation

Discussion Topics When is a program ready for replication? Defining a program’s essential elements. Planning for program growth. Developing and managing a national replication effort.

When is a program ready for replication? Does the program address an important public problem or need? Does the program achieve positive, measurable results? Does it achieve these results in a timely fashion? Can it make a convincing case that the program, not other factors, caused the results?

Defining a Program’s Essential Elements Knowing how a program works and why it works. Closely examining a program’s implementation history and data collection/evaluation. Identifying the core program components and related activities.

Planning for Program Growth Material Development: Development of an implementation manual. Development of standardized training curriculum. Branding/logo/trademarks/copyrights Brochures

Internal Capacity Assessment National Office/Intermediary: What are the staffing requirements? Is there enough funding available to continue growth without negatively affecting initial program? What space is available for replication work? What support is needed for finance and administration: grants and contracts.

External Market Assessment Identify and build relationships with “Partnering” agencies/organizations. Focused presentations at targeted national and state conferences. Advocate for the program with top leaders that control “purse” strings – mayors, governors, state legislators, etc.

Developing and Managing a National Replication Effort A national replication effort should only be considered once a program has had a formal evaluation showing evidence of effectiveness AND after the growth planning process has been completed.

Developing and Managing a National Replication Effort (cont’d) Create internal structures: - Site/state development strategy - Staffing - Standardized training and technical assistance - Universal data collection system - Networking opportunities for sites - Funding information and resources - Marketing strategy/tools

Moving from Demonstration to Evaluation to Replication: A Case Study of Plain Talk

The Plain Talk Vision “All vulnerable youth will have a knowledgeable, caring adult they can turn to for support, information and helpful communication in order to make informed, responsible decisions about sexual behaviors and consequently increase options for connecting to educational and economic opportunities.”

Plain Talk Demonstration Phase Time Frame: 1994 – 1998 Cost: $5M Locations: –Atlanta, Ga. –Hartford, Ct. –New Orleans, La. –San Diego, Ca. –Seattle, Wa.

Plain Talk Overview Plain Talk is about: –Messages; –Messengers; and –Skills, tools and information that messengers want and need to deliver clear, consistent and accurate messages about protection from too- early pregnancy, STDs and HIV/AIDs.

Theory of Change Create Consensus Educate Community Adults Effective Communication Use of Contraception and STD protection Adolescent Knowledge and Attitudes Pregnancy and STD rate Reproductive Health Services

Developing a Blue Print for Change. The Foundation provided: –$1 million grants over 4 years; –A theory of change; –Benchmarks to guide progress through the planning phase; –Data analysis for the community mapping process; –Extensive technical assistance co-designed with sites; and –Cross-site networking.

The Program Demonstration Phase had Strong Results but Were They Compelling Enough to Justify Investments in Replication? Replication challenges: Limited knowledge about how to replicate a teen pregnancy prevention program; Complexities of the community change model; Time; and Money.

Post-Demonstration Phase Work 1998 – 2000 Final household survey, data collection and analysis to develop “Process Report” 2001 – Final Outcomes Report 2001/2003 Conducted replication prototypes to test assumptions about: –Cost –Feasibility –Potential for achieving results 2003/2006 Product development (peer review journal article, Plain Talk Implementation Manual, Training of Trainers Guides)

Moving Beyond Program Demonstration Collect strong & compelling data; Conduct extensive analytical work on issues related to implementation; Develop & test replication prototypes; and Create tools and processes to transfer knowledge in a timely, cost-effective way.

Replication Elements Social policy history in the U.S. reveals that the “vision” for replication is generally limited to “duplication of programs”. However there are broader issues to consider, namely SCALE, LONGEVITY and EFFECTIVENESS.

Replication Elements With this in mind, our Foundation developed a replication strategy that depends upon three critical elements: A Powerful intervention that is evidence-based; Powerful partners with clear roles; and Powerful co-investors (financial, intellectual, political, etc). Intervention Partners Co-Investors

Partnership Examples STATECO-INVESTORADVOCACY PARTNER LOCATION GeorgiaJane FondaG-CAPPPGainesville North Carolina SDOHAPPCNCSiler City CaliforniaCalifornia Endowment, Price Charities CACSAPSan Diego, Brawley MinnesotaDSS/OMH, Caroline Family Foundation MOAPPPSt. Paul, Minneapolis New MexicoDOH, DOENMTPCAlbuquerque OhioLucas County Department of Health Weed & Seed Initiative Toledo

Emerging Lessons Learned After 10-plus Years Program demonstration is a laboratory not a blue print for success; A thoughtful and thorough evaluation provides important insights about what works and why; Program replication is essential but not sufficient for achieving impact…replication must be analyzed & implemented within the context of scale and sustainability; and Public/private partnerships are the key to success.

REPLICATION IS ABOUT REPRODUCING RESULTS - NOT JUST PROGRAMS.

Evaluation Questions  Were the sites able to create a community consensus?  How effective were the community education strategies in educating a large number of adults?  Did the sites improve access to quality, age appropriate reproductive health services?  Did Plain Talk have an effect on adolescents’ behaviors and teen pregnancy rates?

Evaluation Design Multiple methods –Surveys of community adolescents in 1994 and 1998 –Implementation research –Ethnographic research –Birth records data

Key Long-Term Outcomes 1994: 33 % of sexually experienced youth reported pregnancy 1998: 27 % of sexually experienced youth reported pregnancy With no intervention, P/PV estimated that pregnancy rate would have been 38%

Key Long-Term Outcomes Proportion of sexually experienced youth who spoke with adults about topics related to sexuality were half as likely to: – have an STD, – have had or created a pregnancy – have a child Compared to youth who did not talk with adults

Interim Outcomes Increased levels of talk between adults and sexually active youth –61% in 1994 –70% in 1998 “Talkers” showed increased knowledge about sexuality and birth control –1.4 times more likely to know where to get birth control in 1994 –2.1 times more likely to know where to get birth control in 1998.

Linking Evaluation Results With Program Components The two sites that used Peer-Educators (Walkers & Talkers) in conjunction with Home Health Parties had a higher degree of success in educating a larger number of adult residents 800 in New Orleans 1,350 in San Diego Sites that equally focused on both increasing adults sexual knowledge and increasing adults ability to communicate with teens about sexual issues had better success

Assessed program implementation in Atlanta, New Orleans and San Diego Examined elements of each program that were successful Paid close attention to similarities across sites Replication Assessment of Plain Talk Program

Reviewed evaluation Conducted site visits Interviewed all program staff available Interviewed community residents who were involved in Plain Talk Reviewed records and documentation Methods of Assessment

Community Mapping Walkers & Talkers (Peer Educators) Home Health Parties The More Successful Sites had 3 Common Program Components

Replication assessment showed the need for development of uniform program materials to replicate Plain Talk in an efficient, cost-effective manner 1) Plain Talk Implementation Guide 2) Training Curriculum for Walkers & Talkers Program Material Development

THE EVALUATION INFORMED THE REPLICATION ASSESSMENT & THE REPLICATION ASSESSMENT CONFIRMED THE EVALUATION RESULTS

Building a National Replication Structure Services and Supports Provided to Plain Talk Sites by P/PV Funded by The Annie E. Casey Foundation

Ongoing Technical Assistance Provide intensive start-up TA Provide assistance on a continual basis to all program sites. Develop a monthly call schedule to discuss problems, progress, etc. Conduct routine site visits to assist with funders, community leaders, agency staff.

Provide Funding Information Development of a web-based funding guide for program sites that includes information on: – All types of federal and state funding applicable to the ARH field (Title V, Title X, TANF, etc.) – How current program sites access each particular funding source. – The political climate in each state regarding accessing the different types of funding. – What private/foundation funding is available. – Templates for grant proposals. – Partnering organizations in each state to help access funding.

Standardized Training for all Program Sites Training that ensures fidelity to the program model. Use of standardized training curriculum. Consistent schedule of trainings - Training One – Planning for Plain Talk - Training Two – Community Mapping - Training Three – Walkers & Talkers

Networking Between Program Sites Regional conference calls Web-based listserv and bulletinboards Annual conferences Peer-to-peer scheduled site visits

Community Mapping Data Analysis Once Community Mapping is completed, sites send all surveys to P/PV. P/PV provides comprehensive data analysis reports of community mapping results. This analysis saves sites up to $10,000.

Universal Data Collection System All sites collect the same data. Web-based, user-friendly data collection system. Collect both Implementation and Outcome data. All sites have continual access to current data reports. Provide sites with annual cumulative reports.

Site/State Development Activities Stage 1 – DEVELOPMENT ( months) Align external resources with local/state discussions, (e.g.,. National Campaign to Prevent Teen Pregnancy Roundtables) Stage 2 – PLANNING (0 - 3 months) Stakeholder discussions State policy/ administrators Private funders/ Statewide Advocacy network Community partners, including potential lead agency Kicks off when funds are secured. Activities include: Identifying target area and lead agency Developing initial training plan Conducting community mapping Initiating cross- site data collection activities

Site/State Development Activities (cont.) Stage 3 – PROGRAM IMPLEMENTATION ( months) Stage 4 – SCALE & SUSTAINABILITY (after 24 months) Disseminate community mapping data Conduct Home Health Parties Train Walkers & Talkers Identify/strengthen social network activities Develop service improvement strategies Expansion into neighboring communities Activities are similar to Stage 1 with greater emphasis on program results and community advocacy

2005 Site Update Stage 1: Colorado, Iowa, New Jersey, New York, South Carolina, Rhode Island, Pennsylvania Stage 2: Toledo (OH), Minneapolis (MN) Albuquerque (NM) Milwaukee, (WI) Stage 3: Gainesville (GA), New Orleans (LA) Siler City (NC) Brawley, CA Naranjito, PR Stage 4: San Diego (MidCity, Logan Heights), St. Paul (MN)

Questions? For More Information: