1. Building Support through Effective Communication Strategies

2. Robert Kight Chief, Division of WIA Adult Services and the Workforce System, ETA Steven Baker Vice President of Marketing and Communications, Jobs for the Future Jacob Klerman Principal Associate/Scientist, Abt Associates 2

3. Here’s What to Expect Cover the basics of good story telling Framing your project’s story Develop a working draft Leveraging evaluation results 3

4. How to use your story 4 Internal communications Talking points/speeches Presentations Funding proposals/reports Articles/publications Web/social media Press releases/media coverage Community engagement

5. Fundamentals of a good story 5 It is clear and understandable, no matter how technical the work may be It is relatable; the story connects the work to people It is compelling; the outcomes/benefits are easily understood It is motivating; your audience wants you to succeed and they want to help you

6. Framing your story 6 First decide: Who are your primary audiences? Program participants, businesses, funders, community leaders, policy makers What do they care about? Populations, efficiencies, outcomes What will you want them to do? Give direct support Advocate on your behalf

7. Drafting your story 7 What are the problem(s) you’re trying to solve?  Frame in terms of both systems and people What is your solution? How will it help?  Avoid jargon; keep it simple How is your solution unique? What’s innovative?  Don’t forget to take credit; you’re the hero! What are the intended outcomes? What will be different? Better?  Who will benefit? Why should people care?

8. Using Evaluation Results Jacob Alex Klerman WIF Grantees Conference Washington DC, March 2014

9. Abt Associates | pg 9 Outline  WIF as the “Vanguard”  Impact Results  Earlier Results  An Exercise

10. Abt Associates | pg 10 Outline  WIF as the “Vanguard”  Impact Results  Earlier Results  An Exercise

11. Abt Associates | pg 11 WIF Grantees are the “Vanguard”  Until recently, social policy in general and workforce strategies in particular have been set and funds allocated based on “plausibility” –And, rigorous impact evaluation suggests that impacts have often been mediocre  Increasingly (Obama Administration’s Evidence Agenda, ED’s i3, CNCS’s SIF), we see a move towards a new and better strategy: “Evidence Based Policy” –Pilot, rigorously (impact) evaluate, replicate –Broad scale rollout only after success at earlier steps –Leading to better programs and (any and) larger impacts WIF is a key component of the strategy

12. Abt Associates | pg 12 Perhaps “Bleeding Edge” …  Being part of the “vanguard” is not easy  We are moving towards “evidence based policy” exactly because, when evaluated, many programs will not be found to be effective –So participating in WIF—and being evaluated—is the “right thing” –But, the results are not always pleasant  The rest of this talk considers how to deal with that reality

13. Abt Associates | pg 13 Turn Burden into Selling Point  In your materials, note that you are “doing the right thing” –Stepping through the tiers of evidence  And, along the way, showing that: –Your program can actually implemented (i.e., a successful pilot; see third part of talk) –You are incremental tweaking your program (again, see third part of talk) –You can work constructive with an evaluator –You have good internal data systems that can support an evaluation –And, we hope building increasingly high quality evidence of impact (see the next section of the talk) … which are valued by “Evidence Based Policy” funders

14. Abt Associates | pg 14 Outline  WIF as the “Vanguard”  Impact Results  Earlier Results  An Exercise

15. Abt Associates | pg 15 Talking about Results; Especially if they are “Good”  Evaluator should produce a detailed report –Background, methods, data, results  Someone (evaluator or grantee) should produce a brief “Executive Summary” –Standard format is one page, two sides –See handout with example

16. Abt Associates | pg 16 Talking about Results; Especially if they are “Good”  Evaluator should produce a detailed report –Background, methods, data, results  Someone (evaluator or grantee) should produce a brief “Executive Summary” –Standard format is one page, two sides –See handout with example Executive Summary Standard Outline  Overview: One paragraph program and findings  Background  Program Design  Methods  Implementation findings  Impact findings, with 1-2 figures  Discussion  References You really need to do this, whether results are “good” or “bad”

17. Abt Associates | pg 17 Strength of Methodology  “Evidence Based Policy” is associated with “tiered evidence” –Programs move up the “tiers of evidence” –Pilot->pre/post->QED->random assignment->replication –And only then to broad program roll-out  Claim credit for working through the tiers, but, don’t over-state the strength of the evidence –Random assignment provides the strongest evidence of impact –Pre/post and QED tend to over-state impact –They should, therefore, be taken (primarily) as evidence that moving to the next tier of evidence is appropriate Exercise 1: Grantees and evaluators should discuss the evaluation’s methodology and how they will describe it and its strength

18. Abt Associates | pg 18 Consider Precision Carefully  A nearly ideal example: SEIS/Sectoral Employment Impact Study (Maguire, et al. 2010) –Random assignment evaluation found clear evidence of larger impacts –Impact on total earnings of 24 months since randomization  Why do we (“the policy community”) say that?

19. Abt Associates | pg 19 Consider Precision Carefully Case 1/SEIS Impact$4,509 This is a very large impact; 18% of control Standard Error$1,286 Relative to impact this is a small number T-statistic3.5= Impact / Standard Error P-valuep<0.01 **** If no impact, this would happen less than 1% of the time 95% Confidence Interval $1,806 - $9,785 Range of plausible (95% of time) impacts does not include zero Language “Strong Evidence” When p<0.95

20. Abt Associates | pg 20 Several Very Different Cases Case 1/SEISCase 2Case 3Case 4 Impact$4,509 $2,000$200 Standard Error$1,286$2,652$1,286$129 T-statistic3.511.701.56 P-valuep<0.01p<0.10p>0.10 95% Confidence Interval $1,806 – $9,785 -$689 – $9,707 -$521 – $4,521 -$52 – $452 Language “Strong evidence” “Limited evidence” “No evidence”

21. Abt Associates | pg 21 Several Very Different Cases Case 1/SEISCase 2Case 3Case 4 Impact$4,509 $2,000$200 Standard Error$1,286$2,652$1,286$129 T-statistic3.511.701.56 P-valuep<0.01p<0.10p>0.10 95% Confidence Interval $1,806 – $9,785 -$689 – $9,707 -$521 – $4,521 -$52 – $452 Language “Strong evidence” “Limited evidence” “No evidence” Just Can’t Tell No Impact  Very different cases; “Just Can’t Tell” is more common

22. Abt Associates | pg 22 Outline  WIF as the “Vanguard”  Impact Results  Earlier Results  An Exercise

23. Abt Associates | pg 23 Logic Model for Training Welders InputsOutputs Outcomes Short-Term Long-Term Staff Space Materials Partnerships Sessions held Teaching with fidelity Internships Attend sessions Graduate Pass external exam Employed in target industry Earnings Transfer program participation Taxes paid

24. Abt Associates | pg 24 So Far, Discussed Impact InputsOutputs Outcomes Short-TermLong-Term Staff Space Materials Partnerships Sessions held Teaching with fidelity Interships Attend sessions Graduate Pass external exam Employed in target industry Earnings Transfer program participation Taxes paid  “Impact”: Treatment vs. Control for “Long-Term Outcomes” –Not revealed until “later” –Not useful for short-term adjustments

25. Abt Associates | pg 25 Use Logic Model to Refine Program InputsOutputs Outcomes Short-TermLong-Term Staff Space Materials Partnerships Sessions held Teaching with fidelity Internships Attend sessions Graduate Pass external exam Employed in target industry Earnings Transfer program participation Taxes paid  Many “early steps” (inputs, activities/outputs, and short-term outcomes) can be observed early on, in your treatment group, at low cost

26. Abt Associates | pg 26 Learning from Early Results  A program’s Logic Model describes necessary—but perhaps not sufficient—early steps to achieve meaningful impact  You are currently implementing the program; you can check those early steps –Epstein and Klerman (2012) note that you will often find that the early steps are not achieved  If your program is not achieving early steps, you can adjust the program design and implementation –Now, during this implementation (before impact results are known) –For the next implementation (in response to impact results)

27. Abt Associates | pg 27 Some Early Steps to Check 1.Secured partnerships? Recruited and retained the right staff? 2.Attracted target number of participants? 3.Do Enrolled participants complete (enough of) the program? 4.Is the program implemented with fidelity? 5.Do participants show progress on pre/post measures of the program’s short-term outcomes? 6.Do participants pass external exams? 7.Do participants find employment in targeted industry? Program details matter; examples have caveats

28. Abt Associates | pg 28 An Exercise  WIF as the “Vanguard”  Impact Results  Earlier Results  An Exercise

29. Abt Associates | pg 29 Exercise 2: Motivation  Epstein and Klerman (2012) argue that: –Even w/o an impact evaluation, we can establish that some (many?) programs are unlikely to show impact –Because they don’t “succeed” in the earlier steps of their own logic model  This exercise attempts to help you to be ready to use that insight constructively: –In this grant cycle, and in future grant cycles

30. Abt Associates | pg 30 Exercise 2: Part 1  Who: Grantee and evaluator, together  Materials: A copy of your program’s “Logic Model” –Ideally, a figure/graphic and the narrative discussing it  Part 1: Walk through your logic model, identifying each of the verifiable early steps –Inputs acquired –Activities and outputs produced –Outcomes achieved—in the treatment group, during or at the end of treatment

31. Abt Associates | pg 31 Exercise 2: Parts 2-5  Part 2: For each step, define “success” –Often defining success will require a quantitative standard (e.g., number of trainees enrolled, percent of classes actually attended)  Part 3: Establish how and how early you can (and will) easily measure “success” for each earlier step  Part 4: If not “successful” at an early step of your logic model, discuss how you will “adjust” your program –During this grant period or in the next grant period  Step 5: If we identify common issues, we can try to arrange technical assistance

32. Abt Associates | pg 32 For More on these Ideas  Wholey, Joseph 1994. ‘‘Assessing the Feasibility and Likely Usefulness of Evaluation.’’ In Handbook of Practical Program Evaluation, edited by H. P. Hatry, J. S. Wholey, & K. E. Newcomer. San Francisco, CA: Jossey-Bass.  Epstein, Diana and J.A. Klerman. 2012. “When is a Program Ready for Rigorous Impact Evaluation?” Evaluation Review. 36(5): 373-399.  Abt Associates Policy Brief: When is a Social Program Ready for Rigorous Impact Evaluation?

33. Abt Associates | pg 33 Outline  WIF as the “Vanguard”  Impact Results  Earlier Results  An Exercise