1. EVALUATION PLANNING Legal Services Corporation May 19, 2004 Please standby while others connect

2. Slide 2 Innovation Network, Inc. Goals for the Session Review the LSC evaluation plan template Learn how to formulate strong evaluation questions Introduce LSC’s evaluation questions Give you the skills to understand which data collection techniques are best suited to answering what kinds of evaluation questions Gain a better understanding of how program assessment can help legal services organizations improve services

3. Slide 3 Innovation Network, Inc. Objectives At the end of the training grantees will be able to: Complete the LSC evaluation template Integrate LSC’s required evaluation questions with questions you develop on your own Deploy a data gathering strategy that helps you answer the questions you created

4. Slide 4 Innovation Network, Inc. Ground Rules Using Isoph Blue’s features We ask that you not use a speaker phone unless you have a mute button to block external sounds. Please do not use the “Hold” function on your phone, as it can induce a loud beeping noise for the rest of us on the call.

5. Slide 5 Innovation Network, Inc. Questions During the Session If you can, please hold your questions till we get to a question break If you can’t wait, please click the question mark and type your question into the window, we’ll answer as soon as we can. If we do not get a chance to answer your question, please email it to Ehren Reed at ereed@innonet.org.ereed@innonet.org

6. Slide 6 Innovation Network, Inc. What is Evaluation? The systematic collection of information about a program in order to better understand it, improve its effectiveness, replicate it and/or make decisions about future programming.

7. Slide 7 Innovation Network, Inc. Benefits of Evaluation Effectively communicates successes Builds common understanding about the program Helps you improve what you do Turns shortcomings into opportunities Helps staff overcome obstacles Positions your organization as a learning organization

8. Slide 8 Innovation Network, Inc. 10 Steps to Evaluation 1. Develop program goals 2. Identify specific goal targets 3. Outline strategies and activities 4. Develop evaluation questions 5. Create data collection instruments 6. Collect data 7. Analyze and interpret information 8. Present findings 9. Reflect on findings 10. Revise program & institute changes

9. Slide 9 Innovation Network, Inc. Evaluation Plan Project Goal(s)Strategies/ Activities to Achieve Goals Evaluation Questions Evaluation Data The overall aim(s) or intended impact(s) of your project The actions that the project takes to achieve desired outcomes The questions you want to answer about your project The evidence or information that tells you whether or not your project is achieving its outcomes

10. Slide 10 Innovation Network, Inc. Project Goals Clear statements identifying the ultimate results you want to achieve

11. Slide 11 Innovation Network, Inc. Goals & Objectives Develop objectives or more specific targets for your goals. Consider the major steps will you need to accomplish in order to achieve each of them. Make your objectives specific and achievable. Some goal words: Enhance Establish Enable Tip: Review your grant proposal to see what you said you would accomplish

12. Slide 12 Innovation Network, Inc. Some Intake Examples Intake Goal: Enable Asian-American clients in northern Virginia to seamlessly access legal services through a single, integrated intake system Objectives: Design integrated system that allows partner organizations to access critical client information

13. Slide 13 Innovation Network, Inc. Some Infrastructure Examples Infrastructure Goal: Enhance the delivery of legal services to low-income families in rural Mississippi via the website through high quality web applications Objectives: Develop partnerships with providers and case managers to identify critical needs Increase access to services by integrating communication systems

14. Slide 14 Innovation Network, Inc. Some Pro Se Examples Pro Se Goal: Enhance the ability of poor people in Ohio to access quality legal services to assert their rights and interests through online and in-person training and technical assistance Objectives: Identify patterns and causes of fundamental legal programs affecting our client community Develop and test solutions to recurring legal problems faced by our client community Ensure appropriate and sustainable web-based access to legal services

15. Slide 15 Innovation Network, Inc. Strategies & Activities The major actions a program takes to achieve desired outcomes Use action words like: Conduct Train Mentor Provide Support

16. Slide 16 Innovation Network, Inc. Some Intake Examples Intake Develop database program that gathers case information and assesses eligibility for legal aid Develop process for integrating systems and databases in various offices that conduct intake, provide legal advice, and represent clients Train case workers to use the system Identify interpreters for mono-lingual clients

17. Slide 17 Innovation Network, Inc. Some Infrastructure Examples Infrastructure Identify potential partners Hold regular meetings with partners and document community needs Develop web-based services Train providers in how to use web-based services

18. Slide 18 Innovation Network, Inc. Some Pro Se Examples Pro Se Prepare training materials for template developers Participate in national template developers training Prepare logistics for Ohio template developers training Conduct Ohio template developers training Organize multi-state template developer work group

19. Slide 19 Innovation Network, Inc. Questions

20. Slide 20 Innovation Network, Inc. Work on Strategies and Activities Take a look at your grant proposal or work from memory and complete the first two columns of the LSC Evaluation Plan grid. List your activities and organized by category or strategy. Remember to use appropriate action words. When you’re done, click in the check box next to you name

21. Slide 21 Innovation Network, Inc. Check in / Debrief / Discussion

22. Slide 22 Innovation Network, Inc. Evaluation Questions Identify broad questions that will guide your evaluation report. Process questions: What did you do and what were the tangible results of your work? Outcome questions: What change occurred among your constituents as a result of your activities? Try to keep the questions open-ended

23. Slide 23 Innovation Network, Inc. Some Intake Examples Intake How has the intake system assisted clients in accessing seamless services? To what extent has the new system increased efficiency among intake staff?

24. Slide 24 Innovation Network, Inc. Some Infrastructure Examples Infrastructure To what extent has the website helped strengthen client education? How have the partnerships supported the development of quality content for the website? How has the website increased access to legal services for rural clients? In what ways has this project been supported by LSC and the organizations partners?

25. Slide 25 Innovation Network, Inc. Some Pro Se Examples Pro Se To what extent did the training provide necessary skills and knowledge to achieve intended outcomes? Was the training curriculum and delivery offered of high quality? How well have the online services met the needs of the community and addressed the current trends in necessary legal assistance? Has our online process and materials adequately prepared clients to accurately complete their legal forms?

26. Slide 26 Innovation Network, Inc. What are Your Questions? What information do you want to know about your program? Think about the assumptions that you have made about how your program will be planned and implemented. Draft questions that you want to answer about your program. Tip: Avoid Yes/No questions. Instead, ask when, how, how much or to what extent

27. Slide 27 Innovation Network, Inc. Check in / Debrief / Discussion

28. Slide 28 Innovation Network, Inc. Evaluation Data Talk to people Interview, focus group, facilitated discussion Gather written responses Surveys, questionnaires, paper tests Observe Individuals, communities, group activities Review documents Attendance records, progress reports, journals

29. Slide 29 Innovation Network, Inc. ….and the source Where will you find the information? What target group or documents will you need to gather the necessary information?

30. Slide 30 Innovation Network, Inc. Some Intake Examples Intake Survey of intake staff Focus Group with mono-lingual clients Exit survey of mono-lingual clients

31. Slide 31 Innovation Network, Inc. Some Infrastructure Examples Infrastructure Monthly update form completed by project staff Survey of rural clients/users Survey of case workers Content review of website

32. Slide 32 Innovation Network, Inc. Some Pro Se Examples Pro Se Survey clients Document review/Content review of online process

33. Slide 33 Innovation Network, Inc. Finish Your LSC Evaluation Plan List the data collection methods and source of information for each one of your evaluation questions Remember to keep in mind the following questions: Who or what has the answer to each evaluation question (partners, clients, records, etc What is the best way to access it (survey, focus group, document review, etc) Tip: Think about what information you are already gathering and try to integrate those tools into this data collection effort.

34. Slide 34 Innovation Network, Inc. Evaluation Plan Project Goal(s) Strategies/ Activities Evaluation Questions Evaluation Data

35. Slide 35 Innovation Network, Inc. Check in / Debrief / Discussion

36. Slide 36 Innovation Network, Inc. Questions

37. Slide 37 Innovation Network, Inc. Next Steps Let us know where you are in the process Have you completed your plan? Do you want us to review it? Are you ready to create data collection instruments?

38. Slide 38 Innovation Network, Inc. We can provide more assistance Asynchronous training Individual technical assistance Synchronous training Potential topics include: Survey Document review

39. Slide 39 Innovation Network, Inc. For More Information Innovation Network, Inc. 1001 Connecticut Ave., NW, Suite 900 Washington, DC 20036 (202) 728-0727 www.innonet.org