1. How to Evaluate Your Health Literacy Project Jill Lucht, MS Project Director, Center for Health Policy luchtj@missouri.eduluchtj@missouri.edu573-882-2300 Presentation developed with Elizabeth Pafford, Institute of Public Policy, University of Missouri 573.882.5835 pafforde@missouri.edupafforde@missouri.edu

2. University of Wisconsin-Extension, Program Development and Evaluation Everyday example HEADACHEHEADACHE Feel betterGet pillsTake pills Situation INPUTS OUTPUTS OUTCOMES

3. Program Logic Model

4. Inputs Human and material resources that support the program Staff Volunteers Time Money Research base Materials Equipment Technology Partners

5. Outputs Direct products or activities- what you will do to meet your objectives Conduct trainings, workshops, meetings Deliver services Develop products, curriculum, resources Partner/Collaborate Reach target population Target population satisfaction

6. Outcomes Changes or benefits as a result of outputs Short term: Change in… Awareness Knowledge Attitudes Beliefs Skills Opinions Aspirations Motivations Intermediate: Change in… Action Behavior Practice Policies

7. Outcomes Changes or benefits as a result of outputs Long Term (Impact): Change in… Conditions Environment Institutionalized policy

8. Assumptions Beliefs about how the activities correspond with the outcomes- how the program will work and why. Assumptions underlie much of what we do. It is often these underlying assumptions that hinder success or produce less- than-expected results. One benefit of logic modeling is that it helps us make our assumptions explicit.

9. External Factors Biophysical environment Economic structure Housing patterns Demographic makeup Family circumstances Values Political environment Background and experiences Media Policies Influences outside the program that cannot be controlled but do/could affect project outcomes

10. Why do evaluation? Modify current programs or strategies Identify success and challenges Justify spending/resources Prepare grant reports Write future grant applications Identify needs Hold personnel accountable Improve service delivery Enhance record keeping Communicate results to stakeholders Systematic collection, analysis and sharing of information for the purposes of answering specific questions about a program. Program Evaluation

11. Types of Evaluation Process Evaluation: Measuring the inputs, activities and outputs of a program. Fidelity Evaluation: Measuring how program implementation adheres to the prescribed implementation plan. Implementation Evaluation: Describing and assessing program delivery with an eye towards recommending efficiency gains. Outcome Evaluation: Measuring the extent to which the program achieved its desired or intended outcomes. Impact Evaluation: Measuring the long-term outcomes or community level impact experienced by the target population as a result of the program.

12. Evaluating Your Health Literacy Initiative Project Keep it simple! Focus on Process (measuring inputs, activities and outputs of your project) & Outcome (measuring the extent to which your project achieved its desired or intended outcomes) Get the technical assistance you need (just ask Jill!) Use existing tools:* –Logic model –Questionnaire measuring patient/client perception of communication with their health care professional *Only use the tools that make sense for you and your project

13. Program Logic Model

14. 1.How often did this provider give you all the information you wanted about your health? 2.Did this provider encourage you to talk about all your health problems and concerns? 3.How often did this provider ask you to repeat back or describe how you were going to follow the doctor’s instructions? 4.How often did this provider give you easy to understand instructions about how to take your medicines? 5.How often were the results of your blood test, x-ray, or other test easy to understand? *Based on Weidmer, Brach and Hays. 2012. “Development and Evaluation of CAHPS® Survey Items Assessing How Well Healthcare Providers Address Health Literacy.” Medical Care, Vol. 50 (9, Supplement 2): pages S3-S11. 5 Questions* Y ou Can Use to Measure Your Outcomes

15. Thanks! For more information please contact: Jill Lucht Center for Health Policy luchtj@missouri.edu (573) 882-2300 Presentation developed with Elizabeth Pafford, Institute of Public Policy, University of Missouri 573.882.5835 pafforde@missouri.edupafforde@missouri.edu