National Network of Libraries of Medicine Outreach Evaluation Resource Center Planning Outcomes-Based Outreach Programs
You got to be very careful if you don’t know where you’re going because you might not get there -- Yogi Berra
Why talk about outcomes in the planning phase? What gets measured gets done If you can’t recognize failure, you can’t correct it If you cannot see success, you can’t reward it (and may be rewarding failure) If you do not measure results, you cannot see success If you can demonstrate results, you can win public support From Reinventing Government
A plan helps you move in the right direction. Evaluation keeps your eyes on the road
By the end of this workshop you will know how to Use logic models to plan programs around desired results (outcomes) Shape objectives from outcomes Develop evaluation plans based on logic models
The Logic Model Creating the map
Begin with the results you’re seeking
Figure out what you have to do
Then identify what you need
Activities are more under your control than outcomes
Here are examples of the items for each column
Goal: Have a good summer vacation
Vacation: Output versus Outcome
Library Project: Output Focus
Library Project: Outcome Focus
There are two general categories of outcomes
Examples of system-level change
People change in stages
Kirkpatrick’s Model of Program Evaluation is popular for evaluating training programs
Kirkpatrick’s Model can be useful in completing the Logic Model
Last step: Get Feedback from stakeholders and key informants
How could the logic model help plan proposals? Logic ModelProposal InputsBudget ActivitiesStrategies OutputsDeliverables OutcomesResults & Evaluation InfluencesSupport & Barriers AssumptionsReviewers’ questions
Use the logic model to write final reports Logic ModelReport InputsResources you acquired/used ActivitiesWhat you did OutputsDeliverables Outcomes What you accomplished Influences Lessons learned Assumptions Lessons learned
Making outcomes measurable: Writing objectives
Build objectives around your outcomes
Indicators are the observable signs of the outcome Outcome Parents will increase their ability to find online health information Indicator (measurable) Parents will report they were able to use MedlinePlus to locate information on at least one health topic that was important to them
The target sets the criteria for success Indicator (measurable) Parents will report being able to use MedlinePlus to locate information on at least one health topic that was important to them Target (criterion for success) 70% of participants
Time frame tells you when you are going to check on the indicator
Now you can write the outcome objective for an outcome Outcome Parents will increase their ability to find online health information Outcome Objective By the end of the training session, 70% of parents will report being able to use MedlinePlus to locate information on at least one health topic that was important to them
Example Objective: Awareness Outcome The OERC will improve access to information about emerging evaluation trends among NN/LM members Outcome Objective By the May 2015, All 8 regional medical libraries will have re-posted OERC blog articles in their Newsletters or blog
Objective Example: Impact Outcome Clinicians will recommend health information to their patients Outcome Objective One month after training, 60% of clinicians will report providing at least one patient with Information about MedlinePlus
Discussion: What indicators could you use? Increased use of health resources Indicator: ____________________ More access to health resources Indicator: ____________________ More cost effectiveness Indicator: ____________________ Increased efficiency Indicator: ____________________
Webinar 4: Adding Evaluation to Your Plan
What your stakeholders want to know Did you get there? Did you do everything you planned? How Did you get there? Did you do everything you planned? What unexpected things happened? Would you go there again? What advice do you have for others? What would you do differently next time? What nexpected things happened? Would you go there again? What advice do you have for others? What would you do differently next time?
Focus on “Rapid” methods Short surveys Site visits Observations Key informant and focus group interviews Existing data review
Use mixed methods Quantitative methods for numbers Qualitative methods for text or images
Source and Method Where we find dataHow we collect data
Process evaluation: Keeping your eyes on the road
Process assessment is developed for planned activities
Documenting process
How to describe progress
Add check-ins with key players
Comments and feedback allow you to assess quality of your program
Outcomes Assessment: Did you make it and was it worth it?
Outcomes assessment is guided by the “intended results’ columns
How do you evaluate your outcome objective? By the end of the training session, 70% of parents will report being able to use MedlinePlus to locate information on at least one health topic that was important to them DATA SOURCE Participants EVALUATION METHOD Session Evaluation Form DATA COLLECTION TIMING At the end of each training session
Identify Source, Method, Timing By the May 2015, All 8 regional medical libraries will have re-posted OERC blog articles in their newsletters or blog DATA SOURCE NN/LM websites EVALUATION METHOD Review of blogs and newsletters DATA COLLECTION TIMING At the end of each quarter
Example: ¡VIVA! Outreach Project Objectives: When talking with patients, physicians and physicians assistants will increase use of : Plain language Open-ended questions Teach back Patient health information that matches language and literacy needs MedlinePlus as a patient information source
Example: The STISD Health Literacy Internship Project
Most Significant Change Method
Basic MSC Questions What is the most significant change you have experienced since________ Why was this change so significant to you? What different has it made already? What difference will it make in the future?
NN/LM Outreach Evaluation Resource Center Visit us here: Website: Blog: Facebook: Cindy Olney Acting Assistant Director Karen Vargas Evaluation Specialist