IHS Special Diabetes Program Competitive Grants Part 3: Using Idea Maps to Plan for Evaluation Cardiovascular Disease Prevention Planning Cynthia C. Phillips, Ph.D. Lisa Wyatt Knowlton, Ed.D. Evaluation Training ̶ January 2005

Part 3: Overview  Developing Evaluation Questions  Matching Methods to Questions  Selecting Indicators  “Hands-on” Evaluation Planning Exercise

Connecting Idea Maps to Evaluation

Evaluation Design IM--Questions Example: Family Vacation

Fictional CVD Project IM CVD Risk Reduction Program Idea Map * optional, for demonstration only Outcomes Inputs/Resources Strategy/Activities* Outputs Short Term Long Term Impact Individual Activities Community Activities Case Mgrs/ Counselors Communi- cation Resources Clinical Services Social Marketing Skills Prior Health Ed Experience Diabetes Burden Tip Sheets CVD CVD Seminar Exercise Plan CVD PSAs Lifestyle Coaching Case Mgmt Nutrition Education XX% have tips XX Patients in Lifestyle Coaching for XX weeks XX Patients in Nutrition Ed for XX weeks XXXX Contacts Established Better Health Status - Improved Awareness re CVD -Increased Physical Activity -Reduced Blood Pressure -New Attitudes -New Skills Participant Prospects XX Patients in Exercise for XX weeks XX Patients in Case Mgmt for XX weeks Recruit & Select People w Diabetes Reduced CVD Incidence Knowledge re Effective Health Programming Is the project delivering the program as intended? Are project participants making progress toward uniform core outcomes?

Question Examples AudienceTypical QuestionsEvaluation Use Program Management and Staff Are we reaching our target population? Are our participants satisfied with our program? Is the program being run efficiently? How can we improve our program? Programming decisions, day- today operations ParticipantsDid the program help me and people like me? What would improve the program next time? Decisions about continuing participation. Community Members Is the program suited to our community needs? What is the program really accomplishing? Decisions about participation and support. Public OfficialsWho is the program serving? What difference has the program made? Is the program reaching its target population? What do participants think about the program? Is the program worth the cost? Decisions about commitment and support. Knowledge about the utility and feasibility of the program approach. FundersIs what was promised being achieved? Is the program working? Is the program worth the cost? Accountability and improvement of future grantmaking efforts.

Indicators  The specific statements you develop as markers of “success” for activities and outcomes  SMART across each part of the idea map  Meaningful and manageable  May require technical assistance to develop data collection tools

Focus AreaIndicatorsHow to Evaluate[1][1] Influential FactorsMeasures of influential factors--may require general population surveys and/or comparison with national data sets[2].[2] Compare the nature and extent of influences before (baseline) and after the program. ResourcesLogs or reports of financial/staffing status.Compare actual resources acquired against anticipated. ActivitiesDescriptions of planned activities. Logs or reports of actual activities. Descriptions of participants. Compare actual activities provided, types of participants reached against what was proposed. OutputsLogs or reports of actual activities. Actual products delivered. Compare the quality and quantity of actual delivery against expected. Outcomes & Impacts Participant attitudes, knowledge, skills, intentions, and/or behaviors thought to result from your activities[3].[3] Changes in health status factors specified for the program (uniform core) Compare the measures before and after the program[4].[4] Indicator Examples

SAMPLE QuestionOutcomeIndicatorsTechnical Assistance Needed What progress are participants making toward uniform core outcomes? Improved CVD Awareness and Knowledge Knowledge of CVD and Prevention Survey Development, Administration, Analysis Increased Physical Activity Exercise JournalsDatabase Development Data Entry and Analysis Improved Diet/Nutrition Food Diary Improved LifestyleSmoking Cessation Better Health Status Reduced Blood Pressure Reduced Weight, BMI Improved Lipid Panel

Evaluation Planning Exercise

Exercise Steps  In your project team  Identify your key questions about recruitment from your idea maps  Identify the range of possible indicators for successful recruitment activities  At your table  Determine common and unique approaches  Discuss appropriate indicators for activities  Whole group  Are there activities we all ought to do?  Are there indicators we all ought to use?