1. Evaluation of Health Promotion CS 652 Sarah N. Keller

2. Overview Terms Methods Approaches Future Research

3. Terms Formative evaluation Needs assessment; baseline data collection; audience research; results feed into message/campaign design Process evaluation Monitoring; tracking progress & activities; short- term evaluation; in-process impact Impact evaluation Outcomes evaluation; end-user changes in KAP

4. Methods Focus groups Develop clearer understanding of perceptions, values, tastes, psychosocial factors N = 8-12 Data = qualitative data recorded & transcribed Procedure = group interview, moderated by a single facilitator &/or note-taker Analysis = qualitative analysis for key phrases, research questions, language, etc. Purpose = formative research; message design; pre- testing; monitoring research; audience reactions & feedback

5. Methods Surveys Written, telephone, in-person, or online Questions = closed-end or open-end Data = quantitative & qualitative data N = 200 (informal market sample) – 1,500 (minimum for national representation of any subgroup) Analysis – Bivariate correlations; cross-sectional; multivariate regression Purposes = compare baseline to outcome indicators; pre-/post-test comparison; descriptive information; correlations of two or more variables with each other

6. Methods In-depth Interviews Process = face-to-face; one-on-one; structured or unstructured Data = qualitative, recorded & transcribed Analysis = qualitative Purpose = to uncover personal, sensitive or in-depth information; psychosocial factors, anecdotes, personal histories

7. Online Methods Online Surveys Samples = Convenience sample of who comes to a site; Internet users at large, or of a specific demographic of health risk category Experimental design = can direct one group to look at a Web site, and not others (treatment and control) Process = can create online forms attached to Web sites that automatically spit data into a database Data = quantitative or qualitative Analysis = same as with other surveys

8. Online Methods Usability studies Purpose = Find out how users search for a topic or interface with a site; to revise or pre-test a Web site. Process = Convene a group of users in a PC room & ask them to conduct “scavenger hunts” Data = audio, video, observation notes, log data, etc.

9. Online Methods Online Focus Groups Can be conducted in chat rooms, email, or by recruiting volunteers via a Web site form Purpose = Assess users’ opinions of a topic, Web site, or health area; etc. Data = Qualitative Limits = Invasion of privacy, reliability of self-reports, and informed consent process

10. Approaches: CPP Evaluating Community Planning Define the evaluation goals Document that CPP has taken place Document whether program goals are being met Identify strengths & weaknesses Holtgrave, D.R., et al. (1996). Methodological issues in evaluating HIV Prevention Community Planning. Public Health Reports, 111 (suppl. 4).

11. Approaches: CPP Draw a Logic Model of CPP Process Logic model = Graphic representation of an intervention; purpose is to show logical connections between conditions, activities & outcomes. Conditions/problems = What CPP designed to change Activities = Components of CPP undertaken Outcomes = Short-term program goals

12. Approaches: CPP Develop Evaluation Plan Goal #1 (Process objective) – To document that the activities have taken place as planned. Goal #2 (Outcomes) – To document whether short- term program goals are being met. Specific, measurable outcomes derived from general goals. Goal #3 – (Lessons learned) Identify strengths & weaknesses (e.g., an objective might not be reasonable, or more technical assistance may be required).

13. Approaches: CPP Translating plan into action Core objectives – National assessment of community planning does not require data on all process & outcome variables that might be useful at a local level. Budgetary reporting Case studies

14. Example: Hospital Marketing Goals: Determine attributes that physicians consider important in deciding to join hospitals Determine which factors most affect physicians’ evaluation of hospital quality Cronin, JJ., Joyce, M.L. (1987). Medical staff perceptions: Implications for the design of hospital marketing programs. JHCM, 7 (3).

15. Example: Hospital Marketing Lit Review Shows physicians are important influential agents in determining consumer hospital selection Methods Identify research variables Hospital performance factors Attributes that define hospital’s “offering” to physicians

16. Approaches: Hospitals Methods Focus group (n=8) physicians to discuss “What actually defines what a hospital has to offer to you in order to attract you to use facility?” AND, “What determines your evaluation of a particular hospital?” Analysis Attributes categorized into 11 indicators Evaluative dimensions categorized into 4 groups Used for survey design

17. Approaches: Hospitals Methods Mail survey (n=169) – of physicians responded Setting – 4 hospitals in Florida 3-part instrument Questions to capture main Research Q’s Demographics

18. Approaches: Hospitals Instrument Respondents asked to recall decision to join hospital and rate importance of each of 11 attributes on a scale from 1-5. Respondents asked to indicate perception of performance of area’s 4 hospitals based on 11 attributes Respondents asked to evaluate 4 hospitals using 4 evaluative dimensions from focus groups

19. Example: Hospitals Quantitative data analysis Means – Mean response for each of 11 attributes Correlation – Relationship b/t physicians’ perception of hospitals performance (using 11-attributes as IVs) & their evaluation on dimensions (4 dimensions as DVs). Multiple regression See which factors (clusters of attributes) explained most variance in perceptions (evaluation of hospitals)

20. Limitations Research settings & samples need to be expanded for generalizability Privacy, informed consent an issue Cost of research Process vs. outcome Use of theory