1. Jennifer S. Funderburk, PhD
Christina Studts, PhD
William Lusenhop, PhD
Mary Peterson, PhD
Jennifer Wray, PhD
Session A8, #
Friday, 10:30-11:30am, October 14, 2016
Let’s Talk! Identifying Ways to Improve the Quality and Generalizability of Program Evaluation or Quality Improvement Data
Please insert the assigned session number (track letter, period number), i.e., A2a
Please insert the TITLE of your presentation.
List EACH PRESENTER who will ATTEND the CFHA Conference to make this presentation.
You may acknowledge other authors who are not attending the Conference in subsequent slides.
CFHA 18th Annual Conference
October 13-15, 2016  Charlotte, NC U.S.A.
Collaborative Family Healthcare Association 12th Annual Conference

2. Faculty Disclosure
The presenters of this session have NOT had any relevant financial relationships during the past 12 months.
You must include ONE of the statements above for this session.
CFHA requires that your presentation be FREE FROM COMMERCIAL BIAS. Educational materials that are a part of a continuing education activity such as slides, abstracts and handouts CANNOT contain any advertising or product‐group message.
The content or format of a continuing education activity or its related materials must promote improvements or quality in health care and not a specific propriety business interest of a commercial interest. Presentations must give a balanced view of therapeutic options. Use of generic names will contribute to this impartiality. If the educational material or content includes trade names, where available trade names for products of multiple commercial entities should be used, not just trade names from a single commercial entity. Faculty must be responsible for the scientific integrity of their presentations. Any information regarding commercial products/services must be based on scientific (evidence‐based) methods generally accepted by the medical community.
Collaborative Family Healthcare Association 12th Annual Conference

3. Learning Objectives At the conclusion of this session, the participant will be able to:
Describe how to determine the quality and rigor of a quality improvement/program evaluation project 
Identify general strategies for improving the quality and generalizability of quality improvement/program evaluation efforts 
Describe how you can improve the quality and/or generalizability of a project that you are planning or currently implementing 
Include the behavioral learning objectives you identified for this session
Collaborative Family Healthcare Association 12th Annual Conference

4. Bibliography / Reference
Fitzpatrick, J. L., Sanders, J. R., & Worthen, B. R. (2015). Program evaluation: Alternative approaches and practical guidelines. Upper Saddle River, N.J: Pearson Education. 
Green, L.W. & Glasgow, R.E. (2006). Evaluating the relevance, generalization, and applicability of research issues in external validation and translation methodology. Evaluation and the health professions, (29),  
Glasgow, R.E. & Emmons, K.M. (2007). How can we increase translation of research into practice? Types of evidence needed. Annual Review of Public Health, 28,  
Posavac, E.J. (2015). Program evaluation: Methods and case studies. New York; NY: Routledge 
Bridget Gaglio, Jo Ann Shoup, and Russell E. Glasgow. The RE-AIM Framework: A Systematic Review of Use Over Time. American Journal of Public Health: June 2013, Vol. 103, No. 6, pp. e38-e46. 
Continuing education approval now requires that each presentation include five references within the last 5 years.
Please list at least FIVE (5) references for this presentation that are no older than 5 years.
Without these references, your session may NOT be approved for CE credit.
Collaborative Family Healthcare Association 12th Annual Conference

5. Learning Assessment A learning assessment is required for CE credit.
A question and answer period will be conducted at the end of this presentation.
Please incorporate audience interaction through a brief Question & Answer period during or at the conclusion of your presentation.
This component MUST be done in lieu of a written pre- or post-test based on your learning objectives to satisfy accreditation requirements.
Collaborative Family Healthcare Association 12th Annual Conference

6. Overview
Conducting quality research and program evaluation in primary care settings (20 mins)
Four groups (35 mins)
I need help with strategic planning!
I have need help with the methodology/design!
How can I measure!
I just need help (I have no idea where to start)!
Large group discussion, summary, and resources (5 min)

7. Background: Program Evaluation
New programs are continually being developed to improve the quality of care.
It is important to understand the impact of these programs within real- world settings.
Program Evaluation (PE)
Critical tool by which clinicians/administrators can determine whether services are meeting their intended objectives.
Designing systematic, scientifically rigorous program evaluations (PE) is one way to contribute to the significant need to build best practices and a stronger evidence-base for integrated healthcare.
Collaborative Family Healthcare Association 12th Annual Conference

8. Program Evaluation in Integrated Healthcare
PE is useful in examining
An Example
Extent a program is meeting intended objectives
Program strengths/weaknesses
Best ways to refine and improve performance
Best approaches for implementing and sustaining a program
Whether a program is worth investment of time, effort, and money
Examine the feasibility and reactions of BHPs to receiving training for a brief alcohol intervention (BAI) via a webinar
PC examples:
-Satisfaction with the services provided by integrated behavioral health providers (BHPs) (Funderburk et al., 2012)
-Whether meeting with a BHP results in clinical improvement (Sadock, et al., 2014)
-Feasibility and reactions of BHP’s receiving a virtual training for a brief alcohol intervention.
-Assess the reach, effectiveness, adoption, implementation, and maintenance of a depression monitoring service.
Collaborative Family Healthcare Association 12th Annual Conference

9. Well-designed program evaluation
Strategic planning
Methodology and design considerations
Measurement considerations
Maintaining consistency
Relying on researchers in academic settings to answer lingering questions is impractical not only because research funding is limited, but because it is important to understand the impact of these programs within real-world settings.
Collaborative Family Healthcare Association 12th Annual Conference

10. Tips for achieving a well-designed program evaluation: Strategic planning
Examine/define the “program”
Use a conceptual framework or the underlying logic behind the program and what it is supposed to be doing (e.g., Kellogg Foundation logic model, RE-AIM)
Consider purpose and audience of the PE
Identify key question
We looked at the satisfaction with the webinar and the knowledge learned, but we missed the opportunity to assess outcomes that may be relevant, if we had used this model—comfort and confidence when discussing this topic with patients
Collaborative Family Healthcare Association 12th Annual Conference

11. Tips for achieving a well-designed program evaluation: Methodology & design
BAI Example
Having a comparison can increase confidence about effects of the program
Include another condition: No-treatment control, waitlist, or usual care
Compare to changes typically observed within the field
Random assignment may not be feasible
Block or cluster randomization (at clinic or provider level)
Quasi-experimental study design
Nonequivalent group design
Multiple baseline design
Regression-discontinuity design
Did not assess the same outcomes pre- and post-training limiting our ability to draw conclusions regarding change
Diverse sample of providers
Qualitative portion: Open ended questions to get qualitative feedback about components of the webinar that BHPs liked/did not like
Study design can significantly enhance the potential conclusions
Another alternative is quasi-experimental pre/post designs. For example, a nonequivalent group design has groups assigned to conditions based on nature (those who received the program and those who did not) and comparisons based on information collected at two time points (Bryk & Weisberg, 1977). The multiple baseline design is another methodologically rigorous alternative to RCTs with several practical advantages, such as the ability to focus on one practice and collect comparison data from that same practice (see Hawkins, Sanson-Fisher, Shakeshaft, D’Este, & Green, 2007 for details). A regression-discontinuity design may also be an option for PE within integrated settings. This design allows patients who score higher on a specific cutoff (e.g., Patient Health Questionnaire-9 [PHQ-9] ≥ 10; Kroenke, Spitzer, & Williams, 2001) to be assigned to the program and those who score under the cutoff (PHQ-9 < 10) to be assigned to the comparison condition (Thistlethwaite & Campbell, 1960).
Collaborative Family Healthcare Association 12th Annual Conference

12. Tips for achieving a well-designed program evaluation: Measurement
Method of data collection
What type of information to collect and how?
Consider strengths and weaknesses of each approach
Feasibility
Quality of measurement
Consider reliability, validity, and meaningful change
Example: PHQ-9
Supplementing self-report data with objective data
Collaborative Family Healthcare Association 12th Annual Conference

13. Tips for achieving a well-designed program evaluation: Maintaining consistency in the Program
Training is key
Orient clinic staff to the measures
Ensure program is being delivered as intended
Training materials should be overly detailed to ensure that training is delivered consistently
Monitor program adherence over time
Audiotapes or videotapes, fidelity checklists, or direct observations
Pick your favorite pic
Collaborative Family Healthcare Association 12th Annual Conference

14. Applying these tips to your PE
Break into four groups:
I need help with strategic planning!
I have need help with the methodology/design!
How can I measure!
I just need help (I have no idea where to start)!
Collaborative Family Healthcare Association 12th Annual Conference

15. Thank you! Session Evaluation
Please complete and return the evaluation form to the classroom monitor before leaving this session.
Thank you!
This should be the last slide of your presentation
Collaborative Family Healthcare Association 12th Annual Conference