1. Building Standards for Health Policy Evaluation in an Academic Medical Setting Center for Health Policy and Research University of Massachusetts Medical School Teresa Anderson, PhD Judith Savageau, MPH November 3, AEA Annual Meeting Anaheim, CA

2. Presenter Disclosure
We have no conflicts of interest to disclose….

3. Learning Objectives
Participants will describe the 8 internal assessment areas of the “Guidance on Evaluation Reports to the Robert Wood Johnson Foundation”
Participants will use the Evaluation Criteria checklist inclusive of the 8 areas to review two samples of qualitative and quantitative evaluation reports and to develop comments for the authors
Participants will describe how the RWJ tool supports the American Evaluation Association’s Guiding Principles for Evaluators

4. Workshop Timeline Introductions: 5 minutes Meet the Reports: 5 minutes
RWJF Checklist – review assessment areas/elements checklist and how to use it to evaluate different types of reports and report sections: 15 minutes
Small groups review and rate a qualitative or quantitative report per the RWJF checklist: 20 minutes
Small groups report out their ratings: 20 minutes
AEA Guiding Principles for Evaluators: review criteria and how to use them to evaluate different types of reports: 20 minutes
Conclusion: 5 minutes

5. Qualitative and Quantitative Reports for Group Reviews/Rating
Medicaid Reimbursement for Screening and Brief Intervention: Massachusetts’ Preparations; assessment of 10 states’ experience with adopting a benefit; study conducted for MassHealth Office of Behavioral Health.
Recruitment and Retention of Primary Care Providers at Community Health Centers: A Survey of Massachusetts Physicians; evaluation of current workforce to understand current challenges to recruit and retain PCPs in CHCs post MA Health Care Reform; study conducted in collaboration with MA Dept. of Public Health and MA League of Community Health Centers
NOTE: Neither of these reports was subject to an internal review using either the RWJF or AEA criteria at the time of their writing or distribution! 5

6. Guidance on Evaluation Reports to RWJ
Executive Summary
Introduction
Methodology/Design
Data Collection/Analysis
Results/Conclusions
Recommendations
General
Overall Comments

7. RWJF Evaluation Criteria Checklist and AEA Guiding Principles
RWJF Criteria:
Executive Summary
Introduction/Background
Methodology/Design
Data Collection/Analysis
Results/Conclusion
Recommendations
CHPR additional qualitative questions
AEA Guiding Principles:
Systemic Inquiry
Competence
Integrity/Honesty
Respect for People
Responsibilities for General and Public Welfare

8. Small Group Assignments for Report Reviews
Within small group, decide on reading full report vs sections of report per participant
Review document (or section thereof) per the guidelines, assess it based on the checklist item and comment on whether criteria have been met and/or guiding principles can be assessed
Select a group member to report out on behalf of the group or how feedback is to be provided
Compare use of these criteria and guiding principles to reviewing qualitative vs quantitative evaluation reports
Provide a summary statement of the extent to which the reports need re-organization or re-writing to meet standards for peer review in an academic setting

9. Select References American Evaluation Association
Robert Wood Johnson Foundation
Commonwealth Medicine