1. Expanding the evaluation lens: How to include policy assessment in your evaluation Kari Greene, MPH Program Design & Evaluation Services Oregon Public Health Division American Evaluation Association 2014 Annual Conference Denver, CO

2. Overview Define terms 3 step process 1.Operationalize gold standard 2.Code policies 3.Make sense of coding Practice coding Lessons learned Resources & examples

4. DevelopAdoptImplementEnforceRefine

6. Three-Step Policy Assessment Identify & operationalize gold standard Code policies Make meaning of policy scores

7. Step 1: Gold standard Identify a gold standard Operationalize key elements Develop standardized score sheet or codebook

8. Step 2: Code policies Systematically collect policies Train coding team Code policies using standardized score sheet Assess reliability Refine coding

9. Step 3: Make meaning of coding Identify comparison group Weight vs. overall sum of scores Generate scores for each policy Mandatory vs. “bonus” items Establish classification system

10. Coding Exercise! Code example anti-bullying policy Compare wording in the policy to the components required in Oregon Revised Statutes (ORS) found on the score sheet Each item on score sheet given a 0, 1 or 2 0 =Element missing in policy documentation 1 =Not fully compliant with statute: Requires documentation in Notes field 2 =All aspects found in policy: Fully compliant Can do on your own or in pairs After coding, we’ll compare coding

11. Lessons Learned Interdisciplinary project team ideal –Coding best done by attorneys or law students Give ample time for training, coding & cleaning data Assessing cross-sectional policy adoption insufficient for outcomes-based evaluation –Need data on policy implementation & enforcement –Policies adopted then refined & updated over time

12. Examples & Resources Handout of 3-step process Well-SAT (Wellness School Assessment Tool) –www.wellsat.orgwww.wellsat.org Rural Active Living Assessment (RALA) Policy tool –http://activelivingresearch.org/rural-active- living-assessment-rala-toolshttp://activelivingresearch.org/rural-active- living-assessment-rala-tools Public Health Law Research Law Atlas –http://lawatlas.org/welcomehttp://lawatlas.org/welcome

13. Get in touch! Kari Greene kari.greene@state.or.us 971-673-0599