1. Evaluation of State Oral Health Plans Paul W. Mattessich, Ph.D.

2. State Oral Health Plans A comprehensive, integrated approach to meeting the oral health needs of a state’s population. Specific, measurable, time-phased objectives

3. Importance of Planning “If you don’t know where you are going, any road will take you there.” “Failing to plan is planning to fail.”

4. Oral Health Plan Components Oral health infrastructure Healthy People 2010 objectives Water fluoridation School based/linked sealant programs Strategies to identify best practices Evaluation strategies And others

5. Why evaluate state plans? Promote effective development and implementation Can determine what worked and did not work – then make adjustments (in current phase of work, or with next plan) Enables states to learn from one another

6. Categories of Evaluation Questions 1.Process: Do plan development activities occur in a way that will nurture engagement of relevant stakeholders for implementation of the plan? 2.Content: Does the plan contain the necessary elements?

7. 3. Dissemination: Do distribution activities bring the plan to all relevant stakeholders, decision-makers? 4. Awareness: Do relevant audiences recognize and reference the plan? 5. Implementation: Do those supposed to take action actually take action?

8. Logic Model Inputs: Data on Oral Health Conditions, etc. Activities: Production and Dissemination of Plan – writing, approvals, presentations, etc. Outputs: Decisions made; policies change; resource allocations change

9. Logic Model Initial Outcomes: New and/or increased services Evaluation of state plan goes this far Intermediate Outcomes: Initial changes in oral health care delivery; reduction in untreated conditions; etc. Ultimate Outcomes: Changes in the health of the population

10. Questions about Process Were the appropriate stakeholders involved in the development of the state plan? Were the stakeholders satisfied with their level of involvement? Do the involved stakeholders feel “ownership” of the plan?

11. Measurement Participation records Surveys (participants and non- participants)

12. Questions about Content Does the plan include sufficient data describing the oral health of the state’s population? To what extent does the plan address the Healthy People 2010 Oral Health Objectives? Does the plan include the necessary components and sufficiently cover the necessary aspects of oral public health?

13. Does the plan reflect the current state of the art and science in oral public health? Is it science based? Are goals and objectives stated in SMART (or similar) format? Does the plan incorporate national goals as outlined in the CDC/DOH Oral Health Index?

14. Measurement Comparison, by objective individuals, of plan to template or guidelines, e.g. (see report): CDC Oral Health State Plan Index State Plan Index

15. Questions about Dissemination Has an intentional strategy for dissemination of the plan been developed? Has a strategy been implemented?

16. Measurement Recording dissemination activities

17. Questions about Awareness Do those who should know about the plan actually know about it? Do those who should know about the plan (or parts of it) actually understand what it says? Do awareness and understanding remain over time?

18. Measurement Surveys Meetings with legislators, key leaders

19. Questions about Implementation Is the plan being used to guide activities and shape thinking? Is it used to guide decisions? Do decision-makers and agencies reference the plan in their documents? Have significant stakeholders endorsed the plan?

20. Have policies changed to conform to the plan? Has the allocation of resources changed to conform to the plan? Have agencies/organizations identified by the plan for specific roles or actions carried out their assignments?

21. Measurement Annual inventory of references (in documents and/or news articles) Surveys Annual checklist for completion of activities specified in plan

22. EXAMPLE OF TIMELINE – Plan Development & Evaluation July – OctoberNovember – March April – July +July + Fact-finding, state conference Writing, discussion of drafts Formal release, distribution, presentations Implementation Participation records Survey Participation records Survey State Plan Index Recording of dissemination activities Annual inventory of references Annual checklist Surveys