The Role of Evaluation in Public Health The Big Picture The Role of Evaluation in Public Health

Activity Think of your favorite quote or sage advice that helps guide you through your professional work.

Wise Advice I hear and I forget. I see and I believe. I do and I understand. —Confucius (551–479 BC)

CDC Evaluation Framework Standards Utility Accuracy Propriety Feasibility

Objectives Describe the relationship between performance management, program planning and evaluation Describe the steps of program planning and evaluation process Identify key stakeholders in an evaluation Develop the components and elements of a program logic model Create a basic evaluation plan including data collection and analyzing methods List three ways to use evaluation data to draw conclusions about a program Define ROI, cost benefit ratios and social return on investment. 5

Accreditation Accreditation is a major accomplishment for a health department. It means that it is addressing key community health problems. Just as the public expects hospitals, law enforcement agencies, and schools to be accredited, so should they come to expect public health departments. Thomas Frieden, MD, MPH CDC Director Why are we talking about these manufacturing and business based frameworks in public health? Accreditation Accreditation is a major driver for this work in our agencies and with our partners in public health. You can see from Dr. Friedan’s quote….it is expected…

CDC Grants CDC awards grants for public health performance management systems $42.5 million allocated for 94 projects in state, local, tribal, and territories health departments Distributed in 49 states, 16 tribes and territories and nine local jurisdictions Five-year cooperative agreement awards are being funded by the health care reform law There is also resource support from the national level…with the National Public Health Improvement Initiates (NPHII) CDC The Centers for Disease Control and Prevention provided funding for state and local public health departments improve their performance management, training and collaboration by awarding them $42.5 million in grants.

Benefits of Accreditation Accountability & credibility Tool for improvement Highlights HD strengths Greater collaboration Accreditation + Team building Those who champion accreditation list numerous benefits….with good reason. If we look at what is listed here in the slide all of the bubbles are very practical for public health and our partner to want to achieve to… (list each) Recognition & validation Better understanding of public health

Analysis & Environmental Scan Strategic Planning Phase 4 Evaluation Monitoring Implementation Budgets Action Plans Strategies Phase 3 Strategic Issues Goals & Objectives Phase 2 Vision Mission Values SWOC Analysis & Environmental Scan Plan the Plan Readiness Assessment Stakeholder Assessment Mandate Analysis Phase 1

Performance Management (PM) A systematic process aimed at helping achieve an organization’s mission and strategic goals by improving effectiveness, empowering employees, and streamlining the decision-making process.  Performance management is an ongoing, continuous process of communicating and clarifying job responsibilities, priorities and performance expectations in order to ensure mutual understanding between supervisor and employee. Quality improvement in public health is the use of a deliberate and defined improvement process, such as Plan-Do-Check-Act, which is focused on activities that are responsive to community needs and improving population health. It refers to a continuous and ongoing effort to achieve measurable improvements in the efficiency, effectiveness, performance, accountability, outcomes, and other indicators of quality in services or processes which achieve equity and improve the health of the community. (R. Bialek, L. M. Beitsch, A. Cofsky, et al, unpublished data, 2009) Performance = + Behavior Results

They Are Not the Same Quality assurance Quality improvement Reactive Proactive Works on problems after they occur Works on processes Regulatory usually by state or federal law Seeks to improve (culture shift) Led by management Led by staff Periodic look-back Continuous Responds to a mandate or crisis or fixed schedule Proactively selects a process to improve Meets a standard (pass/fail) Exceeds expectations QA is reactive, going back to check on how things had been done. QI is proactive, thinking about ongoing processes in place and how to make measurable improvements. QA often results from regulation, while QI signals an agency’s desire to operate in a culture of always improving Managers typically take the lead in QA, while QI deliberately involves staff at all levels QA is done on a periodic, scheduled basis, while QI is an ongoing effort QA operates on a pass/fail basis – either something was done correctly or not, while QI is all about exceeding expectations – always setting the bar higher

They Are Not the Same Evaluation Quality improvement Assess a program at a moment in time Understand the process that is in place Static Ongoing Doesn’t include identification of the source of a problem or potential solutions Entails finding the root cause of a problem and interventions targeted to address it Doesn’t measure improvements Focused on making measurable improvements Program-focused Customer-focused A step in the QI process Includes evaluation QA is reactive, going back to check on how things had been done. QI is proactive, thinking about ongoing processes in place and how to make measurable improvements. QA often results from regulation, while QI signals an agency’s desire to operate in a culture of always improving Managers typically take the lead in QA, while QI deliberately involves staff at all levels QA is done on a periodic, scheduled basis, while QI is an ongoing effort QA operates on a pass/fail basis – either something was done correctly or not, while QI is all about exceeding expectations – always setting the bar higher

Value of Performance Management Disciplined approach Systemic Systematic Engaged customers and staff Results focused Proven method So why performance management frameworks – what is the value added Well what we know from manufacturing and industry It is a very disciplined approach It is looking at improving systems It is systematic (or step by step) It engages stakeholders (so partners, staff, customers…) It is very results focused (not so much a count on what we did….rather what happened as a result? Been around a while and is a proven methodology 13

Culture of QI Topic Organization-wide Program/unit Improvement System focus Specific project focus QI planning Tied to the strategic plan Program/unit level QI goals Strategic plan Individual program/unit level plans Approaches Baldrige Program Organization QI Council Lean Six Sigma Individual QI Teams Rapid Cycle PDCA It can be useful to think of QI work across a spectrum. At one end of the spectrum is organization-wide QI, or having a culture of QI throughout the health department. At the other end of the spectrum is engaging in QI in a specific program, or even project – just beginning to apply the QI process somewhere in the health department. A culture of QI is highly desirable, although it’s recognized that it’s important to start at the other end of the spectrum. And it’s fine to start small – in fact, it’s desirable to start small. (Describe where your department is on the spectrum, and if appropriate, describe how you would like the department to progress towards an organization-wide approach to QI, or having a “QI culture”)

Performance Management Frameworks Baldrige Six Sigma Lean Performance management frameworks can begin to start to build this system. There are many frameworks that you may have hear about through the years—today I will briefly touch three: Baldrige, Six Sigma, Lean

PDSA Cycle What are we trying to accomplish? How will we know that a change is an improvement? What changes can we make that will result in improvement? Act Study Plan Do Example of performance measurement systems Scorecard Dashboard

Memory Jogger

Tools Logic models and work flow charts Data and analysis tools Customer-supplier relationships Client flow, information flow Root cause tools: fishbone diagram, Pareto chart Force field analysis Interrelationship digraph Logic models provide a graphic illustration of your program plan Flowcharts allow a team to visualize actual flow or sequence of events Pareto’s Principle or Pareto’s Law – 80/20 Rule 20% of something is always responsible for 80% of the results.

PERFORMANCE MANAGEMENT SYSTEM PERFORMANCE STANDARDS Identify relevant standards Select indicators Set goals & targets Communicate expectations PERFORMANCE MEASUREMENT Refine indicators & define measures Develop data systems Collect data PERFORMANCE MANAGEMENT SYSTEM REPORTING OF PROGRESS Analyze data Feed data back to managers, staff, policy makers, constituents Develop a regular reporting cycle QI PROCESS to improve policies, programs, & outcomes Manage changes Create learning organization QI PROCESS Use data for decisions Performance Standards - establishment of organizational or system performance standards, targets and goals and relevant indicators to improve public health practice Performance Measures - application and use of performance indicators and measures Quality Improvement - establishment of a program or process to manage change and achieve quality improvement in public health policies, programs or infrastructure based on performance standards, measurements and reports. Reporting of Progress - documentation and reporting of progress in meeting standards and targets and sharing of such information through feedback Turning Point Performance Management Collaborative, 2003

Questions?