The Big Picture The Role of Evaluation in Public Health Luann D’Ambrosio, MEd

Activity Think of your favorite quote or sage advice that helps guide you through your professional work.

Wise Advice I hear and I forget. I see and I believe. I do and I understand. —Confucius (551–479 BC)

Standards Utility Accuracy Propriety Feasibility CDC Evaluation Framework

Describe the relationship between performance management, program planning and evaluation Describe the steps of program planning and evaluation process Identify key stakeholders in an evaluation Develop the components and elements of a program logic model Create a basic evaluation plan including data collection and analyzing methods List three ways to use evaluation data to draw conclusions about a program Define ROI, cost benefit ratios and social return on investment.Define ROI, cost benefit ratios and social return on investment. Objectives

Accreditation Accreditation is a major accomplishment for a health department. It means that it is addressing key community health problems. Just as the public expects hospitals, law enforcement agencies, and schools to be accredited, so should they come to expect public health departments. Thomas Frieden, MD, MPH CDC Director

CDC Grants CDC awards grants for public health performance management systems $42.5 million allocated for 94 projects in state, local, tribal, and territories health departments Distributed in 49 states, 16 tribes and territories and nine local jurisdictions Five-year cooperative agreement awards are being funded by the health care reform law

Benefits of Accreditation Accountability & credibility Tool for improvement Greater collaboration Recognition & validation Better understanding of public health Team building Highlights HD strengths Accreditation +

Phase 1 Evaluation Monitoring Implementation Budgets Action Plans Strategies Strategic IssuesGoals & Objectives MissionValuesVision SWOC Analysis & Environmental Scan Mandate Analysis Stakeholder Assessment Plan the Plan Readiness Assessment Phase 2 Phase 3 Phase 4 Strategic Planning

Performance Management (PM) A systematic process aimed at helping achieve an organization’s mission and strategic goals by improving effectiveness, empowering employees, and streamlining the decision-making process. Performance Behavior Results = +

They Are Not the Same Quality assuranceQuality improvement ReactiveProactive Works on problems after they occur Works on processes Regulatory usually by state or federal law Seeks to improve (culture shift) Led by managementLed by staff Periodic look-backContinuous Responds to a mandate or crisis or fixed schedule Proactively selects a process to improve Meets a standard (pass/fail)Exceeds expectations

They Are Not the Same EvaluationQuality improvement Assess a program at a moment in time Understand the process that is in place StaticOngoing Doesn’t include identification of the source of a problem or potential solutions Entails finding the root cause of a problem and interventions targeted to address it Doesn’t measure improvements Focused on making measurable improvements Program-focusedCustomer-focused A step in the QI processIncludes evaluation

Value of Performance Management Disciplined approach Systemic Systematic Engaged customers and staff Results focused Proven method

Culture of QI TopicOrganization-wideProgram/unit ImprovementSystem focusSpecific project focus QI planningTied to the strategic plan Program/unit level QI goalsStrategic planIndividual program/unit level plans ApproachesBaldrige Program Organization QI Council Lean Six Sigma Individual QI Teams Rapid Cycle PDCA

Performance Management Frameworks Baldrige Six Sigma Lean

PDSA Cycle What are we trying to accomplish? How will we know that a change is an improvement? What changes can we make that will result in improvement?

Memory Jogger

Tools Logic models and work flow charts Data and analysis tools Customer-supplier relationships Client flow, information flow Root cause tools: fishbone diagram, Pareto chart Force field analysis Interrelationship digraph

QI PROCESS to improve policies, programs, & outcomes Manage changes Create learning organization Turning Point Performance Management Collaborative, 2003 PERFORMANCE MANAGEMENT SYSTEM PERFORMANCE STANDARDS Identify relevant standards Select indicators Set goals & targets Communicate expectations PERFORMANCE MEASUREMENT Refine indicators & define measures Develop data systems Collect data REPORTING OF PROGRESS Analyze data Feed data back to managers, staff, policy makers, constituents Develop a regular reporting cycle QI PROCESS Use data for decisions

Questions?