1. The Role of Evaluation in Public Health
The Big Picture
The Role of Evaluation in Public Health

2. Activity
Think of your favorite quote or sage advice that helps guide you through your professional work.

3. Wise Advice I hear and I forget. I see and I believe.
I do and I understand.
—Confucius (551–479 BC)

4. CDC Evaluation Framework
Standards
Utility
Accuracy
Propriety
Feasibility

5. Objectives
Describe the relationship between performance management, program planning and evaluation
Describe the steps of program planning and evaluation process
Identify key stakeholders in an evaluation
Develop the components and elements of a program logic model
Create a basic evaluation plan including data collection and analyzing methods
List three ways to use evaluation data to draw conclusions about a program
Define ROI, cost benefit ratios and social return on investment. 5

6. Accreditation
Accreditation is a major accomplishment for a health department. It means that it is addressing key community health problems. Just as the public expects hospitals, law enforcement agencies, and schools to be accredited, so should they come to expect public health departments.
Thomas Frieden, MD, MPH
CDC Director
Why are we talking about these manufacturing and business based frameworks in public health?
Accreditation
Accreditation is a major driver for this work in our agencies and with our partners in public health.
You can see from Dr. Friedan’s quote….it is expected…

7. CDC Grants
CDC awards grants for public health performance management systems
$42.5 million allocated for 94 projects in state, local, tribal, and territories health departments
Distributed in 49 states, 16 tribes and territories and nine local jurisdictions
Five-year cooperative agreement awards are being funded by the health care reform law
There is also resource support from the national level…with the National Public Health Improvement Initiates (NPHII)
CDC The Centers for Disease Control and Prevention provided funding for state and local public health departments improve their performance management, training and collaboration by awarding them $42.5 million in grants.

8. Benefits of Accreditation
Accountability & credibility
Tool for improvement
Highlights HD strengths
Greater
collaboration
Accreditation +
Team building
Those who champion accreditation list numerous benefits….with good reason.
If we look at what is listed here in the slide all of the bubbles are very practical for public health and our partner to want to achieve to…
(list each)
Recognition & validation
Better understanding of public health

9. Analysis & Environmental Scan
Strategic Planning
Phase 4
Evaluation
Monitoring
Implementation
Budgets
Action Plans
Strategies
Phase 3
Strategic Issues
Goals & Objectives
Phase 2
Vision
Mission
Values
SWOC
Analysis & Environmental Scan
Plan the Plan
Readiness Assessment
Stakeholder
Assessment
Mandate Analysis
Phase 1

10. Performance Management (PM)
A systematic process aimed at helping achieve an organization’s mission and strategic goals by improving effectiveness, empowering employees, and streamlining the decision-making process. 
Performance management is an ongoing, continuous process of communicating and clarifying job responsibilities, priorities and performance expectations in order to ensure mutual understanding between supervisor and employee.
Quality improvement in public health is the use of a deliberate and defined improvement process, such as Plan-Do-Check-Act, which is focused on activities that are responsive to community needs and improving population health.
It refers to a continuous and ongoing effort to achieve measurable improvements in the efficiency, effectiveness, performance, accountability, outcomes, and other indicators of quality in services or processes which achieve equity and improve the health of the community. (R. Bialek, L. M. Beitsch, A. Cofsky, et al, unpublished data, 2009)
Performance =
Behavior
Results

11. They Are Not the Same Quality assurance Quality improvement Reactive
Proactive
Works on problems after they occur
Works on processes
Regulatory usually by state or federal law
Seeks to improve (culture shift)
Led by management
Led by staff
Periodic look-back
Continuous
Responds to a mandate or crisis or fixed schedule
Proactively selects a process to improve
Meets a standard (pass/fail)
Exceeds expectations
QA is reactive, going back to check on how things had been done. QI is proactive, thinking about ongoing processes in place and how to make measurable improvements.
QA often results from regulation, while QI signals an agency’s desire to operate in a culture of always improving
Managers typically take the lead in QA, while QI deliberately involves staff at all levels
QA is done on a periodic, scheduled basis, while QI is an ongoing effort
QA operates on a pass/fail basis – either something was done correctly or not, while QI is all about exceeding expectations – always setting the bar higher

12. They Are Not the Same Evaluation Quality improvement
Assess a program at a moment in time
Understand the process that is in place
Static
Ongoing
Doesn’t include identification of the source of a problem or potential solutions
Entails finding the root cause of a problem and interventions targeted to address it
Doesn’t measure improvements
Focused on making measurable improvements
Program-focused
Customer-focused
A step in the QI process
Includes evaluation
QA is reactive, going back to check on how things had been done. QI is proactive, thinking about ongoing processes in place and how to make measurable improvements.
QA often results from regulation, while QI signals an agency’s desire to operate in a culture of always improving
Managers typically take the lead in QA, while QI deliberately involves staff at all levels
QA is done on a periodic, scheduled basis, while QI is an ongoing effort
QA operates on a pass/fail basis – either something was done correctly or not, while QI is all about exceeding expectations – always setting the bar higher

13. Value of Performance Management
Disciplined approach
Systemic
Systematic
Engaged customers and staff
Results focused
Proven method
So why performance management frameworks – what is the value added
Well what we know from manufacturing and industry
It is a very disciplined approach
It is looking at improving systems
It is systematic (or step by step)
It engages stakeholders (so partners, staff, customers…)
It is very results focused (not so much a count on what we did….rather what happened as a result?
Been around a while and is a proven methodology 13

14. Culture of QI Topic Organization-wide Program/unit Improvement
System focus
Specific project focus
QI planning
Tied to the strategic plan
Program/unit level
QI goals
Strategic plan
Individual program/unit
level plans
Approaches
Baldrige Program
Organization QI Council
Lean Six Sigma
Individual QI Teams
Rapid Cycle PDCA
It can be useful to think of QI work across a spectrum. At one end of the spectrum is organization-wide QI, or having a culture of QI throughout the health department. At the other end of the spectrum is engaging in QI in a specific program, or even project – just beginning to apply the QI process somewhere in the health department.
A culture of QI is highly desirable, although it’s recognized that it’s important to start at the other end of the spectrum. And it’s fine to start small – in fact, it’s desirable to start small.
(Describe where your department is on the spectrum, and if appropriate, describe how you would like the department to progress towards an organization-wide approach to QI, or having a “QI culture”)

15. Performance Management Frameworks
Baldrige
Six Sigma
Lean
Performance management frameworks can begin to start to build this system.
There are many frameworks that you may have hear about through the years—today I will briefly touch three: Baldrige, Six Sigma, Lean

16. PDSA Cycle What are we trying to accomplish?
How will we know that a change is an improvement?
What changes can we make that will result in improvement?
Act
Study
Plan Do
Example of performance measurement systems
Scorecard
Dashboard

17. Memory Jogger

18. Tools Logic models and work flow charts Data and analysis tools
Customer-supplier relationships
Client flow, information flow
Root cause tools: fishbone diagram, Pareto chart
Force field analysis
Interrelationship digraph
Logic models provide a graphic illustration of your program plan
Flowcharts allow a team to visualize actual flow or sequence of events
Pareto’s Principle or Pareto’s Law – 80/20 Rule
20% of something is always responsible for 80% of the results.

19. PERFORMANCE MANAGEMENT SYSTEM
PERFORMANCE STANDARDS
Identify relevant standards
Select indicators
Set goals & targets
Communicate expectations
PERFORMANCE MEASUREMENT
Refine indicators & define measures
Develop data systems
Collect data
PERFORMANCE MANAGEMENT SYSTEM
REPORTING OF PROGRESS
Analyze data
Feed data back to managers, staff, policy makers, constituents
Develop a regular reporting cycle
QI PROCESS
to improve policies, programs, & outcomes
Manage changes
Create learning organization
QI PROCESS
Use data for decisions
Performance Standards - establishment of organizational or system performance standards, targets and goals and relevant indicators to improve public health practice
Performance Measures - application and use of performance indicators and measures
Quality Improvement - establishment of a program or process to manage change and achieve quality improvement in public health policies, programs or infrastructure based on performance standards, measurements and reports.
Reporting of Progress - documentation and reporting of progress in meeting standards and targets and sharing of such information through feedback
Turning Point Performance Management Collaborative, 2003

20. Questions?