Performance Management through Supervision, Monitoring and Evaluation 14/01/2019 Performance Management through Supervision, Monitoring and Evaluation Dr. Shahram Yazdani

The Importance of Regulatory Activities

Policy-Operation Mismatch Policies Public, Administrative, Clinical Operations Risk Reduction, Public Health, Clinical Services, Research, Education,… Health Outcomes

Policies are not self-implementable! Public, Administrative, Clinical Policy Formulation Policy Legislations Policy Evaluation Policy Implementation Operations Risk Reduction, Public Health, Clinical Services, Research, Education,… Health Outcomes

Intra vs. Inter-sectoral operations Policies Public, Administrative, Clinical Intrasectoral Operations Intersectoral Operations Health Outcomes

Setting Regulatory Standards Policies Public, Administrative, Clinical Regulatory Standards Intrasectoral Operations Intersectoral Operations Health Outcomes

Performance Monitoring Policies Public, Administrative, Clinical Regulatory Standards Performance Monitoring Intrasectoral Operations Intersectoral Operations Health Outcomes

Regulatory Activities (Reward & Punishment) Policies Public, Administrative, Clinical Regulatory Standards Performance Monitoring Regulatory Activities Intrasectoral Operations Intersectoral Operations Health Outcomes

Regulatory Activities (Reward & Punishment) Policies Public, Administrative, Clinical Accreditation Licensing Financial Flow Regulatory Standards Performance Monitoring Regulatory Activities Intrasectoral Operations Intersectoral Operations Health Outcomes

Governance Regulatory Standards Intrasectoral Operations Policies Public, Administrative, Clinical Regulatory Standards Performance Monitoring Regulatory Activities Intrasectoral Operations Intersectoral Operations Health Outcomes

Intra-sectoral Governance Policies Public, Administrative, Clinical Intrasectoral Governance Intrasectoral Operations Intersectoral Operations Health Outcomes

Affecting extra-sectoral operations Policies Public, Administrative, Clinical Stakeholder analysis Intrasectoral Governance Advocacy Conflict Negotiation Intrasectoral Operations Intersectoral Operations Health Outcomes

Inter-sectoral Leadership Policies Public, Administrative, Clinical Stakeholder analysis Intrasectoral Governance Advocacy Conflict Negotiation Intrasectoral Operations Intersectoral Operations Health Outcomes

Inter-sectoral Leadership Policies Public, Administrative, Clinical Intrasectoral Governance Intersectoral Leadership Intrasectoral Operations Intersectoral Operations Health Outcomes

Policy Implementation Policies Public, Administrative, Clinical Intrasectoral Governance Intersectoral Leadership Policy Formulation Policy Legislations Policy Implementation Policy Evaluation Intrasectoral Operations Intersectoral Operations Health Outcomes

Stewardship Intrasectoral Operations Intersectoral Operations Policy Making Intrasectoral Governance Intersectoral Leadership Intrasectoral Operations Intersectoral Operations Health Outcomes

Input for Policy-making Policies Public, Administrative, Clinical Impact Evaluation Intrasectoral Governance Intersectoral Leadership Intrasectoral Operations Intersectoral Operations Health Outcomes

Input for Policy-making Policies Public, Administrative, Clinical Impact Evaluation Operation Monitoring Intrasectoral Governance Intersectoral Leadership Intrasectoral Operations Intersectoral Operations Health Outcomes

The Importance of MIS & HIS Policies Public, Administrative, Clinical Impact Evaluation Operation Monitoring HIS MIS Intrasectoral Governance Intersectoral Leadership Intrasectoral Operations Intersectoral Operations Health Outcomes

Input for Policy-making Policies Public, Administrative, Clinical Impact Evaluation Health Research Operation Monitoring HIS MIS Intrasectoral Governance Intersectoral Leadership Intrasectoral Operations Intersectoral Operations Health Outcomes

Evidence Based Policy Making Policies Public, Administrative, Clinical Impact Evaluation Health Research Operation Monitoring HIS MIS Intrasectoral Governance Intersectoral Leadership Intrasectoral Operations Intersectoral Operations Health Outcomes

Inputs to Evidence Based Policy Making What is the current body of evidence? Systematic Review of International Literature What is the current practice and success stories? National and International Benchmarking Comparative Studies What is the unique specification and requirements? Situation Analysis What is the current mode of practice? Performance Monitoring What is the effect of previous policies? Impact Evaluation What are the evidence gaps that should be filled ? Original National Research Evidence Based Policymaking

Ideal Balance in Structure and Function in Public Sector Information Regulation Finance Policymaking Mandates Provision

The Case of Hierarchical (Multi-level) Organizations

Organization 1 (O1) Organization 1-1 Organization 1-2 Organization 1-3

Objectives Mission Goals Organization 1 Organization 1-1 Objectives

Mission Goals Objectives

Mission Goals Objectives

Mission Goals Objectives Mission Goals Objectives Mission Goals

Mission Goals Objectives Interventions Mission Goals Objectives

Mission Goals Objectives Interventions Mission Goals Objectives

Mission Goals Objectives Interventions Mission Goals Objectives

Mission Goals Objectives Interventions Mission Goals Objectives

Responsibilities Mission Goals Objectives Interventions Main responsible Body Interventions Collaborating Bodies Responsibilities Coordinating Body Controlling Body

Objectives Mission Goals Objectives Mission Goals Organization 1

Objectives Mission Goals Objectives Mission Goals Organization 1

Objectives Mission Goals Objectives Mission Goals Organization 1

Superior Organization Setting Goals and Objectives Planning Coordinating Monitoring Subordinate Organization Setting Goals and Objectives Planning According to Preset Goals and Objectives Implementation of preset plan

Superior Organization Scenario Superior Organization Setting Goals and Objectives Determining Interventions (Planning) Assigning Interventions to Agencies Monitoring Subordinate Organization Setting Goals and Objectives Planning According to Preset Goals and Objectives Implementation of preset plan

Hierarchical Structure and Hierarchy of Merits Aligned and Assigned (Assumed) Aligned but not Assigned (Assumed) Not Aligned but Valued Not Aligned and Not Valued

Related Terminology and Definitions

Measurement (اندازه گیری) Measurement refers to the process by which the attributes or dimensions of some physical object are determined. When we measure, we generally use some standard instrument to determine how big, tall, heavy, voluminous, hot, cold, fast, or straight something actually is. Standard instruments refer to instruments such as rulers, scales, thermometers, pressure gauges, etc. To apply a standard scale or measuring device to an object, series of objects, events, or conditions, according to practices accepted by those who are skilled in the use of the device or scale.

Assessment (ارزیابی) Assessment is a process by which information is obtained relative to some known objective or goal. Assessment is a broad term that includes  testing. A test is a special form of assessment. Tests are assessments made under contrived circumstances especially so that they may be administered.  In other words, all tests are assessments, but not all assessments are tests. We test at the end of a lesson or unit. To assess means To stipulate the conditions by which the behavior specified in an objective may be ascertained. Such stipulations are usually in the form of  written descriptions.

Evaluation (ارزشیابی) Evaluation is the systematic and impartial collection, analysis and interpretation of data needed to make decisions about an activity, project, programme, strategy, policy, topic, theme, sector, operational area, or institutional performance. It aims at determining the relevance, impact, effectiveness, efficiency and sustainability of the interventions and contributions of the organizations.

Evaluation (ارزشیابی) 14/01/2019 Evaluation (ارزشیابی) Key elements: assessment conducted at a single point in time (before, during or after). focuses on determining whether what was planned actually happened, and why it did or did not happen. Assessing: relevance – whether the intervention was appropriate impact – whether it made a difference in the lives of people effective – whether it achieved what it set out to efficient – whether it did so at the lowest cost sustainable –whether it will leading to lasting change.

Evaluation Value Judgment V1 V2 V3 V4 V5 V6 V7 V8 V9 V10 A1 A2 A3 A4

Audit (ممیزی)   Systematic, independent and documented process for obtaining evidence and evaluating it objectively to determine the extent to which audit criteria are fulfilled. 

Audit V1 V2 V3 V4 V5 V6 V7 V8 V9 V10 A4 S1 S2 S3 S4 S5 S6 S7 S8 S9 S10

Accreditation (اعتباربخشی) The process carried on by an agency or association leading to some kind of formal decision (mostly Yes/No) about a institute, or specialized program of study based on implicit or explicit quality standards.

Accreditation Yes/No Decision V1 V2 V3 V4 V5 V6 V7 V8 V9 V10 A4 S1 S2

Evaluation, Audit and Accreditation Relationship Between Evaluation, Audit and Accreditation Accreditation Audit Evaluation Systematic Data Collection Judgment Embedded Decision Yes No Yes No Yes

Accreditation Models: Multi-group Standards LCME V1 V2 V3 V4 V5 A1 A2 A3 A4 A5 A6 A7 A8 A9 A10 V6 V7 V8 V9 V10 S6 S7 S8 S9 S10 S1 S2 S3 S4 S5 Should Criteria Must Criteria

Accreditation Models: Multilevel Standards WFME V1 V2 V3 V4 V5 V6 V7 V8 V9 V10 A1 A2 A3 A4 A5 A6 A7 A8 A9 A10 S’1 S’2 S’3 S’4 S’5 S’6 S’7 S’8 S’9 S’10 Top Standards S1 S2 S3 S4 S5 S6 S7 S8 S9 S10 Basic Standards

Scoring (امتیازدهی) ∑ S = wiAi w1 w2 w3 w4 w5 w6 w7 w8 w9 w10 V1 V2 V3 V4 V5 V6 V7 V8 V9 V10

Ranking (رتبه بندی) ∑ S = wiAi S1> S2> S3> S4> S5 w1 w2 w3 w4 w5 w6 w7 w8 w9 w10 A4 A6 A10 A3 A7 A5 A2 A8 A9 A1 V1 V2 V3 V4 V5 V6 V7 V8 V9 V10 S1> S2> S3> S4> S5

14/01/2019 Monitoring (پایش) Management’s continuous examination of progress achieved during the implementation of an undertaking to track compliance with the plan and to take necessary decisions to improve performance. Key elements: - continuous examination of implementation progress - tracking compliance against planned objectives - generating data and information on performance to enable corrective measures to be taken

Supervision (نظارت) Supervision is an action of overseeing/looking the actions or the work going on (similar to monitoring). The supervisor is basically in more contact or interaction with the job or the work going on (e.g. giving feedback to workers). The monitor doesn’t have to interact with the work very much as well. The supervisor has some limitations as well. While the monitor has the fewer boundaries and the limitations as compared to the supervisor.

14/01/2019 Inspection (بازرسی) A general examination that seeks to identify vulnerable areas and malfunctions and to propose corrective action.

14/01/2019 Investigation (تفحص) A specific examination of a claim of wrongdoing and provision of evidence for eventual prosecution or disciplinary measures.

14/01/2019 Research (پژوهش) A systematic examination designed to develop or contribute to knowledge.

Different oversight functions 14/01/2019 Different oversight functions Programme Cycle Management Monitoring Audit Evaluation Inspection Continuous Assessment Research Performance Compliance Modelling Wrong Doing

Evaluation as Value Judgement? Who’s Judgement?

Why it is important to determine the reason for evaluation Performance is assessed for a number of reasons (administrative reasons, performance improvement) Various performance appraisal methods are appropriate for some purposes, not for others. Employee comparison methods such as forced choice and ranking: good for administrative purposes, not good for performance improvement 360-degree evaluation: good for performance improvement, not good for administrative purposes

Traditional Performance Appraisal: Who does it? 90% of appraisals are done by supervisors Others, however, may also participate in the appraisal Peers Subordinates/Direct reports (upward feedback) Customers Employee himself or herself (rather rare) 28% of large universities and companies use some form of multi-source evaluations

Different Results There is often little agreement when different sources evaluate an item Supervisors and peers: 0.34 correlation Supervisors and subordinates: 0.22 correlation Supervisors and the employee: 0.22 correlation Peers and subordinates: 0.22 correlation Peers and the employee: 0.19

Two reasons why results may differ Those groups see different aspects of the performance The supervisor may see the results, but not see how the person interacts with peers, direct reports and customers The employee may interact very differently with the supervisor than with peers and subordinates

Peer assessments In general, when peers are similar and know the person well peer ratings have good reliability and validity Reliability - different peers rate the employee the same way; Inter-observer agreement Validity - predict future success of employees However, employees react worse to negative feedback from peers than from supervisors/experts Main obstacle to peer assessments is poor acceptance by employees

Ratings by subordinates Not surprisingly, accurate subordinate ratings can be hard to obtain. Why? Employees fear a backlash, particularly if there are only a few of them and the supervisor is likely to be able to identify them. Data also indicate that feedback /ratings from subordinates brings about the greatest performance change when compared to feedback /ratings by supervisors and peers

360 degree feedback A manager/supervisor Rates himself Is rated by his manager/supervisor Is rated by his peers Is rated by his subordinates What percentage of managers saw themselves as others saw them? Only 10%!

Self-assessment Main problem? Self-ratings correlate Inflation: we think we are better than we are Self-ratings correlate only moderately with actual performance poorly with subordinate ratings poorly with management ratings but interesting cross-cultural data Ratings of Japanese, Korean, & Taiwanese workers suffer from strictness (modesty) Ratings from workers in the US, mainland China, India, Singapore, and Hong Kong suffer from leniency

Different Approaches to Evaluation

Alternative Approaches Stakeholders: individuals and groups who have a direct interest in, and may be affected by, evaluation; should be involved early, actively & continuously Program: activities that are provided on a continuing basis; typically what is evaluated There are a variety of alternative, often conflicting, views of what evaluation is and how it should be carried out

Why so many alternatives? The way one views evaluation directly impacts the type of activities/methods used Origins of alternative models stem from differences in: Philosophical & ideological beliefs Methodological preferences Practical choices

Philosophical & Ideological Beliefs Epistemologies (philosophies of knowing) Objectivism (social science base of empiricism; replicate) Subjectivism (experientially-based; tacit knowledge) Principles for assigning value (parallel obj/subj) Utilitarian: focus on group gains (avg scores); greatest good for the greatest number Intuitionist-pluralist: value is individually-determined

Classification Schema for Evaluation Approaches Objectives-oriented: focus on goals/objectives & degree to which they are achieved Management-oriented: identifying and meeting informational needs of decision makers Consumer-oriented: generate information to guide product/service use by consumers Expertise-oriented: use of professional expertise to judge quality of evaluation object Participant-oriented: stakeholders centrally involved in process

Thank You ! Any Question?