1. Performance Management through Supervision, Monitoring and Evaluation
14/01/2019
Performance Management through Supervision, Monitoring and Evaluation
Dr. Shahram Yazdani

2. The Importance of Regulatory Activities

3. Policy-Operation Mismatch
Policies
Public, Administrative, Clinical
Operations
Risk Reduction, Public Health,
Clinical Services, Research,
Education,…
Health Outcomes

4. Policies are not self-implementable!
Public, Administrative, Clinical
Policy Formulation
Policy Legislations
Policy Evaluation
Policy Implementation
Operations
Risk Reduction, Public Health,
Clinical Services, Research,
Education,…
Health Outcomes

5. Intra vs. Inter-sectoral operations
Policies
Public, Administrative, Clinical
Intrasectoral Operations
Intersectoral Operations
Health Outcomes

6. Setting Regulatory Standards
Policies
Public, Administrative, Clinical
Regulatory Standards
Intrasectoral Operations
Intersectoral Operations
Health Outcomes

7. Performance Monitoring
Policies
Public, Administrative, Clinical
Regulatory Standards
Performance
Monitoring
Intrasectoral Operations
Intersectoral Operations
Health Outcomes

8. Regulatory Activities (Reward & Punishment)
Policies
Public, Administrative, Clinical
Regulatory Standards
Performance
Monitoring
Regulatory
Activities
Intrasectoral Operations
Intersectoral Operations
Health Outcomes

9. Regulatory Activities (Reward & Punishment)
Policies
Public, Administrative, Clinical
Accreditation
Licensing
Financial Flow
Regulatory Standards
Performance
Monitoring
Regulatory
Activities
Intrasectoral Operations
Intersectoral Operations
Health Outcomes

10. Governance Regulatory Standards Intrasectoral Operations
Policies
Public, Administrative, Clinical
Regulatory Standards
Performance
Monitoring
Regulatory
Activities
Intrasectoral Operations
Intersectoral Operations
Health Outcomes

11. Intra-sectoral Governance
Policies
Public, Administrative, Clinical
Intrasectoral Governance
Intrasectoral Operations
Intersectoral Operations
Health Outcomes

12. Affecting extra-sectoral operations
Policies
Public, Administrative, Clinical
Stakeholder analysis
Intrasectoral Governance
Advocacy
Conflict Negotiation
Intrasectoral Operations
Intersectoral Operations
Health Outcomes

13. Inter-sectoral Leadership
Policies
Public, Administrative, Clinical
Stakeholder analysis
Intrasectoral Governance
Advocacy
Conflict Negotiation
Intrasectoral Operations
Intersectoral Operations
Health Outcomes

14. Inter-sectoral Leadership
Policies
Public, Administrative, Clinical
Intrasectoral Governance
Intersectoral Leadership
Intrasectoral Operations
Intersectoral Operations
Health Outcomes

15. Policy Implementation
Policies
Public, Administrative, Clinical
Intrasectoral Governance
Intersectoral Leadership
Policy Formulation
Policy Legislations
Policy Implementation
Policy Evaluation
Intrasectoral Operations
Intersectoral Operations
Health Outcomes

16. Stewardship Intrasectoral Operations Intersectoral Operations
Policy Making
Intrasectoral Governance
Intersectoral Leadership
Intrasectoral Operations
Intersectoral Operations
Health Outcomes

17. Input for Policy-making
Policies
Public, Administrative, Clinical
Impact Evaluation
Intrasectoral Governance
Intersectoral Leadership
Intrasectoral Operations
Intersectoral Operations
Health Outcomes

18. Input for Policy-making
Policies
Public, Administrative, Clinical
Impact Evaluation
Operation
Monitoring
Intrasectoral Governance
Intersectoral Leadership
Intrasectoral Operations
Intersectoral Operations
Health Outcomes

19. The Importance of MIS & HIS
Policies
Public, Administrative, Clinical
Impact Evaluation
Operation
Monitoring
HIS
MIS
Intrasectoral Governance
Intersectoral Leadership
Intrasectoral Operations
Intersectoral Operations
Health Outcomes

20. Input for Policy-making
Policies
Public, Administrative, Clinical
Impact Evaluation
Health Research
Operation
Monitoring
HIS
MIS
Intrasectoral Governance
Intersectoral Leadership
Intrasectoral Operations
Intersectoral Operations
Health Outcomes

21. Evidence Based Policy Making
Policies
Public, Administrative, Clinical
Impact Evaluation
Health Research
Operation
Monitoring
HIS
MIS
Intrasectoral Governance
Intersectoral Leadership
Intrasectoral Operations
Intersectoral Operations
Health Outcomes

22. Inputs to Evidence Based Policy Making
What is the current body of evidence?
Systematic Review of International Literature
What is the current practice and success stories?
National and International Benchmarking
Comparative Studies
What is the unique specification and requirements?
Situation Analysis
What is the current mode of practice?
Performance Monitoring
What is the effect of previous policies?
Impact Evaluation
What are the evidence gaps that should be filled ?
Original National Research
Evidence
Based
Policymaking

23. Ideal Balance in Structure and Function in Public Sector
Information
Regulation
Finance
Policymaking
Mandates
Provision

24. The Case of Hierarchical (Multi-level) Organizations

25. Organization 1 (O1) Organization 1-1 Organization 1-2 Organization 1-3

26. Objectives Mission Goals Organization 1 Organization 1-1 Objectives

27. Mission
Goals
Objectives

28. Mission
Goals
Objectives

29. Mission Goals Objectives Mission Goals Objectives Mission Goals

30. Mission Goals Objectives Interventions Mission Goals Objectives

31. Mission Goals Objectives Interventions Mission Goals Objectives

32. Mission Goals Objectives Interventions Mission Goals Objectives

33. Mission Goals Objectives Interventions Mission Goals Objectives

34. Responsibilities Mission Goals Objectives Interventions
Main responsible Body
Interventions
Collaborating Bodies
Responsibilities
Coordinating Body
Controlling Body

35. Objectives Mission Goals Objectives Mission Goals Organization 1

36. Objectives Mission Goals Objectives Mission Goals Organization 1

37. Objectives Mission Goals Objectives Mission Goals Organization 1

38. Superior Organization
Setting Goals
and Objectives
Planning
Coordinating
Monitoring
Subordinate Organization
Setting Goals
and Objectives
Planning
According to
Preset Goals
and Objectives
Implementation
of preset plan

39. Superior Organization
Scenario
Superior Organization
Setting Goals
and Objectives
Determining
Interventions
(Planning)
Assigning
Interventions
to Agencies
Monitoring
Subordinate Organization
Setting Goals
and Objectives
Planning
According to
Preset Goals
and Objectives
Implementation
of preset plan

40. Hierarchical Structure and Hierarchy of Merits
Aligned and Assigned (Assumed)
Aligned but not Assigned (Assumed)
Not Aligned but Valued
Not Aligned and Not Valued

41. Related Terminology and Definitions

42. Measurement (اندازه گیری)
Measurement refers to the process by which the attributes or dimensions of some physical object are determined.
When we measure, we generally use some standard instrument to determine how big, tall, heavy, voluminous, hot, cold, fast, or straight something actually is. Standard instruments refer to instruments such as rulers, scales, thermometers, pressure gauges, etc.
To apply a standard scale or measuring device to an object, series of objects, events, or conditions, according to practices accepted by those who are skilled in the use of the device or scale.

43. Assessment (ارزیابی)
Assessment is a process by which information is obtained relative to some known objective or goal.
Assessment is a broad term that includes  testing.
A test is a special form of assessment. Tests are assessments made under contrived circumstances especially so that they may be administered. 
In other words, all tests are assessments, but not all assessments are tests. We test at the end of a lesson or unit.
To assess means To stipulate the conditions by which the behavior specified in an objective may be ascertained. Such stipulations are usually in the form of  written descriptions.

44. Evaluation (ارزشیابی)
Evaluation is the systematic and impartial collection, analysis and interpretation of data needed to make decisions about an activity, project, programme, strategy, policy, topic, theme, sector, operational area, or institutional performance.
It aims at determining the relevance, impact, effectiveness, efficiency and sustainability of the interventions and contributions of the organizations.

45. Evaluation (ارزشیابی)
14/01/2019
Evaluation (ارزشیابی)
Key elements:
assessment conducted at a single point in time (before, during or after).
focuses on determining whether what was planned actually happened, and why it did or did not happen.
Assessing:
relevance – whether the intervention was appropriate
impact – whether it made a difference in the lives of people
effective – whether it achieved what it set out to
efficient – whether it did so at the lowest cost
sustainable –whether it will leading to lasting change.

46. Evaluation Value Judgment V1 V2 V3 V4 V5 V6 V7 V8 V9 V10 A1 A2 A3 A4

47. Audit (ممیزی)  
Systematic, independent and documented process for obtaining evidence and evaluating it objectively to determine the extent to which audit criteria are fulfilled. 

48. Audit V1 V2 V3 V4 V5 V6 V7 V8 V9 V10 A4 S1 S2 S3 S4 S5 S6 S7 S8 S9 S10

49. Accreditation (اعتباربخشی)
The process carried on by an agency or association leading to some kind of formal decision (mostly Yes/No) about a institute, or specialized program of study based on implicit or explicit quality standards.

50. Accreditation Yes/No Decision V1 V2 V3 V4 V5 V6 V7 V8 V9 V10 A4 S1 S2

51. Evaluation, Audit and Accreditation
Relationship Between
Evaluation, Audit and Accreditation
Accreditation
Audit
Evaluation
Systematic
Data Collection
Judgment
Embedded
Decision
Yes No
Yes No
Yes

52. Accreditation Models:
Multi-group Standards
LCME
V1 V2 V3 V4 V5 A1 A2 A3 A4 A5 A6 A7 A8 A9
A10
V6 V7 V8 V9 V10 S6 S7 S8 S9
S10 S1 S2 S3 S4 S5
Should Criteria
Must Criteria

53. Accreditation Models: Multilevel Standards
WFME
V1 V2 V3 V4 V5 V6 V7 V8 V9 V10 A1 A2 A3 A4 A5 A6 A7 A8 A9
A10
S’1
S’2
S’3
S’4
S’5
S’6
S’7
S’8
S’9
S’10
Top
Standards S1 S2 S3 S4 S5 S6 S7 S8 S9
S10
Basic
Standards

54. Scoring (امتیازدهی) ∑ S = wiAi w1 w2 w3 w4 w5 w6 w7 w8 w9 w10
V1 V2 V3 V4 V5 V6 V7 V8 V9 V10

55. Ranking (رتبه بندی) ∑ S = wiAi S1> S2> S3> S4> S5
w1 w2 w3 w4 w5 w6 w7 w8 w9 w10 A4 A6
A10 A3 A7 A5 A2 A8 A9 A1
V1 V2 V3 V4 V5 V6 V7 V8 V9 V10
S1> S2> S3> S4> S5

56. 14/01/2019
Monitoring (پایش)
Management’s continuous examination of progress achieved during the implementation of an undertaking to track compliance with the plan and to take necessary decisions to improve performance. Key elements: - continuous examination of implementation progress - tracking compliance against planned objectives - generating data and information on performance to enable corrective measures to be taken

57. Supervision (نظارت)
Supervision is an action of overseeing/looking the actions or the work going on (similar to monitoring).
The supervisor is basically in more contact or interaction with the job or the work going on (e.g. giving feedback to workers). The monitor doesn’t have to interact with the work very much as well.
The supervisor has some limitations as well. While the monitor has the fewer boundaries and the limitations as compared to the supervisor.

58. 14/01/2019
Inspection (بازرسی)
A general examination that seeks to identify vulnerable areas and malfunctions and to propose corrective action.

59. 14/01/2019
Investigation (تفحص)
A specific examination of a claim of wrongdoing and provision of evidence for eventual prosecution or disciplinary measures.

60. 14/01/2019
Research (پژوهش)
A systematic examination designed to develop or contribute to knowledge.

61. Different oversight functions
14/01/2019
Different oversight functions
Programme Cycle Management
Monitoring
Audit
Evaluation
Inspection
Continuous Assessment
Research
Performance
Compliance
Modelling
Wrong Doing

62. Evaluation as Value Judgement?
Who’s Judgement?

63. Why it is important to determine the reason for evaluation
Performance is assessed for a number of reasons (administrative reasons, performance improvement)
Various performance appraisal methods are appropriate for some purposes, not for others.
Employee comparison methods such as forced choice and ranking: good for administrative purposes, not good for performance improvement
360-degree evaluation: good for performance improvement, not good for administrative purposes

64. Traditional Performance Appraisal: Who does it?
90% of appraisals are done by supervisors
Others, however, may also participate in the appraisal
Peers
Subordinates/Direct reports (upward feedback)
Customers
Employee himself or herself (rather rare)
28% of large universities and companies use some form of multi-source evaluations

65. Different Results
There is often little agreement when different sources evaluate an item
Supervisors and peers: 0.34 correlation
Supervisors and subordinates: 0.22 correlation
Supervisors and the employee: 0.22 correlation
Peers and subordinates: 0.22 correlation
Peers and the employee: 0.19

66. Two reasons why results may differ
Those groups see different aspects of the performance
The supervisor may see the results, but not see how the person interacts with peers, direct reports and customers
The employee may interact very differently with the supervisor than with peers and subordinates

67. Peer assessments
In general, when peers are similar and know the person well peer ratings have good reliability and validity
Reliability - different peers rate the employee the same way; Inter-observer agreement
Validity - predict future success of employees
However, employees react worse to negative feedback from peers than from supervisors/experts
Main obstacle to peer assessments is poor acceptance by employees

68. Ratings by subordinates
Not surprisingly, accurate subordinate ratings can be hard to obtain. Why?
Employees fear a backlash, particularly if there are only a few of them and the supervisor is likely to be able to identify them.
Data also indicate that feedback /ratings from subordinates brings about the greatest performance change when compared to feedback /ratings by supervisors and peers

69. 360 degree feedback A manager/supervisor
Rates himself
Is rated by his manager/supervisor
Is rated by his peers
Is rated by his subordinates
What percentage of managers saw themselves as others saw them?
Only 10%!

70. Self-assessment Main problem? Self-ratings correlate
Inflation: we think we are better than we are
Self-ratings correlate
only moderately with actual performance
poorly with subordinate ratings
poorly with management ratings
but interesting cross-cultural data
Ratings of Japanese, Korean, & Taiwanese workers suffer from strictness (modesty)
Ratings from workers in the US, mainland China, India, Singapore, and Hong Kong suffer from leniency

71. Different Approaches to Evaluation

72. Alternative Approaches
Stakeholders: individuals and groups who have a direct interest in, and may be affected by, evaluation; should be involved early, actively & continuously
Program: activities that are provided on a continuing basis; typically what is evaluated
There are a variety of alternative, often conflicting, views of what evaluation is and how it should be carried out

73. Why so many alternatives?
The way one views evaluation directly impacts the type of activities/methods used
Origins of alternative models stem from differences in:
Philosophical & ideological beliefs
Methodological preferences
Practical choices

74. Philosophical & Ideological Beliefs
Epistemologies (philosophies of knowing)
Objectivism (social science base of empiricism; replicate)
Subjectivism (experientially-based; tacit knowledge)
Principles for assigning value (parallel obj/subj)
Utilitarian: focus on group gains (avg scores); greatest good for the greatest number
Intuitionist-pluralist: value is individually-determined

75. Classification Schema for Evaluation Approaches
Objectives-oriented: focus on goals/objectives & degree to which they are achieved
Management-oriented: identifying and meeting informational needs of decision makers
Consumer-oriented: generate information to guide product/service use by consumers
Expertise-oriented: use of professional expertise to judge quality of evaluation object
Participant-oriented: stakeholders centrally involved in process

76. Thank You ! Any Question?