THEME:EMPOWERING FACILITY MANAGEMENT AND HEALTHCARE TECHNOLOGY IN KENYA TITLE: Monitoring and Evaluation as a tool for quality assurance and control in our healthcare institutions PRESENTER: Mr Dennis Nyarangi (Bsc.Ppm, H Dip Ent Edu, H Dip Ele Eng , Dip ppm, Dip Me)

Definition of terms Quality assurance is a set of activities for ensuring quality in the process by which products are developed. QA aims at preventing defects with a focus on the process used to make the product. It is a proactive quality process. The goal of QA is to improve development and test process so that defects do not a rise when the product is being developed. Example of quality assurance is verification. Who is responsible for quality Assurance, All team members

Definition cont,s Quality Control is a set of activities for ensuring quality in the product. The activities focus on identifying defects in the actual products produced. Quality control aims to identify and correct defects in the finished product. Quality control therefore is a reactive process. The goal of Quality control is to identify defects after a product is developed and before its is released to the market. Validation /software testing is an example of quality control. Quality control is usually the responsibility of a specific team that tests the product for defects

Going from work plan to M&E Look at your activities and sub activities and ask yourselves these questions. -What information do you need on a continuous basis to make sure your activities are being carried out properly? -How will you collect that information? (site visits, facility data, maintenance activities data minutes, meetings, informant interviews) -What information do you need to determine the quality of those activities. -How could you collect that information? Do you need baseline data? -What information do you need to determine if your objectives are being met, your activities are having the desired impact?

Choosing your indicators You can now choose indicators that will give you the required information. Indicators should be: -Measurable - Directly related to your activity -Provide actionable information (will it identify problems so that you can solve them, will you be able to use information so you can self-correct)

Choosing your targets Targets should be: Set for each indicator Realistic and achievable within the time frame If your target is below what you would expect to make impact, then ask if you should increase resources to that activity or prioritise another activity i.e. If your indicator is monthly ppm and your target is to increase it by 5%,is that useful?

Key concepts used in M&E .Efficiency: The amount of outputs created and their quality in relation to the resources invested. This is usually measured in terms of cost incurred in the process. -Outputs are direct measurable results (goods and services) of the project which are largely under project management. -Is the project delivered in a timely and cost effective manner? .Effect. The more immediate, tangible and observable, in relation to initial and established objectives, which has been brought about as a direct result of project activities.

Concepts cont’d Effectiveness: This expresses the extent to which the planned outputs, outcomes and intended impacts are being or have been produced or achieved. It expresses the extent to which a project/programme achieves its objectives. -To what extent does the intervention achieve its objectives? What are the supportive factors and implementation? Impact: This refers to the long term, largely indirect consequences or end products of the project/programme for the intended beneficiaries. Impact can be positive or negative. Impact can be unpredictable .What happened as result of the project?

Impact Vs. outcomes Outcome refers to changes in the way people do things as result of the project e.g. Biomedical engineering professionals engaging in design and manufacturing of simple medical equipment. Impacts are long term, e.g. lowered importation of the locally manufactured equipment.

Concepts cont’d Relevance: The extent to which the project/programme is addressing or has addressed the problems of priority, mainly as viewed by the beneficiaries.

Concepts Cont’d Sustainability: This means the maintenance or positive changes induced by the programme after its phase out. It is ‘a withdrawal strategy’ of the project/programme. - Are there lasting benefits after the intervention is completed? Replicability: The feasibility of replicating the particular programme or parts of it in another context.

What is monitoring? Monitoring is a systematic and continuous process of collecting and analysing information about the progress of a project/programme. Monitoring is a process of asking ‘how well are we progressing?’

Monitoring cont’d Monitoring is a tool for: Identifying strengths and weakness in a project/programme. provide the people responsible for the programme/project planning and decision- making with sufficient information to make the right decisions at the time to improve its quality.

The need for M& E 1. For managers, stakeholders and donors: the need to know to what extend their projects are meeting their desired effects. 2. M&E builds greater transparency and accountability in terms of project resources.

M&E system cont’d 3. Information generated through M&E provides project staff with a clearer basis for decision making. 4. Future planning and development is improved if guided by lessons learnt from project experience.

Types of monitoring 1.Impact monitoring A measure of changes which show whether conditions of the target group and its environment have changed in a significant way as a result of programme/project intervention. 2. Physical progress monitoring Avery common practice of monitoring focusing on continuous review and surveillance of activities and results of a programme/project. In particular overseeing the planned verse the actual performance, collecting relevant information and the rescheduling of activities and resources.

Types of monitoring 3. Technical monitoring Focuses on use of technology in relation to resources. 4. Financial monitoring Monitoring actual expenditure patterns against planned budgets and implementation schedules.

Steps (principles) of effective monitoring Determine what to monitor, and at what intervals Determine how to monitor. Develop a checklist. Monitor as planned State any problem identified. Describe each problem. Identify possible causes of each problem. Develop alternate solutions for each problem identified Choose the most appropriate solution for each problem. Implement, Monitor and provide feedback to stakeholders.

Methods of Monitoring Reports: Written or verbal concerning particular aspects of work progress as per execution plan, any special problems or difficulties and how they have been solved. Visits: Very important as they add the human touch and ensure first hand information and boost staff morale. Checklist: It ensures all relevant aspects of the work have been enquired into. There has been no oversight because of the pressing problems.

Methods cont’d Meetings: Important to review and alter local targets and future planning. Complaints/compliments : Complaints are good indicators that there is something wrong, they must be followed up. On the other hand compliments are indicators that things are good.

Immediate benefits of Monitoring In healthcare delivery regular monitoring will lead to us avoiding common problems like; Shortage of spare parts due to delayed ordering. poor maintenance of machines and equipment. Reduced life value of machines and equipment. low customer satisfaction due to denied services. High mortality rates of our patients. High operating costs for our health care delivery institutions.

Importance of Monitoring Project management, staff and stakeholders: provide continuous assessment of plans, resources, infrastructure, and use of services by beneficiaries. Donors and stakeholders: Increase project accountability. Improve planning ,Donor satisfaction, project quality, and enhance learning from experience. Managers and staff: Opportunity to reinforce initial positive results, strengths and success. Target group: Check condition and changes brought by the project activities. Check if assumptions are valid.

Evaluation Is a Periodical and systematic process of gathering information to make judgments about a project’s achievements. Episodic or periodic assessment of the project interventions and their impact (both expected and unexpected) in relation to stated objectives.

Evaluation cont’d Evaluation measures: The extent to which change in outcomes can be attributed to the project. The difference in outcome of interest attributable to the project is the ‘impact’. Evaluation of this kind is known as impact evaluation.

Five basic Questions of evaluation What was intended to happen (objectives)? What has actually been achieved so far compared with objectives? What value should be place on the methods used (process)? What can be made of the information gained from question 1,2 and 3 (feedback)? What is the while exercise teaching us about managing future activities?

Types of Evaluation Types purpose Formative Initial assessment of the target population and contextual environment. Determines concept and design Process Seeks to identify the extent to which planned activities have been achieved and assesses the quality of the activities/services. Outcome Examines specific project outcome and accomplishment. What changes were observed. What does it mean and if changes are a result of the interventions. Impact Gauges the project’s overall impact and effectiveness. Aims to strengthen design and replication of effective projects and strategies.

Types of Evaluation Evaluation may be classified in terms of timing, agent and scope (UNDP Tools) By agent Internal or self valuation: This is conducted by those directly involved in formulation, implementation and management of the programme or project. External or Independent Evaluation: This is conducted by those who are not directly involved in the formulation ,implementation and management of the programme or project.

BY TIMING Mid term evaluation: Is conducted at the mid-point of the programme/project implementation. Terminal evaluation: Is conducted at the end of programme or project implementation. Ex-post evaluation: Is conducted usually two years or more after the completion of the programme or project.

BY SCOPE Programme/project Evaluation: Evaluation of a project or programme. Sectoral Evaluation: Cluster Evaluation of programmes/projects in a sector or sub sector. Thematic Evaluation: Cluster Evaluation of projects or programmes addressing a particular theme that may cut across sectors or geographical boundaries. Policy Evaluation: Cluster Evaluation of programmes/projects dealing with particular policy issues at the sectoral or thematic level. It aims to support policy making by recommending new policies or changes in the policies. Process Evaluation: A cluster Evaluation of the programmes or projects to assess the efficiency and effectiveness of a particular process or modality they have adopted e.g. an evaluation of the programme approach.

WHO SHOULD BE INVOLVED IN CONDUCTING THE EVALUATION The people/the groups to include in an evaluation may be determined by the approach taken in the evaluation. There are two broad approaches to an evaluation. These include: -the expert driven approach and -the participatory approach

The expert driven Approach In the expert driven approach to project/programme evaluation, one or more outside evaluators are given full responsibility for conducting the evaluation including: defining evaluation objectives designing the evaluation methodology Collecting and analysing information and Formulating their own conclusions and recommendations about the project.

The participatory Approach In the participatory approach to project evaluation, an evaluation conductor, often from outside the program or organisation, works in partnership with project stakeholders in all phases of the evaluation process. The programme stakeholders are those individuals who have a stake in how the evaluation comes out. They are persons who will later make decisions and use the information generated by the evaluation.

WHY EVALUATE ?

Why Evaluate ? Determine program/project effectiveness Show impact Strengthens financial responses and accountability Promotes a learning culture focused on service delivery improvement. Promotes replication of successful interventions

A comparison of Monitoring & Evaluation Continuous process Focuses on the activities being implemented according to plan. Answers questions on what, when how. Acts as an early warning system. Done internally by the project staff. Hands on; eyes on. Evaluation Periodic Focuses on the process and the results. Answers questions on how well and why. Identifies lessons for new projects. Done internally or externally. Eyes on Hands off

CHALLENGES IN MONITORING AND EVALUATION

WHY Monitoring & Evaluation? Monitoring & Evaluation should be part of the design of a program. Ensures systematic reporting Communicates results and accountability Measures efficiency and effectiveness. Provides information for improved decision making. Ensures effective allocation of resources. Promotes continuous learning and improvement.

Collectively ,the M&E system must provide the answer to the question Are we making a difference

THANK YOU FOR YOUR TIME ! 