Monitoring & Evaluation

Objective Learn the why, what and how-to approach to monitoring Review monitoring techniques and define the roles monitoring plays in our lives both personally and professionally Understanding the types of monitoring and their impact on program operations Provide insight on the critical elements of monitoring Open Discussion

What Is Monitoring? Terminology Minefield Monitoring is a very widely used and ambiguous term Merriam Webster Monitoring is to watch, keep track of, or check for a special purpose General Definition Used to describe the process of checking or regulating some defined activity

Why Monitor?

Importance of Monitoring Through assessment, we know how well we are doing Through assessment, we know how effective and efficient we are Through assessment, we are able to monitor our resources You may go on using resources to no useful end

Why Monitor? Monitoring can be an invaluable management tool that enables you to assess the quality and impact of your work against your action plans

Why Monitor? Like riding a bicycle, you can adjust as you go along and ensure that you are on the right track

Why Monitor? Reasons Monitoring Is Not Popular It’s intrusive If it’s not broke, no need to fix it Other competing priorities No complaints, no problems Not necessary Like pulling teeth

Why Monitor? What If We Didn’t Monitor Our Health What We Eat Restaurants- Food Wardrobe Check-book Investments Pollution Homes Government Water Quality

Role of Monitoring Monitoring Is Used In… Project planning and implementation Analyzing the situation Determining whether the inputs in the project are well utilized Identifying problems facing the project or program and finding solutions

Role of Monitoring Monitoring Is Used To… Ensure all activities are carried out properly by the right people and in the allotted amount of time Apply lessons from one project experience on to other projects Determine whether project planning and problem solving is as appropriate and efficient as possible

Role Of Monitoring 4 Principles of Monitoring 1.Monitoring is the regular observation and recording of activities taking place in a project or program 2.Monitoring is a systematic and purposeful observation based on defined inputs 3.Monitoring involves giving feedback about the progress of the project or program 4.Monitoring enables the gathered information to be used in making decisions for improving project performance – Lessons learned

Role of Monitoring Monitoring Is A Feedback Loop  For Example Checking to ensure that we did what we said we would do Checking the assumptions and estimates used to make decisions Checking on program quality and efficiency Feedback and continuous improvement

Role Of Monitoring Monitoring Can Be Carried Out To meet one or more of a wide variety of objectives By measuring one or more of a wide range of variables At different levels of resolution

What To Monitor What Are We Monitoring Program Requirements Sensitivity and Risk Analysis Can determine key variables in developing a monitoring plan and monitoring requirements

World of Monitoring What? Why? How? Training Performance Implementation plans Certifications Effectiveness Efficiency Validation Impact Onsite or remotely Records Interviews

World of Monitoring Strategic Plan ImplementMonitor Reflect Learn Decide Adjust Feedback

Types Of Monitoring The phrase “monitoring is used in a wide variety of situations” Examples Include: Fiscal Monitoring Validation Monitoring Compliance Monitoring Effectiveness Monitoring Administrative Monitoring Programmatic Monitoring

Program Monitoring Programmatic Compliance Monitoring Consists of 2 Components Administrative Compliance Asks the question, “how well does RWB administrative documentation reflect actual conditions on the ground before and after program activities took place?” Field Compliance Asks the question, “how well are the rules and the terms of the program being applied on the ground?”

Program Monitoring Programmatic Compliance Monitoring The primary focus of programmatic monitoring is: To assess program operational practices, procedures and processes for compliance with applicable federal and State laws, policies and guidance To ensure that policies and practices relative to program quality and outcomes meet the program’s objectives To assist the RWB in identifying problem areas and recommending alternatives and practices that will promote positive practices and outcomes

Program Monitoring Critical Elements Define population of interest and take a representative sample Compare estimates from the representative sample with specific benchmarks Various levels of resolution depends on objectives and target population

Sampling Techniques Statistical Sampling - probability sampling where every item in the population under review has a known chance of selection Judgmental Sampling – sample size or composition determined by professional judgment rather than the laws of probability Sliding Scale - approximates the sample size determined using a semi-statistical approach Bootstrap Method?

Sampling Techniques DEO’s program monitoring section uses a statistical, random sampling method to select program files to be reviewed for each program

Sampling Techniques NO MATTER What methodology is used, make sure that it is a valid representative of the population being reviewed

Tools &Guides Types Of Tools/Guides Used: Process Management Review Tool Administrative Process Specific Review Tool Compliance

Tools & Guides Submission To Regions Monitoring tools are normally submitted to RWB staff at the beginning of the annual monitoring cycle for review and feedback

Tools & Guides Process Management Review Tool Asks questions about program processes, design elements, and organization Allows you to become familiar with program operations Evaluates program design in the context of whether it is demand-driven Allows you to draw objective conclusions about program operations, including promising “Best Practices”

Tools & Guides Program Specific Tool Used to determine compliance with specific program requirements Serves as an indicator of whether objectives or requirements are being met Identifies issues that may be contrary to law or other requirements

Tools & Guides A separate tool is developed for each program monitored Areas covered by tools include questions related to: Training Services Support Services Credential Attainment Participant Eligibility Criteria Assessment and Plan Development Grievance/Complaint/EEO requirements

Local Initiatives Quality Assurance Establish goals and objectives on what you want accomplished Develop an action/implementation plan based on the goals, activity levels, targets, etc Develop local operating policies/procedures/guidelines based on core management functions and program operations Develop tools based on your objectives, rules, tenets of law and other guidance (one or more indicators to assist in determining if objectives have been met) Train staff

Local Initiatives Case Managers Ensure all files have required documentation Develop a checklist of what should be included in the files Make sure detailed case notes are entered in system Conduct peer reviews/evaluations Exchange case files for review by different case managers Check accuracy of source documents in files against system data

Summary Monitoring is a very ambiguous term that is used for continuous improvement Don’t worry about what a monitoring program is called Focus on The pre-defined objectives What is being done Why it’s being done What impact is it having Remember that monitoring is the feedback loop that can provide useful information in planning and making changes

References Phil Bartle, PhD Civicus J.S Thrower & Associates Ltd.

Questions? For More Information On Monitoring and Evaluations Please Contact Ken Williams at lorida.com