Monitoring & Evaluation
Objective Learn the why, what and how-to approach to monitoring Review monitoring techniques and define the roles monitoring plays in our lives both personally and professionally Understanding the types of monitoring and their impact on program operations Provide insight on the critical elements of monitoring Open Discussion
What Is Monitoring? Terminology Minefield Monitoring is a very widely used and ambiguous term Merriam Webster Monitoring is to watch, keep track of, or check for a special purpose General Definition Used to describe the process of checking or regulating some defined activity
Why Monitor?
Importance of Monitoring Through assessment, we know how well we are doing Through assessment, we know how effective and efficient we are Through assessment, we are able to monitor our resources You may go on using resources to no useful end
Why Monitor? Monitoring can be an invaluable management tool that enables you to assess the quality and impact of your work against your action plans
Why Monitor? Like riding a bicycle, you can adjust as you go along and ensure that you are on the right track
Why Monitor? Reasons Monitoring Is Not Popular It’s intrusive If it’s not broke, no need to fix it Other competing priorities No complaints, no problems Not necessary Like pulling teeth
Why Monitor? What If We Didn’t Monitor Our Health What We Eat Restaurants- Food Wardrobe Check-book Investments Pollution Homes Government Water Quality
Role of Monitoring Monitoring Is Used In… Project planning and implementation Analyzing the situation Determining whether the inputs in the project are well utilized Identifying problems facing the project or program and finding solutions
Role of Monitoring Monitoring Is Used To… Ensure all activities are carried out properly by the right people and in the allotted amount of time Apply lessons from one project experience on to other projects Determine whether project planning and problem solving is as appropriate and efficient as possible
Role Of Monitoring 4 Principles of Monitoring 1.Monitoring is the regular observation and recording of activities taking place in a project or program 2.Monitoring is a systematic and purposeful observation based on defined inputs 3.Monitoring involves giving feedback about the progress of the project or program 4.Monitoring enables the gathered information to be used in making decisions for improving project performance – Lessons learned
Role of Monitoring Monitoring Is A Feedback Loop For Example Checking to ensure that we did what we said we would do Checking the assumptions and estimates used to make decisions Checking on program quality and efficiency Feedback and continuous improvement
Role Of Monitoring Monitoring Can Be Carried Out To meet one or more of a wide variety of objectives By measuring one or more of a wide range of variables At different levels of resolution
What To Monitor What Are We Monitoring Program Requirements Sensitivity and Risk Analysis Can determine key variables in developing a monitoring plan and monitoring requirements
World of Monitoring What? Why? How? Training Performance Implementation plans Certifications Effectiveness Efficiency Validation Impact Onsite or remotely Records Interviews
World of Monitoring Strategic Plan ImplementMonitor Reflect Learn Decide Adjust Feedback
Types Of Monitoring The phrase “monitoring is used in a wide variety of situations” Examples Include: Fiscal Monitoring Validation Monitoring Compliance Monitoring Effectiveness Monitoring Administrative Monitoring Programmatic Monitoring
Program Monitoring Programmatic Compliance Monitoring Consists of 2 Components Administrative Compliance Asks the question, “how well does RWB administrative documentation reflect actual conditions on the ground before and after program activities took place?” Field Compliance Asks the question, “how well are the rules and the terms of the program being applied on the ground?”
Program Monitoring Programmatic Compliance Monitoring The primary focus of programmatic monitoring is: To assess program operational practices, procedures and processes for compliance with applicable federal and State laws, policies and guidance To ensure that policies and practices relative to program quality and outcomes meet the program’s objectives To assist the RWB in identifying problem areas and recommending alternatives and practices that will promote positive practices and outcomes
Program Monitoring Critical Elements Define population of interest and take a representative sample Compare estimates from the representative sample with specific benchmarks Various levels of resolution depends on objectives and target population
Sampling Techniques Statistical Sampling - probability sampling where every item in the population under review has a known chance of selection Judgmental Sampling – sample size or composition determined by professional judgment rather than the laws of probability Sliding Scale - approximates the sample size determined using a semi-statistical approach Bootstrap Method?
Sampling Techniques DEO’s program monitoring section uses a statistical, random sampling method to select program files to be reviewed for each program
Sampling Techniques NO MATTER What methodology is used, make sure that it is a valid representative of the population being reviewed
Tools &Guides Types Of Tools/Guides Used: Process Management Review Tool Administrative Process Specific Review Tool Compliance
Tools & Guides Submission To Regions Monitoring tools are normally submitted to RWB staff at the beginning of the annual monitoring cycle for review and feedback
Tools & Guides Process Management Review Tool Asks questions about program processes, design elements, and organization Allows you to become familiar with program operations Evaluates program design in the context of whether it is demand-driven Allows you to draw objective conclusions about program operations, including promising “Best Practices”
Tools & Guides Program Specific Tool Used to determine compliance with specific program requirements Serves as an indicator of whether objectives or requirements are being met Identifies issues that may be contrary to law or other requirements
Tools & Guides A separate tool is developed for each program monitored Areas covered by tools include questions related to: Training Services Support Services Credential Attainment Participant Eligibility Criteria Assessment and Plan Development Grievance/Complaint/EEO requirements
Local Initiatives Quality Assurance Establish goals and objectives on what you want accomplished Develop an action/implementation plan based on the goals, activity levels, targets, etc Develop local operating policies/procedures/guidelines based on core management functions and program operations Develop tools based on your objectives, rules, tenets of law and other guidance (one or more indicators to assist in determining if objectives have been met) Train staff
Local Initiatives Case Managers Ensure all files have required documentation Develop a checklist of what should be included in the files Make sure detailed case notes are entered in system Conduct peer reviews/evaluations Exchange case files for review by different case managers Check accuracy of source documents in files against system data
Summary Monitoring is a very ambiguous term that is used for continuous improvement Don’t worry about what a monitoring program is called Focus on The pre-defined objectives What is being done Why it’s being done What impact is it having Remember that monitoring is the feedback loop that can provide useful information in planning and making changes
References Phil Bartle, PhD Civicus J.S Thrower & Associates Ltd.
Questions? For More Information On Monitoring and Evaluations Please Contact Ken Williams at lorida.com