Evaluation and Monitoring Methodologies Strengthening the Legislature – Challenges and Techniques K. Scott Hubli, NDI

Overview General Comments on Monitoring and Evaluation Special Considerations in Monitoring and Evaluating Legislative Strengthening Programs Practical Tips and Considerations

General Comments on Monitoring and Evaluation Evaluation (and Baseline Assessments) --Use to develop program design; use for major course corrections --More costly and less frequent than monitoring (every two to three years) --Typically done at the beginning and the end of a program, but often also done after a major change in the political landscape (e.g., regime change, ethnic conflict settlement, etc.) --Used for accountability to partners, donors, stakeholders, not for ongoing project management

General Comments on Monitoring and Evaluation Performance Monitoring --Ongoing monitoring; used to manage performance of implementation --Track changes (but less analysis) -- Informed by baseline assessment and, if well designed, it can reduce future evaluation costs -- May indicate a need for a evaluation or updated baseline -- Focus on low cost, regular data collection (workshop evaluations, information available from parliament, regular focus groups, etc.)

General Comments on Monitoring and Evaluation Always distinguish among: --Inputs (e.g., consultants, computers, etc.) --Outputs (e.g., 40 people trained in a workshop on oversight techniques) --Outcomes (e.g., increased knowledge of oversight investigation techniques) --Objectives (e.g., increased oversight hearings) --Goals (e.g., increased government accountability)

How are legislative strengthening programs different from other programs with respect to monitoring and evaluation?

Special Considerations in Monitoring and Evaluating Legislative Strengthening Programs Legislatures are highly complex institutions --They involve multiple actors seeking to achieve multiple goals simultaneously --Where possible, disaggregate data (by gender, party, region, etc.) --Identify clear goals and targeted groups; watch for unintended consequences

Special Considerations in Monitoring and Evaluating Legislative Strengthening Programs Long-term goals, short-term programs -- Resist the tendency to monitor outputs rather than progress in achieving desired outcomes, objectives and goals -- Find ways to measure small changes in large goals; or outcomes that can be affected with the project time frame

Special Considerations in Monitoring and Evaluating Legislative Strengthening Programs Programs focus on process, not outputs --Example: number of laws passed --Emphasize qualitative over quantitative information --Use detailed process descriptions in establishing baselines --Use monitoring and evaluation to help strengthen this process and to teach results-based management, where possible (“Monitoring and evaluation should be managed as joint exercises with development partners.”)

Special Considerations in Monitoring and Evaluating Legislative Strengthening Programs Monitoring and evaluation is often highly political --Involving partners can sometimes further politicize evaluation and monitoring; use caution and judgment --Can be hard to get necessary information --Politics may cause people to be less than fully honest --Results can be used as a political weapon

Special Considerations in Monitoring and Evaluating Legislative Strengthening Programs Legislatures have natural cycles --Elections, post-election learning curves, legislative floor periods, recesses, budget processes, etc. --Example: constituency relations --Expect uneven development in performance monitoring, but try to attribute fluctuations in data --Time evaluations carefully – look for “normal” periods

Special Considerations in Monitoring and Evaluating Legislative Strengthening Programs Many intervening variables --Economic conditions, geopolitical developments, ethnic conflict, death of a key politician, etc. --No substitute for nuanced political analysis --Measure outcomes, objectives, goals – not just outcomes; this can help identify these intervening variables

Special Considerations in Monitoring and Evaluating Legislative Strengthening Programs Perceptions matter --Importance of qualitative over quantitative indicators --Use of focus groups, opinion polls, etc. --Even anecdotal evidence is useful if it captures a political mood or issue

Special Considerations in Monitoring and Evaluating Legislative Strengthening Programs Difficulty of comparative benchmarking --Only one national legislature; cross-country comparisons are of limited utility --Comparisons across time more important; use of thorough baselines --Implications on setting goals and targets – use of reasonable/consensus expectations

What are some practical strategies for dealing with these unique aspects of monitoring and evaluating legislative strengthening programs?

Practical Tips & Considerations General Issues --Be pragmatic in designing an evaluation or monitoring plan; tie evaluation and monitoring to the purpose or objectives. Avoid evaluation for evaluation’s sake. Consider: --resource availability for evaluation --novelty of the program --confidence in program design or implementation --needs of funder --Budget sufficient resources – (Costs for legislative strengthening evaluation may exceed those for other program types – soft assistance, new field, etc.)

Practical Tips & Considerations Issues in Doing a Baseline --Limit scope to allow for detailed coverage of program areas --Protect against biases of person(s) doing the baseline by: --Using teams --Using clear, detailed terms of reference --Incorporating documentary evidence --Seeking consistency in future assessments

Practical Tips & Considerations Issues in Doing a Baseline (cont.) --Pick timing carefully; describe any special circumstances --Prepare carefully for baseline assessment team --Cover the range of stakeholders --Get out of the capital --Consider focus groups or creative methods for documenting perceptions and processes (e.g., a sample of 10 legislators to track periodically every 3 years) --Pay attention to protocol; build good will.

Practical Tips & Considerations Using outside evaluators --Outside evaluators can not only provide objectivity but also insulation from the political consequences of an evaluation --Combine multiple backgrounds (academic or legislative strengthening specialists and MPs or staff from similar systems) --Recognize value of “time in the trenches” --Designate a lead person with responsibility for producing the document --Get a sufficient time commitment

Practical Tips & Considerations Issues in Performance Monitoring --Draw on baseline and prior evaluations --Design performance monitoring plan up front; adjust it as project evolves: --Imposes discipline; keeps program on track --Provides clarity of expectations to partners --Keep it current, modify as needed --Make these changes explicit

Practical Tips & Considerations Issues in Performance Monitoring (cont.) --Tie to likely performance issues --Draw on low-cost existing information sources; may be more quantitative, with less analysis --May focus on outcome level, rather than objective or goal level --Consider quarterly or semi-annual monitoring --Expect, but explain, fluctuations --When you can’t explain repeated fluctuations, consider updating a baseline to try to identify issues --Often done, in part, by those implementing program

Final Thoughts Be creative: legislative strengthening is an art, not a science Be willing to accept criticism; fight structural bias for “spinning” results Share lessons learned, both internally and externally