MODULE 8: PROJECT TRACKING AND EVALUATION REHABILITATION ENGINEERING RESEARCH CENTER ON TECHNOLOGY TRANSFER (T2 RERC) TRAINING MODULES ON TECHNOLOGY TRANSFER

Objective of the Module: Participants will: explain basic concepts and terms used in program evaluation/research; define indicators of effectiveness and efficiency for a project of their interest suggest methods to track the indicators; identify how the results would be used for project monitoring and evaluation Slides 13-15

BASIC CONCEPTS AND TERMS Research and Evaluation Merit and Worth Management Orientation: Evaluation for Development Effectiveness and Efficiency Formative and Summative Evaluation Outcome and Impact

Merit and Worth Merit refers to intrinsic quality of person/ product/ system its potential, or promise to be “valued” in its broader context. Ex: Program inputs (resources, design, theory); processes (operations) and outcomes indicate its merit Worth refers to extrinsic value of person/ product/ system its actual “value” in the broader context Ex: program effects/ impacts on internal and external environments indicate its worth.

Fig 2. Evaluation and the Planning Cycle ↓ ← Context Evaluation Cycle Planning Cycle ß ↑ What needs to be done? [ Defining objective] Product Þ Are we there yet? How to do it? Recycle output or [Defining Ü Input disseminate? resources ] Evaluation ↓ ↓ What is the ↑ ↑ ↓ ↓ optimal path ↑ ↑ ↓ → → to get to → → ↑ ↓ goal? ↑ ↓ [process ↑ ↓ improvement ↑ ↓ ] ↑ ↑ ↓ Ý ↑ ↓ ↑ ↑ → → → → → → → Process → → → → → → → → Evalu ation

System characteristics: INPUT ---- à P ROCESS ---- à OUTPUT ↑ ↓ ← ← ← ← ← ← ← FEEDBACK Effective: gets outcome at the desired quality/quantity level Efficient: Maximizes effectiveness, minimizes cost System characteristics: .Uses the “input” resources; .“Processes” them [facilitates and manages the dynamics of their interaction]; .In function of a pre-defined objective [i.e., effort directed to achieving the corresponding output] .Does “self-regulation”[i.e., checks/obtains feedback information about the product taking shape, constantly adjusts its processes so the desired quality of product is obtained]. Efficiency =Output/ Input Slides 13-15

Formative vs. Summative Evaluation Assesses quality of outcome during the process, while outcome is taking shape Provides basis for improving/maintaining program actions SUMMATIVE Assesses quality of outcome at the end of process when outcome is in final stage Provides basis for establishing program effectiveness & efficiency; [accountability]

Outcomes and Impacts Outcomes direct outputs or immediate effects assess program’s performance- its effectiveness and efficiency; an indicator of merit Impacts long term effects; benefits to staff, participants, other stakeholders and community at large assess program’s validity/relevance; indicator of worth

Framing program evaluation: three basic questions Is the program effective? - has it achieved its goals? Is it efficient? - did it optimize its methods, resources to achieve the goals? Is it valid? - are its goals relevant to the context? Is it valued by its stakeholders and beneficiaries?

In Response: what to assess Effective? --- Outcome quality; outcome quantity (for duration of program) Efficient? -- Cost per outcome; Time per outcome Valid? Worth it? -- internal and external impacts; long term effects on/ benefits to stakeholders and others involved NOTE: outcome,cost/time [merit indicators] - assessed through tracking program performance. Impact [worth indicator]: - assessed through tracking effects beyond program period

Slides 13-15

Slides 13-15

Slides 13-15

Use of Information Monitoring project Performance Improve intermediary outcomes [quality vs. timelines] Improve process re-distribute personnel review protocols/timelines Describing performance level -[effectiveness and efficiency]

Questions, Comments, Suggestions Related Issues Questions, Comments, Suggestions

Workshop Session Two For your project, identify, or discuss applicability of: Goals, Protocols, Action Plan (or similar framework) Expected Outcomes - final [and intermediary, if applicable] Outcome Tracking: (a) process and (b) use- improving outcomes (formative eval.) and finding effectiveness (summative eval.) Tracking resource utilization (time/cost …): (a) process and (b) use for process improvement and finding efficiency (summative eval.) Instrument and database alternatives