Role of Training in Program Evaluation: Evidence from the Peace Corps Projects Shahid Umar Ph.D. Candidate Rockefeller College of Public Affairs University at Albany, State University of New York

Outline Background Research questions Method of study Results Recommendations Conclusion

Background Most of evaluation capacity building (ECB) efforts include training as ECB strategy Using a case study method, a large number of studies have concluded that training as ECB strategy was successful

Background The effectiveness of training as ECB strategy has not been assessed using data from multiple sites (projects) in an international context The present study fills this gap using data from 184 projects of the Peace Corps operating in 72 countries (posts)

Background: The Peace Corps (PC) U.S. government agency Sends volunteers to developing countries unlike other international development agencies The PC provides a few weeks of training to both volunteers and staff Training includes a component of monitoring and evaluation Training about monitoring and evaluation is not more than a couple of days

Research questions To what extent does training induce the collection of baseline data and the use of these data for evaluation purposes? Do the Peace Corps posts have sufficient capacity to undertake impact evaluation?

Method of study The data come from the 2012 Project Status Report (PSR) survey of the Peace Corps Open-ended questions were asked about: – Baseline data collection – Baseline data use – Data collection tools – Baseline data storage – Training – Support needed from headquarters – Other comments

Method of study Overall qualitative responses were used to generate frequency and proportion statistics Comments were analyzed qualitatively and incorporated in the results

Results: Univariate (N=184) VariableYesNo Baseline data collected 0.85 (157)0.15 (27) Data collection tools 0.81 (149)0.19 (35) Baseline data stored 0.76 (139)0.24 (45) Baseline data used 0.75 (138)0.25 (46) Training given 0.72 (132)0.28 (52) HQ support needed 0.81 (149)0.19 (35) Unit of Analysis: Project

Training given Baseline data used (p=.92) Baseline data NOT used (p=.08) Training NOT given Baseline data used (p=.31) Baseline data NOT used (p=.69) Probability of using or not using baseline data in case of training given or not (N=184) Unit of Analysis: Project

Training given Baseline data NOT used (p=.07) Baseline data NOT collected (p=.01) Baseline data used (p=.93) Baseline data collected (p=.99) Training NOT given Baseline data not used (p=.38) Baseline data NOT collected (p=.50) Baseline data used (p=.62) Baseline data collected (p=.50) Conditional probability of using data in case of training given or not given (N=184) Unit of Analysis: Project

Results Projects receiving training are more likely to collect and use baseline data As expected, there is a strong positive association between staff training (as ECB strategy) and professional evaluation practices

The Peace Corps EC? (N=184) VariableNo Baseline data collected0.15 (27) Data collection tools0.19 (35) Baseline data stored0.24 (45) Baseline data used0.25 (46) Training0.28 (52) Unit of Analysis: Project Given these numbers, the Peace Corps does not seem to have reasonable EC and is not ready to undertake impact evaluation

Recommendations for posts (country level) Training, in addition to other strategies such as technical support and ICTs, should be used for ECB Improvement should be directed at post (country) level, not project level Greater monitoring and evaluation expertise should be deployed to the PC posts

Recommendations for the PC headquarters Leadership support Best practices, already present at some posts, should be more widely shared Standardized practices across all the posts should be promoted (e.g., data collection tools, data storing) Centralized data management system: – Data should be collected using web surveys – Quantitative responses should be sought in surveys

Conclusion There are some good news for the Peace Corps: 85% of the posts collect and 75% use baseline data But also a need for more substantial efforts to prepare for and undertake impact evaluation The study provides empirical evidence that training is necessary but perhaps not sufficient for ECB