1. Inter-American Development Bank BIMILACI 2007 QUALITY PROCUREMENT Third Party Review May 2007 Project Procurement Division

2. Independent Procurement Review SCOPE OF THE ASSESSMENT In 2006, Bank commissioned a consulting firm to carry out a pilot independent review of procurement process based on a small sample of six EAs of Bank-financed projects under execution in three countries. The review, organized in two phases, included two projects per country. Project Procurement Division

3. Independent Procurement Review PHASE I: Evaluation of institutional capacity of EA in procurement PHASE II: Design & preparation of training programs and materials for EAs Project Procurement Division

4. PHASE 1 - Objective a) Evaluate the proficiency of EAs to administer procurement in general and in Bank-financed procurement in particular; b) Verify compliance with Bank’s policies & procedures; c) Identify and rate specific shortcomings in each EA and potential risks associated with the procurement process; d) Identify means to strengthen the procurement capacity of each EA; and e) Identify and recommend best practices that could be replicated in other EAs elsewhere. Project Procurement Division

5. Methodology & Key activities METHODOLOGY: Assessments of the EAs were made in two overall areas: a) Procurement Cycle Management (PCM), and b) Overall Agency Management (OAM). a) PCM: three dimensions of best practices were evaluated, based on 66 criteria: i) Preparation (planning and documentation); ii) Evaluation of Bids and Proposals; and iii) Award and Execution Supervision. b) OAM: six dimensions were assessed as follows: i) Public Procurement Environment; ii) Agency Management; iii) Integrity Management; iv) Relations with Stakeholders; v) Human Resources; and vi) Finance and Controlling. KEY ACTIVITIES: The firm conducted site visits, interviews with borrowers, EA and COF personnel, and sampled procurement documents and procedures, as well as, other documentation covering the six projects in six EAs. Project Procurement Division

6. Projects selected for review Education Sustainable development in Rural Areas Agriculture Sanitary Protection Strengthening of Judicial System Municipal Development Infrastructure - Rural Roads Project Procurement Division

7. Documentation Reviewed Within the Executing Agencies: Procurement Plans; General Procurement Notices; Specific Procurement Notices; Bidding Documents; RFPs; requests for clarifications; Bid and Prequalification Evaluation Reports; Award Notifications; Contract amendments; any protest associated documentation; and EA Progress Reports Within the COFs: Project Reports; Loan Contracts; Institutional Evaluation Reports (procurement); Bank’s Inspection Visits Reports (procurement); Bank’s no- objection for each step of the procurement process; any protest associated documentation; and Project Progress Monitoring Reports (PPMRs). Project Procurement Division

8. PHASE 2 Based on the assessment made in Phase One regarding the need for capacity building and training for each EA ; In Phase Two: Consultant prepared individual training materials for each EA based on the assessment of their training needs, including basic and intermediate levels of procurement training In addition, lessons learned will be applied to develop a more comprehensive review and support program on a wider scale to identify gaps and to support EAs. Project Procurement Division

9. CONCLUSIONS & RECOMMENDATIONS CONCLUSIONS & RECOMMENDATIONS Sample for EA 1  Human Resources: Lack of consistency between training needs and the operational reality (not documented) and the lack of evaluation of staff, the agency has not taken actions to satisfy the staff needs. Curricula proposed: to cover Ethics, Social Accountability and Environmental Management System.  Activities: The impact assessment to stakeholders is not structured and implemented. Curricula proposed: to cover “Focus Stakeholder” and Quality Function Deployment (QFD)  System: Internal Audit Report covering the management aspects of the organization was not available. Curricula proposed: to include Balance Scored Card (BSC) and Quality Management System.  Supervision: Execution of Works Contract is followed using Excel tables. It is a simple follow up of progress against disbursement and vice versa. Recommendation: construction works of schools, should use MS Project to do the follow up of relatively complex operations. A training in MS Project could be provided to the technical staff.  Technical Specifications (TS): Lack of clear TS contributed to have bid price variation of 300%. Recommendation: Conduct specific training on writing Terms of Reference and Technical Specifications Project Procurement Division

10. Sample for EA 2 Activities: The four steps of the continual improvement < 70%. Due to lack of: procurement activities annual and monthly reports, regular meetings with stakeholders’ and low stakeholder satisfaction evaluation. Curricula proposed: Focus Stockholders (satisfaction), Internal management audit and Action plans & identification of causes. Human resources: The four steps of the continual improvement < 70%. Lack of: personnel management guidelines, training program, satisfaction evaluation of personnel.. Curricula proposed: Social Accountability and Environmental Management System. System: The steps “check” and “act” of the continual improvement < 70%. Lack of: Consistency check between its policies and values /principles, internal management audit. Curricula proposed : Documentation, Balanced scored card (BSC) and Quality Management System. Project Procurement Division

11. Recommended Capacity Building and Training for both EAs The training outline focuses on Procurement Cycle Management. Three levels of training (Basic, Intermediate and Proficiency) can be proposed, depending on the risk profiling analysis carried out by the consultants. The first two levels can be addressed through the provision of Training Materials while the final scope for the Proficiency level (more tailor-made) is to be determined on the basis of an entrance test. Recommendation for both EAs: - EA1 two aspects need to be dealt with: Agency Management and Procurement Cycle Management. (1) In Agency Management it needs BASIC TRAINING in: (a) Management System; (b) Human Resources Integrity Management; and (c) Relations with Stakeholders; - EA2 two aspects need to be dealt with: Agency Management and Procurement Cycle Management. (1) In Agency Management it needs BASIC TRAINING in: (a) Management System; (b) Human Resources/Integrity Management; (c) Finance and Controlling; and (iv) Relations with Stakeholders In Procurement Cycle Management, both EAs need to start with the BASIC LEVEL. Project Procurement Division

12. CONCLUSIONS & RECOMMENDATIONS CONCLUSIONS & RECOMMENDATIONS Main Strengths In EA’s procurement management practices included: –good documentation management and archiving; –sound knowledge of the Bank’s policy and procedures; –proper use of operations manuals; and –solid contract preparation. The EAs that received top ratings were ranked as: “Proficient” : An EA having an overall sound and solid management of the entire process with all safeguards and all procedures in place and full compliance with IDB procurement policies and standards. Project Procurement Division

13. CONCLUSIONS & RECOMMENDATIONS CONCLUSIONS & RECOMMENDATIONS Main Areas for Improvement In EA that received lower-end ratings, the main areas that need to be improved are: 1.Human Resources Policies: - codes of ethics, - whistleblower protection policies, and - lack of conflict of interest clauses; 2. Procurement Cycle Management: - weaknesses in bid management and evaluation caused by lack of trained personnel; - insufficient follow up on procurement plans; - inadequacy of written guidelines and manuals; and - insufficient mastery of IDB procurement policy and procedures. Project Procurement Division

14. CONCLUSIONS & RECOMMENDATIONS CONCLUSIONS & RECOMMENDATIONS Lessons Learned Lessons particularly singled out are: 1.At Project Preparation Stage: The need for better institutional evaluation of the capacity of EA, and 2.During Execution Stage: Closer monitoring of EA capacity and monitoring of policy and procedures compliance, Include parallel support activities such as training of EAs. Project Procurement Division

15. SCORES BY DIMENSIONS OF BEST PRACTICES Next two tables illustrate strengths and weaknesses encountered for each of the EAs included in the review. Strengths and weaknesses are benchmarked against nine dimensions of Best practices in Procurement. Project Procurement Division

16. SCORES BY DIMENSIONS OF BEST PRACTICES Project Procurement Division

17. SCORES BY DIMENSIONS OF BEST PRACTICES Project Procurement Division