1. Is the Footprint Rightly Measured?
Challenges to evaluate environmental and social performance and impacts in private sector projects
Jouni Eerikainen
Senior Environmental Specialist
Independent Evaluation Group of the World Bank Group

2. Setting the stage Environmental & Social (E&S) performance evaluation at IFC
E&S sustainability is a strategic pillar at International Finance Corporation (IFC), EBRD and several other private sector multilateral development banks (MDB)
IFC’s 2006 Performance Standards (PSs) for E&S Sustainability have been adopted by Equator Principle Financial Institutions and EBRD
IEG has evaluated E&S Effects in IFC projects since 1996 and prepared E&S review reports with performance indicators since 2004
Annually evaluations, 10-15% with site visits
IEG’s evaluation approach follows ISO standard “Environmental Performance Evaluation”
Introduction of PS framework enables evaluation of E&S impacts as the change of performance indicators during the project course

3. Independent Evaluation Group
Independent Evaluation Group (IEG) Private Sector Evaluation (IFC and MIGA)
President
World Bank
Group
Board
Multilateral
Inv. Guarantee
Agency
Director-General,
Evaluation (DGE)
Public Sector
Evaluation
IEG’s organization changed in early 2011 and now both IFC’s and MIGA’s evaluations are conducted by a new unit, Independent Evaluation Group Private Sector Evaluation (IEGPE). This means that IEGPE evaluates now the Environmental and Social Effects for both IFC and MIGA.
Independent Evaluation Group
Private Sector
Evaluation
Country, Corporate
&
Global Evaluation
Strategy,
Learning &
communication

4. Environmental and Social (E&S) risks and opportunities in IFC projects

5. IFC’s self-evaluation system
Expanded Project Supervision Report (XPSR)
50% sample of mature projects (5years since approval)
Highly unsuccessful
Unsuccessful
Mostly unsuccessful
Mostly successful
Successful
Highly successful
1. Development Outcome
Satisfactory
Excellent
Unsatisfactory
Partly
Project’s impacts on:
Business success
Economic sustainability
Private sector development
E&S Effects
2. IFC’s Investment Outcome
Loan
Equity
Delete check mark and white line at SAT/Unsat
3. IFC’s Work Quality:
Screening, appraisal, structuring
Supervision
Role and contribution

6. Material compliance with
IEG evaluation instructions: rating Environmental and Social Effects (ESE) of real sector projects
RATING
Material compliance with
S&E Manag. System
(SEMS)
Beyond compliance Role model
Current Environmental & Social (E&S) standards (= IFC Performance Standards)
At-appraisal E&S standards & requirements (ERS, EIAs)
Excellent (E)
YES and YES
Excellent
Yes
Satisfactory (S)
YES or YES
Partly Unsatisfactory
(PU)
NO and NO
but deficiencies are addressed
Unsatisfactory
(U)
mitigation is unlikely or non-compliance resulted in environmental damage
No Opinion Possible (NOP)
Relevant information for rating is not obtainable, e.g. missing AMRs – last resort, consider a site visit
Not Applicable (N/A)
Category C projects with no E&S reporting requirements and no adverse impacts through the lifetime; if factual impacts are known then actual rating should be applied (infrequent)

7. Example of E&S Evaluation Summary A chemical plant in the East Europe and Central Asia region

8. Example of Environmental and Social Effects Evaluation – detailed analysis on air emissions

9. Environmental and Social Effects
ESE evaluation results on projects committed and evaluated
Environmental and Social Effects
The Environment and Social Effects ratings have done some rollercoastiong in past years and in 2010 the overall success rating was only 51% for the real sector projects. The FI sector rating was 60%.
IFC sampling is based on limiting the sampling error to five percent in three years rolling sample. Ins such statistics the ESE rating is still statistically in the range of long-term average, 67%.

10. By region – all projects
ESE evaluation results on projects committed and evaluated
By region – all projects
Better performance
Area of improvement

11. By region –development of all projects
ESE evaluation results on projects committed and evaluated
By region –development of all projects
ECA is constantly above average
Africa region has improved

12. By sector – all projects
ESE evaluation results on projects committed and evaluated
By sector – all projects
Good performance
Area of improvement

13. Most projects in process & manuf. industries with similar EHS aspects:
ESE evaluation results on non-FI projects committed and evaluated
Most projects in process & manuf. industries with similar EHS aspects:
Good performance
Info gap
Area of improvement

14. ESE evaluation results on FI projects committed and evaluated ; standard indicators
Good performance
but with info gaps
Area of improvement
Info gap
Info gap
Info gap

15. Evaluation results on real sector projects evaluated 2004 - 2010
IFC’s E&S Work Quality
at Appraisal and Supervision correlate with Environmental and Social Effects (ESE)
P<<1%; variables are very dependent

16. Evaluation results on real sector projects evaluated 2004 - 2010
Clients Social and Environmental Management System (SEMS) quality correlates with project Environmental and Social Effects (ESE), but no correlation was found between ISO and ESE
p<<5%; variables are dependent p>>5%; variables are independent

17. CESI Resource gap 99-04; only one person for FI sector
Evaluation results on 247 real sector and 168 FI projects evaluated
IFC’s E&S Work Quality at Appraisal and Supervision and E&S Role & Contribution
CESI Resource gap 99-04; only one person for FI sector
Good performance
Positive development with increased resources (5 persons 09) but still area of improvement

18. New challenges to evaluate E&S Effects of Performance Standards (PS) projects
PSs implemented since 2006 and evaluated from 2011
Environmental and Social Effects
Environmental and Social Performance
Meeting materially IFC requirements
Indicators rated at appraisal and supervision
Environmental and Social Impacts
Changes in performance
Wider impacts, sector & region
In the old system the E&S Effects evaluation focused on E&S performance, meeting materially IFC requirements because baseline was not often available to evaluate the E&S impacts.
With a better baseline from Environmental and Social Review Documents, we are now able to better evaluate the EA&S impacts as changes in performance. We should also assess the project’s demonstration effect and be able to present it as a role model.
The new evaluation system recognizes the E&S impacts and changes between appraisal and evaluation

19. E&S Effects Evaluation for Performance Standards projects1
Rate indicators at Appraisal (A) and Evaluation (●) 2
Rate PS 1-8 at Appraisal (A) and Evaluation (◙) 3
Rate E&S Performance at Appraisal (A) and Evaluation (◙)
The new template that IEG wishes you to use is based on the same idea of building a pyramid.
You first rate the indicators at Appraisal (A) and Evaluation (dot) based on the Environmental and Social Review Document as for example air emissions and waste management under PS3. Then you rate all Performance Standards and finally give ratings for overall E&S performance at the time of appraisal (A) and at the time of evaluation (square).
In this case there was a significant difference between appraisal (U) and evaluation (S) and thus the E&S impact has been large. You will consult the matrix E&S Performance at appraisal (green area) and at evaluation (yellow area), and in the crossing cell in the blue area you will find the symbol E slash E, which means Excellent E&S impact and Excellent overall Environmental and Social Effects.
This rating system for Environmental and Social Effects will give credit for those projects that have been started from a very low, Unsatisfactory performance, and ended to fully Satisfactory performance. This is now possible because we should be able to better catch the baseline performance of the project to evaluate the changes in indicators and overall performance. 4
Rate E&S Effects and Impacts based on rating matrix and demonstration effect

20. E&S evaluation of Performance Standards projects
Building a pyramid from indicators….
The E&S evaluation is like building a pyramid from indicators to measure project’s E&S footprint.
Each indicator is a theme under the PS as you have learned when filling the Environmental and Social Review Document. The themes are rated with four point rating scale (U,PU,S,E) and an overall rating is given for the Performance Standard.
I like this IFC’s Environment & Social Development Dept presentation of project performance as a footprint in an eight edge diamond. The footprint may be big in the orange area at the outset but with IFC’s assistance, it will be reduced during the project course to cover only the green Satisfactory or excellent area.
…to measure E&S footprint

21. Challenge to present E&S performance and impact concisely in complex projects
E&S Footprint is presented in a Performance Standards Diamond – key aspects at one glance
The goal is to show how IFC’s intervention has reduced the E&S Footprint area

22. Opportunities and challenges
The PS framework provides an opportunity to evaluate E&S impacts
Weighing indicators and calculating an overall score will be piloted
Challenges:
lack of baseline and monitoring data
corporate investments to several facilities
large number of performance indicators
evaluation of wider impacts in the region and industry sector
presenting evaluation results concisely and coherently in complex projects