1. Tracking Adaptation Measuring Development (TAMD)
Monitoring and Evaluation (M&E) of Climate Change Responses
Tracking Adaptation Measuring Development (TAMD)
Paul Steele and Neha Rai
For Asia Pacific Regional Forum in Bangkok, February 2018

2. Outline Why M&E for climate change responses? What is TAMD?
Country experiences
Lessons and challenges

3. Why climate M&E?
Climate adaptation investments increasing and there is the opportunity to compare and contrast them.
Evidence is needed to shape future investments
Developing countries need their own evaluative frameworks to be (a) able to judge best next adaptation investments, and (b) bargain harder for climate finance.
Governments need to report on delivery; evidence and benchmarking are a good starting point for effective planning & implementation at national scales.

4. Growing emphasis on M&E
Adaptation Committee meeting-Fiji emphasised the need to integrate M&E for adaptation into the national development M&E framework and to integrate learning from M&E as an important part of any evaluation exercise.
ADP text - Para 29 Monitoring and evaluation of, reporting on, and learning from plans, policies and programmes shall be strengthened and /or institutionalized
Green Climate Fund Paragraph 57: “The programmes and projects, as well as other activities, funded by the Fund will be regularly monitored for impact, efficiency and effectiveness in line with the rules and procedures established by the Board. The use of participatory monitoring involving stakeholders will be encouraged.”

5. M&E challenges in adaptation
Current focus on efficiency / outputs / coverage:
“Current results frameworks on resilience are not outcome-oriented and risk emphasising spending over results.”
Independent Evaluation Group of the World Bank (IEG, 2013)
In large part due to challenges that are particularly pronounced in this context…
No single metric:Adaptation needs, measures & goals are highly context specific.
Attributing changes to adaptation intervention is a challenge—requires robust methods and ToC.
Shifting baselines due to changing climatic context—require contextualisation
Longer term time scales—results from adaptation intervention are evident in the long run.
Data is limited for baselines and understanding climate contexts
No single metric: Adaptation needs, measures & goals are highly context specific. Vary with sector, location, timescale, livelihood contexts, etc.
Attributing Success of adaptation interventions, as outcomes may be as a result of other development indicators.
Shift in climatic data or severity and frequency of climatic changes can act as a confounding factor for development outcomes. E.g Mortality outcomes may be affected by severity in storms (variability).
Data is limited for baselines and understanding climate contexts
Requires cross-sectoral work
Not just something to add at the end
Long time frames

6. What is TAMD?
An evaluative framework for assessing the effectiveness of adaptation and adaptation-relevant interventions
Addresses outcomes and longer term impacts - can be used in planning.
TAMD logic:
Improved climate risk management decisions will lead to better (less damaged) development outcomes.
M&E of climate risk management and climate vulnerability enables improved decision making.
4. Started conceptual development
5. Country pilots in initial 5 countries over 2012 – 2014, expanded into new areas on demand (8 in total).
6. Evidence base and bespoke frameworks on adaptation M&E
Shortcomings in existing adaptation practices and associated results framework highlight a need for new approaches that:
Address the problems associated with assessing adaptation when timescales are long.
Address the problem of how to assess adaptation against a changing climate risk baseline. Will require contextualisation of development outcome indicators with respect to change exposure to risk, or the use of other indicators such as those representing vulnerability.
Moving beyond incremental, climate proofing. To transformational change. Linking capacity indicators with development impact indicators.

7. Tracking Adaptation & Measuring Development (TAMD) framework
A general theory of change linking climate risk management to adaptation & development performance
Institutions, policies, capacities
Monitoring, evaluation, verification, learning
Global
National
Regional
Local
Track 1: Climate risk management
Better CRM improves resilience, adaptive capacity, secures well-being, reduces losses in face of CC
Track 2: Adaptation & development performance
Monitoring, evaluation, verification, learning
TAMD is a “twin-track” framework that seeks to evaluate the extent and quality of climate risk management (CRM) processes and actions on the one hand (Track 1), and development and adaptation outcomes “on the ground” on the other (Track 2). TAMD is intended to be a flexible framework for evaluating adaptation and adaptation-relevant development interventions in diverse situations. It can be adapted according to the adaptation context. The two tracks are illustrated graphically on this slide.
Tracks 1 and 2 of the TAMD framework encompass a wide variety of interventions, processes, outputs, outcomes and impacts. Links between outputs, outcomes and impacts may exist within and between the tracks (Figure 2). For example, evaluation might seek to address the influence of outputs from CRM interventions at the national level (upper part of Track 1) on vulnerability outcomes and impacts at the local level (lower part of Track 2), or on CRM practices at the regional or local level (middle and lower parts of Track 1).
Interventions at the local level might also influence regional and national level CRM processes, for example through the generation of information and insights that inform national-level adaptation and development policies. Local-level interventions that are not directly related to CRM might nevertheless reduce vulnerability, for example by reducing poverty, improving educational outcomes, enhancing mobility and access to markets, increasing livelihood diversity and so on. Interventions with such “serendipitous” or indirect adaptation benefits might be associated with outputs, outcomes and impacts all located in the lower part of Track 2.
Populations, systems (natural, economic, managed, etc.)
TRACK 1: Improved capacity of institutions to anticipate, plan for and respond to climate-related hazards & risks
TRACK 2: Greater resilience, better well-being, improved functioning, reduced losses, etc.

8. 7 Steps to evaluating an adaptation intervention
Step 1: Define the evaluation context
Define the purpose of evaluation
Entry point, relevant scales (global, national, regional, local).
Step 2: Establish a Theory of Change
Define logical change pathways
Identify relevant outputs, outcomes and impacts and assumptions for change.
Step 3: Identify type of Indicators
Identify and define indicators for outputs, outcomes and impacts
Identify relevant indicators for normalisation and contextualisation
Step 4: Define Methodologies
Define methods to be used for data gathering and analysis.
Step 5: Data gathering
Establish Baseline data, gather data at regular intervals and collect stakeholder narratives
Step 6: Analysis
Analyse indicators and data at different levels of Track 1 and Track 2
Address attribution, comparability.
Step 7: Lessons learnt
Disseminate lessons learnt from M&E results
Iterative learning. Integration of M&E results into national or sub-national planning.

9. Multiple projects/ Programmes
Where is TAMD being applied?
Multiple projects/ Programmes
Large Programmes
National level
Pakistan
Nepal
Ethiopia
Cambodia
Uganda
Kenya
Bespoke application (a) National (b) Local © multiple portfolio (d) Single intervention level.
Local level adaptation planning : Kenya, Tanzania and Zanzibar, Mozambique. Kenya - County Adaptation Fund
National level M&E
Cambodia: supporting the design of national M&E framework.
Portfolio of programmes:
Nepal: effectiveness and linkages of 3 country level investments
Local governance and Community Development Programme (LGDCP)
Livelihoods forestry programme
LAPA – NCCSP.
Single intervention:
Pakistan: Rainwater harvesting programme
Bio Gas programme.
Sub national level
Zanzibar
Mozambique

10. Cambodia Support to national climate M&E system and linking to PPCR
Developing institutional indicators and vulnerability index
Started on main strategy now working on public works and transport and health sector

11. Improvements in CRM at national level Institutional readiness
Cambodia
Improvements in CRM at national level
TRACK 1
National
Institutional readiness
indicators (PROCESS)
Indicator 1: Status of dev. of national CC policies/strategies/ action plans.
Indicator 2: CC Climate Integration into planning: NSDP/PIP CC
Mainstreaming
Indicator
Process indicators
Sectoral
Indicator 3: Coordination: Establishment and functionality of a national coordination mechanism.
Indicator 4: Climate information: Production, access & use.
Indicator 5: Climate Integration into financing: Availability and effectiveness of a Financial Framework for Climate Change response.
Indicator 2: Institutional arrangements for integration of climate change in national development planning and budgeting.
A2.1 (core). Degree of integration of climate change in national planning, including sector planning. B2 (core). Evidence of strengthened govt capacity and coordination mechanism.
A2.2 (optional). Changes in budget allocations at national and possibly subnational level of government to take into account effects of CV [climate variability] and CC [climate change.
B3 (optional). Evidence showing that climate information products/ services are used in decision making in climate sensitive sectors
Simplified CRM at the national and sectoral level. Located outputs in the upper part of Track 1
Identified the key Ministry responsible for CRM
And then Sectoral line ministries responsible for CRM at the national – e.g. Ministry of public works, ministry of Agriculture
Out of the 8 Indicators we have simplified and reduced the indicators to 5 in case of Cambodia- at the national level. Subjective questions on extent of capacity, stakeholder awarenss etc have been removed, which are better measurable at the project and programme level- lower scale of evaluation.
Indicator 5: Climate Integration into financing.

12. Example Indicator Scorecard
Use of indicators on extent and quality of CRM
Assessing how, and how well the CRM benefits the climate vulnerable
Use of standard development indicators to see if development is “on track”
Indicators to show changes in vulnerability, resilience and adaptive capacity

13. Track 1 Indicators- Approach
A Ladder based approach - The sub indicators comprise of ladder based parameters that reflects how the country is moving towards achieving its institutional milestones.
Use scorecards- each ladder is scored using ‘Yes’ (2) or ‘No’ (0)or ‘Partial’ (1). A weighted percentage is calculated.
Narratives- Scores will be aggregated to yield an overall score. Narratives are also used to understand the reasoning behind the score.
A ladder based approach is used to understand how Cambodia stands in the overall process of climate change policy and institutional development and how the country is moving towards achieving its milestones. The ladder describes the process that the indicator is measuring starting from the initial phases, even if they have already been completed.
The ladder represents how country is adding policy and institutional building block to address its climate risks and respond to climate change. The ladders are not necessarily incremental.

14. Supporting evidence/narrative
Indicator 5: Availability and effectiveness of a Financial Framework for Climate Change response. `
Step
Milestone
Y/N/P
Supporting evidence/narrative
A national pilot trust fund for climate change is established.
Yes
A climate public expenditure review is conducted jointly by the MoE and Finance, and a reference baseline for climate finance is available.
An inter-ministerial sub-working group on climate finance is established
A Climate Change FW is approved. No
Not yet. Under development.
A national fund for coordinated management of climate finances is established.
Coordinated funding arrangement for climate change response exists. (E.g. a coordination mechanism for budgetary and extra-budgetary resources).
Its project based at the moment.
A Climate Change Expenditure Review is regularly conducted and is included in the CCCSP progress report.
A code to track climate expenditure is established and is consistently applied to produce regular climate expenditure review in CDC/CRDB ODA Database.
A code to track climate relevant expenditure is established and is consistently applied to produce regular climate expenditure review in NCDD-S / Sub-national funds
A budget code to track climate relevant expenditure is established and is consistently applied to produce regular climate expenditure review in the national budget.
Formal Procedures are in place in MEF for screening major national budget investment against climate risk.
Budgetary and extra-budgetary resources mobilized are between 30%-50% of the annual requirements identified in the CCAP
Budgetary and extra-budgetary resources mobilized are at least 80% of the annual requirements identified in the Climate Change Action Plan.
Total score: (3Y*2)/13*2*1=23%
Identify the roles of department
Identify how well they are managing
Identifying gaps in management.

15. Track 1 Indicators- Outcomes of scoring
PPCR &GCF
 Indicator
2014 baseline 1
Climate policy and strategy
30% 2
Climate integration into development planning
25% 3
Coordination
45% 4
Climate information
17% 5
Climate integration into financing
PPCR & GCF

16. Change in Vulnerability
Track 2: Development Impact Indicators
Indicator 1: Vulnerability Index
Aggregation
Impacts
Dev indicators -
Change in Vulnerability
Indicator 2: Damage and loss from extreme climate events
Impact indicator
Results indicators
Cambodia will track national vulnerability reduction as a response to CC measures.
TAMD is providing support to develop appropriate vulnerability index
Cambodia has 3 indices – disaster risk index, climate vulnerability index, vulnerability index.
TAMD is refining the index by statistically analysing the strength of correlations between predictive indicators of vulnerability (outcome indicators) and Impact indicators (e.g. losses and damage).

17. Cambodia Hazard specific vulnerability indicators (Track 2)
Context (indicative)
Storm (Vs)
Flood
(Vf)
Drought
(Vd)
Overall Index
(VToT)
VI= V(Flood) + V(Storm) + V(Drought)
The TAMD team is currently providing support in developing appropriate methodologies for analysing and processing existing information to produce and refine compound vulnerability indices. Cambodia presently has three indices of vulnerability – disaster risk index, climate vulnerability index, vulnerability index - which are composite indices comprising of socio-economic indicators, health indicators, business indicators, etc. The TAMD team is working to refine this index by statistically analysing the strength of correlations between predictive indicators of vulnerability and outcome indicators, such as those which measure the on-ground loss and damage impacts of hazards associated with climate change. If indicators of vulnerability, resilience, and adaptive capacity are good, they should be able to predict variations in impacts across populations exposed to the same hazards. These analyses help to identify the indicators which are most important so that they can then be streamlined. Contextualisation by landscape type (urban/rural), hazard, and geographical zones will be the next step to arrive at a refined vulnerability index, disaggregated by hazards.

18. Pakistan: impacts on development (track 2)
Retrospective assessment of earthquake rehabilitation progamme and biogas
Focus on gender and resilience
Looking at province level frameworks

19. An Example of the Theory of Change in the TAMD Pilot in Pakistan (track 2)
Socio-economic and human development
&
Resilience to climate change
i.e. uncertainty of water availability
Kitchen gardening, animal rearing or other economic activity
Increased income + food security
Reduced workload of fetching water
Rain Water Harvesting
Water available for latrines at homes and schools
Higher school enrolment + retention
Increased water availability at household/ community level
Health units functional
Improved health
Theory of change for measurement (Ex post)
What are the assumptions about how the adaptation activities or climate risk management system will lead to the desired outcomes?
What are causal mechanisms that lead from inputs to outputs to outcomes and from outcomes to impacts.
On which track(s) are the outputs, outcomes and impacts located?
Where on the track(s) are the outputs, outcomes and impacts located (i.e. which scales)?
Challenges with establishing ToC at the local
Based on assumptions. Assumptions may be not always be accurate.
Application at the local level is more based on stated local theories of change. Empirical or theoretical evidence maybe lacking.
Resource intensive, based on focussed group discussion with different stakeholders
1ISET 2014:

20. Nepal: measuring impacts
How to track all these things at the national level, to measure progress against national objectives whilst making meaningful conclusions from different indicators and M&E frameworks?

21. Nepal: track 1 and track 2
Focus on district and village level councils
Nepal adaptation and development focus through decentralised planning mechanisms and community-led plans. Seeking to link these up and aggregate them in any easily trackable way so the GoN can assess what gains are being made across the country both in terms of institutions and also in terms of development outcomes at VDC/DDC level.
Looking at 3 main hazards – floods, droughts, landslides, and 3 interventions.
One – Local Governance and community development programme is a national programme on local institutional strengthening and participatory ward committees
Two – Livelihoods Forestry Programme, long standing programme around forest management
Three – climate specific planning (LAPAs)

22. TAMD: Lessons learnt and challenges
Country bespoke approaches for government buy-in, which is crucial.
Application at different scales is fruitful-National, local, subnational
TAMD Track 1 is useful for shorter term immediate indicators
TAMD Track 2 flexible enough to work in two directions – adaptation to development, development to adaptation
Linkages between adaptation and development outcomes shape the need to use TAMD by involving planning and finance ministries
TAMD relevant for NDC reviews and measurement of adaptation goal (adaptive capacity (track 1) and improved resilience (track 2)
Challenges
Can be difficult to assess attribution so assess contribution
Survey level analysis can be resource intensive, so existing data needs to be used

23. Published frameworks
Research reports
Step by step guide
Briefings on gender, scorecards, resilience, climate data

24. Thank you