1. Pathfinder Kick-off Meeting Research Methodology Introduction, Cardiff 2 nd March 2011 Richard Redfern, Steve Tarry 2 nd March 2011

2. Welcome and Introductions from presenters and participants

3. Content: General introduction to the day Introduction to the research to be undertaken Evaluation overview Baseline and contexts for the research Project Officers’ role – the monitoring, recording, reviewing and reporting of data and relevant information

4. General Introduction to the Day: A recap on what was covered in day 1: –Study participants, roles and responsibilities –Pivotal role of Pathfinder Officers (POs) Aim of day 2: –To present an overview of the planned research –Explain how POs will help in developing the evaluation approach and populating the study Evaluation Framework –Address any questions or concerns about what is proposed –Begin work on developing some of the tools required How the day will run: –Presentation of particular topics associated with the research methodology –Discussion of PO role in particular and any issues arising –Interaction required to ensure POs have confidence in what is required

5. Hand Out Agenda at This Stage

6. General Introduction to the Day (cont): Agenda: –Before lunch: Introduction to the research and overview of evaluation approach –Lunch 12.30 – 13.15 –After lunch: Specific context for the study, the role of case studies and the work of the POs –Finish at 16.15 Expected outcome: –insight into what will constitute an appropriate evaluation programme and an clear understanding of PO role Required deliverables: –Tools required to support POs in their particular role What will follow on: –Day 3 - Exploration of individual sectors within the study –Meetings with host authorities to being process of engagement

7. Introduction to Planned Research The objectives of the Pathfinder Community Research project are to: –Generate robust evidence of which projects work in reducing greenhouse gas emissions; –Support the trial of different approaches to community action on climate change; –Evaluate the different ‘models’ of community based project interventions; –Evaluate the delivery of the pathfinder project programme by the programme management team; and –Assess the role of Pathfinder Officers in delivering programmes. These are to be achieved through the evaluation of 18 case study initiatives spread across 6 Local Authority areas. The exact nature and location of the Case Studies is to be determined during this scoping phase of the project –this will define the structure and scope of the research activities undertaken in subsequent phases of the project

8. Introduction to Planned Research (cont) In each of the six Local Authority areas the following outputs are to be achieved by the end of the two year pilot: –A ‘map’ of communities who are already engaged in action and those which have potential and/or have demonstrated interest in engaging in action in the future. –A ‘map’ of the communities who could be persuaded and encouraged to take action –A record of carbon reduction work being undertaken by the community –Specific community groups identified as willing to reduce their carbon footprint –Baseline data for the carbon footprint of the willing community –A carbon reduction target of 10 to 20% in the willing community within two years

9. Research Approach The research will look at initiatives within different sectors –Energy –Transport –Water –Food Research will look at behaviour changes / effects achieved by initiatives both within and across different sectors ‘Theory of Change’ will be used to explore the differences noted and the logic behind what has transpired The detailed approach will need to reflect the specific nature of the initiatives that are taken

10. Participants to think of the nature of the initiatives that might be taken: Energy Transport Water Food Record these; then Review with results of previous Brainstorm – as summarised on next slides

11. Possible Initiatives? Energy  Energy efficient appliances  Energy efficient equipment  Energy efficient lighting  Energy efficient homes  Green buildings and homes  Green business  Green energy  Retrofitting insulation  Programmeable thermostats  Energy conservation Transport  Anti-idling  Biking  Carpooling  Fuel efficient vehicles  Mass transit  Telecommuting  Vehicle maintenance  Walking

12. Possible Initiatives? Water  Drinking water  Greywater  Lawn watering  Low flow showerheads  Low flow toilets  Maintaining wells  Water tanks  Water efficiency  Water restrictions Food Local food  Fertiliser use  Reducing meat and dairy consumption  Biodiversity  Forestry  Pesticide use  Organic food  Reduce packaging

13. Theory of Change Robust framework for considering: What has changed ? Why ? In what context ? Tailored to address complex interventions, packages or programmes ‘Mapping’ produced to record and test underlying logic of investment: Links between investment and short term outcomes Hypotheses of links between short and long term behavioural change Work with stakeholders to review mapping and recognise their role within individual ‘blocks’ and in relation to specific ‘links’

14. Logic Mapping – the Theory: What is invested, e.g. money, skills, people, activities What has been produced Short and medium term results Short and medium term results Long-term outcomes Input Output Outcomes Impact The frame-work within which an intervention is located Context Analysis of context Analysis of objectives Analysis of the logic of the intervention

15. Logic Mapping – an Example

16. Hand out Spreadsheet example for capturing appropriate data from which to develop appropriate Logic Map

17. Practical Example - Specific to the Study Activities Short Term Outcomes Medium Term Outcomes Long Term Outcomes Cycle Training for Year 5/6 Children More positive attitude to cycling by parents/children Increased cycle skills Increased risk awareness Increased propensity for parents to let children cycle Increased cycling to school Reduced accidents involving child cyclists Reduced carbon emissions Cycle Training for Year 5/6 Children More positive attitude to cycling by parents/children Increased cycle skills Increased risk awareness Increased propensity for parents to let children cycle Increased cycling to school Reduced accidents involving child cyclists Reduced carbon emissions Poorly delivered training Course not well designed Low take-up rate Misses ‘at risk’ children

18. Role of Case Studies Case Studies will give us practical examples of Inputs and Outputs Inputs are the investment, resources and processes applied in the implementation of case study interventions, including Local Authority and other stakeholders’ time and investment. Accurately monitoring, analysing and reporting the full range of inputs will be a primary focus for the evaluation framework Outputs are the initiatives delivered, as a result of the inputs directed towards the Pathfinder programme. These will cover the range of intervention types (to be defined), including transport, energy, water and food

19. Categorisation of Initiatives Categorisation of initiatives: –provides a focus for data collection –assists when reviewing the transferability of findings about similar types of interventions. Typology of initiatives according to, for example: –Primary objective –Primary target population –Scheme ownership –Geographical scale.

20. The Research Programme Short term –Engagement –Identify possible Case Studies –Identify data sources –Set up evaluation processes –Identify Indicators of success (and possibly failure) Medium term –Data collection –Look at inputs and outputs –Review evaluation approach Longer term –Interim Look at outcomes and impacts Review research approach –Final Look at outcomes and impacts Address specific research questions

21. Q and A

22. Evaluation Overview - What is Evaluation? “Evaluation is the post implementation assessment” of: Programmes Policies and strategies Investment ‘packages’ Individual schemes/interventions Two key areas in evaluation: Outcome or Impact evaluation Process evaluation Both apply to the Pathfinder Research

23. Evaluation within the Programme / Project Lifecycle Rationale ObjectivesFeedback Appraisal Evaluation Monitoring Implementation

24. Benefits of Evaluating BenefitDetailed Consideration Planning and Policy Setting Improve carbon reduction plans and policies Improve decision-making for policy makers and communities Identify benefits and impacts of individual schemes Implementation Improve the efficiency of carbon management interventions Better policy delivery across sectors (transport, water, energy) Develop delivery processes for businesses and community groups Target engagement approaches and activities Institutional Strengthening Improve co-operation of partners Promote inter-partner working and carbon management Accountability Evidence and justification for investment (emission reductions) Ex-post evaluation of value for money and effectiveness Carbon trading and management outcomes Knowledge Production Identify what has worked well in different circumstances and why Contribute to the ongoing learning process Develop and refine project development procedures Learn lessons

25. Relationship to Logic Mapping Impacts Outputs Inputs Objectives Outcomes Rationale Rationale – Policy context Objectives – Local priorities for investment Inputs – capital investment and human resources Outputs – delivered interventions Outcomes – changes in key indicators Impacts – longer term changes Traditional realm of monitoring Realm of evaluation

26. Phases of Delivery Phase 1: Evaluation Scoping Phase 2: Interim Evaluation Phase 3: Outcome Evaluation This phase defines the specific focus of the evaluation and, importantly, the interventions to be included. It includes the development of the evaluation framework and its constituent parts. This phase will identify existing evidence gaps and design data collection activities to fill these. Initial stakeholder and community engagement will be undertaken and process evaluation work will be undertaken. Ongoing data collection and detailed process evaluation activities undertaken during and after the implementation phase of measures. This will include further engagement and action research activities.

27. Phase One (Scoping) – Overview Evaluation Scoping Define interventions and packages Confirm evaluation objectives Select research questions Select evaluation indicators Select core indicators Select contextual indicators Stakeholder liaisonConduct ToC mapping Identify new data needed Define evaluation approach ToC mapping

28. Objectives of the Research The objectives of the Pathfinder Community Research project can be grouped around three general themes: Outcomes: –Generate robust evidence of which projects work in reducing greenhouse gas emissions; Engagement : –Support the trial of different approaches to community action on climate change; –Evaluate the different ‘models’ of community based project interventions; Processes: –Evaluate the delivery of the pathfinder project programme by the programme management team; –Assess the role of Pathfinder Officers in delivering programmes. The nature of research questions reflect the specific theme under consideration

29. Participants to think about the nature of questions / hypotheses under each objective on preceding slide Record; then Compare with what is on the following slides

30. Research Questions Objective 1: To generate robust evidence of which projects work in reducing greenhouse gas emissions Questions in terms of impacts:  What levels of greenhouse gas emission reductions have been achieved?  What are the key techniques/factors/activities that have reduced green house gas emissions?  How do the reductions vary between investment themes?  What are the timescales for outturn reductions?

31. Research Questions Objective 1: Questions in terms of behavioural change contributing to outcomes:  What changes in behaviour have been generated, how and why? Are these sustainable?  What are the key drivers and motivators for behavioural change? Do they vary in the different themes? Do these vary between locations and population sub-groups?  Which approaches and techniques have generated the highest levels of behavioural change?  To what extent has investment generated low-carbon communities?  How was the long term sustainability of the project planned for and secured?

32. Research Questions Objective 2: To support the trial of different approaches to community action on climate change Questions in terms of community engagement:  What approaches have been adopted to engender community action?  Which approaches achieve greatest community engagement and support and why?  Which project theme generated the greatest level of community engagement and why?

33. Research Questions Objective 2: Questions in terms of penetration and sustainability:  How does the delivery of impacts of different approaches vary between locations, across socio-economic groups and why?  What are the key factors in a given approach that accelerated success?  What are the key factors that resulted in less than expected good results?

34. Research Questions Objective 3: To evaluate the different ‘models’ of community based project interventions  What are the different models/approaches to community based project development?  Which models/approaches are the most efficient in terms of costs/benefits?  What are the key processes involved in each model/approach and what lessons can be transferred to other locations/projects?  Which elements of approaches are most effective/important in reducing greenhouse gas emissions and why?  Which approaches generate the best advancement in community knowledge and skills in addressing climate change? How do these vary between locations and case studies?

35. Research Questions Objective 4: To evaluate the delivery of the pathfinder project programme by the programme management team  What co-ordination/leadership approaches by the programme management team facilitated dialogue and collaboration with key stakeholders and between Pathfinder Officers (POs)?  What approaches acted as barriers to meaningful dialogue and collaboration?  What support, training and direction provided for clarity of work by the POs?  How effective was the Advisory Group in helping to deliver the work of the POs?  What key strengths of the programme management team enabled action by the POs and what gaps/weaknesses acted as barriers?  What other barriers challenged work of the whole programme?

36. Research Questions Objective 5: To assess the role of Pathfinder Officers in delivering programmes  What role did the PO have in accelerating the work of the community projects they supported?  What particular skills, knowledge and abilities of the PO supported the success of the community projects?  What approaches to community participation taken by the individual POs supported community projects?  What barriers did the POs face that acted as a barrier to the delivery of the work of the community project? How were they removed, if removable?

37. Phase One (Scoping) – Overview Evaluation Scoping Define interventions and packages Confirm evaluation objectives Select research questions Select evaluation indicators Select core indicators Select contextual indicators Stakeholder liaisonConduct ToC mapping Identify new data needed Define evaluation approach ToC mapping

38. Participants to consider what would make a Good Indicator Record, then Discuss individual merits

39. What Would Make a Good Indicator SMART: –Specific –Measurable –Achievable –Relevant –Time based

40. What Would Make a Good Indicator (cont) But, what will be Practical?: –What data are available? –Who owns or has access to the data? –What are the timescales for anticipated change?

41. Q and A

42. Lunch

43. Baseline and Contexts

44. Top Down Policy – Bottom up Community Actions Current Baseline Emissions Future Target

45. What Differences are Made as a Result of Pathfinder Officer Actions? Project might progress at a faster pace – with impact felt sooner How to assess what would have happened without engagement? Informed by ‘backward’ review and comparison with other similar projects without PO engagement

46. What Differences are Made as a Result of Pathfinder Officer Actions? Overall impact of Project might increase The availability of data for reviewing the baseline / activities prior to PO engagement will be a factor in the selection of case Studies

47. Q and A

48. Phase One (Scoping) – Overview Evaluation Scoping Define interventions and packages Confirm evaluation objectives Select research questions Select evaluation indicators Select core indicators Select contextual indicators Stakeholder liaison Conduct ToC mapping Identify new data needed Define evaluation approach ToC mapping

49. Monitoring and Recording Following initial engagement with Stakeholders –Focus will be on collating basic ‘headline’ data for each potential case study (within a long list of, say, 120) –This will enable actual case studies to be selected for evaluation Criteria to be determined to facilitate the selection of final case studies –Criteria should allow comparisons to be made across candidate case studies –Should aim for a representative cross section of case studies, covering all relevant sectors and differing approaches to community involvement, project development and delivery More detailed data will then to be recorded for selected case studies, to fully define and characterise the case study in advance of the evaluation phase The actual basic data to be collated will reflect the nature of the initiative and the criteria used for choosing between candidate case studies

50. Participants to brainstorm some of the criteria / data needs Record Handout initial draft schema for reviewing projects (seeking financial support under sustainable communities programme) Participants to take this away and provide feedback on how this can be adapted to the needs of the Pathfinder research

51. Q and A

52. Evaluation is Not Just About the Data Data Facts Information Intelligence Insight Journal to be kept : For each activity undertaken: Describe the activity Reflect on:  what went well  what didn’t go well and why not  what should be done differently next time Note down progress and additional information as required

53. Capturing the Data (Facts, Information, Intelligence and Insights) Precise Tools to be Developed But, likely use of dedicated site to store information and share this across the study team Possible input from Development Officers in designing appropriate data capture proforma

54. Hand out example field sheet Participants to review this and feedback later how this might be adapted - to meet their needs / needs of the study team

55. Wrap-up Future Actions

56. Q and A

57. Thank You richard.redfern@aecom.com steve.tarry@aecom.com