Long term evaluation – summarizing “How-to-do-it” CIVITAS Thematic Groups, Interactive training 7 May 2014, ECOMM 2014, Florence, Italy Anders Wretstrand, Lund University

2 « How-to-do » long term evaluation Each measure has one or more objectives may be a combination, a bundle, of measures Objectives may be like Modal shift Vivid city centre Coordinated urban freight These have to be operationalized into indicators that are Linked to objectives Quantifiable Measurable Each objective relate to several measures Each measure should be linked to several indicators

3 Impact and process evaluation Impact evaluation is carried out in several steps starting with a baseline study (before) after implementation (after - short term, 1 st or 2 nd year) after “a while” (after - long term, 5 th or 10 th year) what matters is what impacts the measure has on the particular area that the measure aims at, revealed by monitoring indicators Process evaluation a means to facilitate dissemination and transferability of the results tells the story of planning and design implementation and operation reveals the “road to success” (or failure)

4 Short and long term Short term evaluation generally covers one year or year or duration of project funding duration uses before and after data Long term evaluation can include projecting impacts into future Forecasting, scenario-building can involve time series data for programs/measures over years running, ongoing surveys planned, repeated long-term effect surveys Different results… no impact evidence on short term – but positive on long term impact on short term – but no “up-scaling” on long term in both cases: the process (the story) is important to capture!

5 About effects – impact assessment Measure Before After - Short term Time Impact Effect of measure Long term Beware!

6 About effects – impact assessment Measure Before After - Short term Time Impact Effect of other factors Effect of measure Long term ”Business as usual”

7 Evaluation design options Before Measure After I Measure site BaselineImpact I After II Impact II BeforeAfter I Control site After II Changes? The ideal situation with control site – nice but quite rare. To strive for!

8 Evaluation design options Before Measure After I Measure site BaselineImpact I After II Impact II After I Control site After II Changes? Control site without baseline

9 Evaluation design options Measure After I Measure site Impact I After II Impact II After I Control site After II Changes? No baseline data

10 Evaluation design options Before Measure After I Measure site BaselineImpact I After II Impact II Changes? No control site to compare with

11 Evaluation design options Measure After I Measure site Impact I After II Impact II Changes? No control site or baseline data to compare with Common situation when asked to evaluate!

12 Evaluation design options Measure Measure site After Impact Changes? No control site, previous evaluation or baseline data to compare with Common situation when asked to do long-term evaluation!

13 Examples – measures from Cases planning: SUMP health and accessibility: healthy walking public transport: more efficient trolley buses mobilit management in residential areas

14 Examples – SUMP MeasureObjectiveIndicatorMethodBaseline SUMPTo (plan to) implement more “sustainable” transport measures than previously “Sustainability” of measures previously implemented and in new SUMP; and those from SUMP that are actually implemented. Score using checklist methodology (e.g. Ecomobility SHIFT, or a simplified version) Analysis of recently implemented measures, One of the objectives and indicators Very suitable for long term evaluation!

15 Examples – Planning for healthy walking MeasureObjectiveIndicatorMethodBaseline Healthy walking Improve health especially amongst older people in the CIVITAS area Levels of physical activity that target group in area undertake – self-reported Questionnaire to patients at each Centro de Salud; and a panel of the same 30 people each year. First survey results before implemen- tation One of the objectives and indicators Very suitable for long term evaluation! But still… …difficult to assess long term effects, since people age!!!

16 Examples – More efficient trolley buses MeasureObjectiveIndicatorMethodBaseline ”Supercaps”Economic efficiency Operating and capital costs Calculations using records and collecting technical data Existing records and data sets One of the objectives and indicators Easy to do short term evaluation as well! Long term evaluation reveals if system e.g. requires more maintenance. Baseline is less of a problem when dealing with technical data!

17 Examples – Electromobile living MeasureObjectiveIndicatorMethodBaseline Offer shared e-vehicles and pedelecs for tenants Less car- dependent lifestyle Travel behaviour, modal split Case/control, annual travel diary First survey One of the objectives and indicators THEORETICALLY easy to do short term evaluation, but might only detect attitudes and awareness! Long term reveals if people substantially have changed their behaviour, e.g. got rid of 2nd cars

18 Finally The « How-to-guide on long term evaluation» from CiViTAS CAPITAL will be available for all participants after approval as a final document through us or Mobiel21 Now, as a hand-out Checklist: recommendations for long term evaluation of MM activities If you or anyone else have done a previous evaluation of the measure, or If no previous evaluation exists

19 Checklist: based on existing short-term evaluation StepsRemarks Impact evaluation Identify policy/measure objectives Break down the policies/objectives into discrete parts Reflect on cause and effect relations Possible long-term behavioural adaptations and changes? Select short-term evaluation indicators They should match and measure the discrete objectives Analyse short-term impact evaluation Good methods to be continued, poor methods to be left out Revise and refine indicator descriptions Make use of ongoing surveys, revise some indicators if needed Collect new data Align with short term evaluation to continue the time series Analyse data Use proper methods for analysis, based on guidelines Interpret data Are some indicators influenced by other measures? Bundles? Process evaluation Evaluation of existing documents New policies in place, upscaling of measures, new funding, etc. Self-assessment of stakeholders Interviews, workshops: reported failure, success, lessons learned

20 Checklist: no short-term evaluation available StepsRemarks Impact evaluation Identify policy/measure objectives Break down the policies/objectives into discrete parts Reflect on cause and effect relations Possible long-term behavioural adaptations and changes? Propose indicators They should match and measure the discrete objectives Seek for available historical data Which technical and/or survey data sources are available? Revise and refine indicator descriptions Change of direction may be needed, because of poor data Collect new and historical data New = current situation; historical = available time series Analyse data Use proper methods for analysis, based on guidelines Interpret data Are some indicators influenced by other measures? Bundles? Process evaluation Evaluation of existing documents New policies in place, upscaling of measures, new funding, etc. Self-assessment of stakeholders Interviews, workshops: reported failure, success, lessons learned

21 Thank you! Anders Wretstrand Tom Rye Contact Details Lund University P.O. Box 118, SE Lund, SWEDEN