CIVITAS PLUS Renaissance – BATH Overview of Evaluation: Methods & Some Key Results Prof Graham Parkhurst, Dr Miriam Ricci – UWE Bristol Renaissance & CIVINET Conference, 13 March 2012 - Bath

Evaluation – overview Identify EU added value: potential for upscaling locally and exploitation elsewhere Impact evaluation: to identify (and often to quantify) the range of impacts in relation to: Economic and energy efficiency (CBA on selected demonstrations) Congestion, Air quality and CO2 emissions Public awareness & acceptance Modal shift (stated preferences) Process evaluation: to understand how demonstrations have been implemented and the underlying drivers and barriers

Impact evaluation - methodology Before/After: 4.3: Wayfinding 5.4: St James Rampire With/Without: 1.3: Hybrid bus trial (with CBA) 6.3: City Car Club (with CBA) Case study: 3.4: Demand management 6.4: Bike-sharing 7.2: Freight Consolidation Centre (with CBA)

Impact evaluation – methods Public awareness, acceptance & modal shift Demonstration Method Notes All measures On-street survey in central Bath Before and after key demonstrations 1.3 – Hybrid bus trial Survey of bus passengers and car park users To compare passengers’ perceptions of hybrid bus to those of diesel bus; to discover perceptions of non-users; to examine stated-preferences for low C bus technology. 3.4 – Demand management Survey of logistics/delivery providers Evaluation to be conducted 4.3 - Wayfinding On-street survey + qualitative wayfinding tasks Mixed methods (quantitative + qualitative) 5.4 – St James Rampire On-street survey + qualitative site walk-throughs

Impact evaluation – methods Public awareness, acceptance & modal shift Demonstration Method Notes 6.3 – City Car Club Online survey of Bath City Car Club members + in-depth qualitative interviews with selected survey respondents Mixed methods (quantitative + qualitative). To compare and contrast subjective experience and perceptions of the Toyota Prius with conventional vehicles. To understand change in travel behaviours/car ownership. 6.4 – Bike-sharing Survey of users of the scheme Evaluation to be conducted 7.2 – Freight Consolidation Centre Qualitative interviews with participating retailers + online survey of non-users of the centre Evaluation in progress.

Measure 1.3: Hybrid bus trial Commercial partner: FIRST Group Vehicle: Wrightbus HEV diesel-electric parallel hybrid Vauxhall 1.9l diesel engine (less than a quarter of those for conventional buses) Ballard hybrid power system (37kW battery pack) Vehicle capable of operation in “zero emission” mode (electric only) in the city centre Commenced service on 1st Sept 2010 on all 3 P&R routes in Bath - variety of hilly and flat routes Trial to last 18 months, until end of Feb 2012 Conventional P&R vehicles in Bath are EURO 3 VOLVO diesel buses

Indicators No. Impact Indicator Data used Comments 2B Costs Operating costs Cost per km £ per km, quantitative, measured 3 Fuel Consumption Vehicle fuel efficiency Fuel used per vkm MJ per km, quantitative, measured 8-11 Emissions CO2, CO, NOx, and particulate emissions Emissions per vkm by type Gram per km, quantitative, derived or measured 12 Noise Noise perception Perception of noise; Actual noise levels Noise perception collected through passengers’ questionnaire survey; Noise levels measured in-service 13 Awareness Awareness level Quantitative data from questionnaire survey Assessed through passengers’ questionnaire survey 14 Acceptance Acceptance level 18 Service Reliability Accuracy of time keeping Incidence of failure; availability for service; number and/or percentage of service arriving/departing on time; drivers’ perceptions Service reliability assessed by combining measurements and survey data 19 Quality of service Perception of quality of service including comfort, etc. 28 Modal split Average Modal Split - trips Stated preference of modal shift based on attractiveness of new bus Assessed through passengers’ and car park users’ questionnaire survey

Cost-Benefit Analysis approach Analysis assumptions: Comparison of same route with two separate vehicles: Identified benefits: savings in vehicle operating costs (VOC) reduction in emissions Required inputs: Costs excluding tax: Variable VOC (per mile or per hour) : fuel, oil, tires, maintenance; Fixed VOC (annual cost translated to per mile or per hour): depreciation, insurance; New vehicle value: cost of procurement of new vehicle excluding tax. Scrap value: estimate of bus vehicle value at end of life (15 years). Average annual millage for this route. Emissions per mile or per hour: NOx, CO2.

Evaluation of hybrid bus trial Survey methods: Quantitative questionnaire survey on board the buses (N=1,106), administered by researchers Quantitative questionnaire survey in the major city centre car parks (N=856), administered by researchers Questions: understanding P&R user profile, motivations to use/not use P&R perceptions of hybrid bus attitudes towards cleaner, low C buses

Bus passenger survey – key findings

Bus passenger survey – key findings

Bus passenger survey – key findings

Bus passenger survey – key findings Difference is not statistically significant (95% CL)

Bus passenger survey – key findings Difference is statistically significant (95% CL)

Bus passenger survey – key findings Difference is not statistically significant (95% CL)

Bus passenger survey – key findings Difference is statistically significant (95% CL)

Bus passenger survey – key findings

Bus passenger survey – key findings

Bus passenger survey – key findings

Bus passenger survey – key findings

Measure 4.3: Wayfinding The old The new New system comprises: A unique range of street furniture A unique wayfinding and interpretation system Public transport infrastructure including bespoke shelters and flags Ancillary print and web services to aid visitors’/residents’ understanding and experience of Bath A unique graphic identity including typography for communicating the above products and services Design of a lifestyle marketing campaign to promote sustainable transport options in Bath The new

Indicators No. Impact Indicator Data used Comments 2A Costs Capital costs Development cost of wayfinding system and implementing the pilot Data supplied by B&NES 2B Operating costs Maintenance costs of pilot area 13 Awareness Awareness Level Attitudinal data from quantitative questionnaire survey. Complementary qualitative data. On-street questionnaire survey conducted by UWE in May 2010 and repeated in May 2011. Qualitative data obtained through wayfinding tasks in March 2010, repeated in March 2011. 14 Acceptance Acceptance Level 19 Quality of service Quality of Service 28 Modal Split Average Modal Split-trips Percentage of trips per each mode in relevant areas of the city centre Data are supplied by B&NES (pedestrian counts etc.)

Evaluation of new Wayfinding and Information system Objective: to evaluate and compare residents’ and visitors’ experience of the new and the old wayfinding system (functionality, aesthetics etc) Method: Before/After quantitative on-street questionnaire survey (N=1,000+) Qualitative wayfinding tasks, before and after measure implementation Questions: understanding perceptions/change in perception of wayfinding system across different user profiles

Wayfinding – key evaluation results Positive change, but not statistically significant

Wayfinding – key evaluation results No change, overall positive public perceptions in both years

Wayfinding – key evaluation results No change, overall positive public perceptions in both years

Wayfinding – key evaluation results Positive change, it is statistically significant

Wayfinding – key evaluation results Positive change, it is statistically significant However, in both years perceptions are still mostly negative – perhaps awareness of walking distance/times provision in new wayfinding elements is still low

Wayfinding – key evaluation results Positive change, it is statistically significant

Wayfinding – key evaluation results Overall positive perceptions in both years, no statistically significant changes

Wayfinding – key evaluation results Overall positive perceptions in both years, but decrease in strong agreement (statistically significant) in 2011

Wayfinding – key evaluation results

Wayfinding – key evaluation results

Measure 5.4: St James Rampire Re-design of the central area of St James rampire, including: Improved pedestrian area New lighting solution Wayfinding street elements Testing paving solutions

Measure 5.4: Indicators (1) No. Impact Indicator Data used Comments 2A/2B Costs Capital costs /Operating Costs Costs related to the re-design and maintenance of the demonstration area (including new lighting system installed) Data supplied by B&NES 3 Fuel Consumption Energy Efficiency Energy consumption figures Data referring to the new lighting installed. Data supplied by B&NES 12 Noise Noise Perception Perception data from quantitative questionnaire survey. Complementary qualitative data. On-street questionnaire survey conducted by UWE in May 2010 and repeated in May 2011. Qualitative data obtained through in-situ walk-throughs in March 2010, repeated in March 2011. 13 Awareness Awareness Level Attitudinal data from quantitative questionnaire survey. Complementary qualitative data. On-street questionnaire survey conducted by UWE in May 2011. Qualitative data obtained through in-situ walk-throughs in March 2011. 14 Acceptance Acceptance Level

Measure 5.4: Indicators (2) No. Impact Indicator Data used Comments 15 Spatial Accessibility Perception of Accessibility Qualitative observations on walking, cycling and driving patterns before and after implementation Qualitative data obtained through in-situ walk-throughs in March 2010, repeated in March 2011. 17 Security Perception of security Attitudinal data from quantitative questionnaire survey. Complementary qualitative data. On-street questionnaire survey conducted by UWE in May 2010 and repeated in May 2011. Qualitative data obtained through in-situ walk-throughs in March 2010, repeated in March 2011. 19 Quality of service Test of marking and damage in-situ at regular intervals. Test of surface materials in different areas and affects of different levels of traffic. Data supplied by B&NES 20 Transport Safety Safety – Skid resistance Manufacturer and research test guidelines. Data supplied by B&NES Safety – Perceptions of safety 22 Traffic flow by vehicle type Traffic levels Classified traffic count of all junction movements Classified junction counts were carried out in April 2009 prior to the junction remodelling and in April and October 2011, after the changes.

Measure 5.4: Evaluation Positive change in perceptions

Measure 5.4: Evaluation Positive change in perceptions

Measure 5.4: Evaluation Positive change in perceptions

Measure 5.4: Evaluation Positive change in perceptions

Measure 5.4: Evaluation Positive change in perceptions

Measure 5.4: Evaluation Positive change in perceptions

Measure 5.4: Evaluation

Measure 5.4: Evaluation

Measure 6.3: City Car Club Six news parking bays have been introduced (see map, in red) in addition to those already existing in Bath Expansion of the Bath City Car Club (CCC) fleet with 6 new hybrid vehicles (Toyota Prius) since April 2010, in addition to 6 Ford Fiestas

No Indicator Data used & methods 1 Operating revenues CCC (indexed), comparison to conventional vehicles of similar class 2 Capital & operating costs CCC (leasing costs & operating costs, indexed), comparison to conventional vehicles of similar class. Used for CBA 3-4 Fuel consumption Manufacturer’s data and CCC, comparison to conventional vehicles of similar class. Used for CBA 8-11 Emissions (CO2, CO, NOx, PM) Manufacturer’s data, comparison to conventional vehicles of similar class. Used for CBA 12 Noise perception Survey of members (questionnaire & interviews), comparison to petrol-fuelled vehicles in the Bath fleet 13 Awareness Awareness of demonstration. Survey of members (questionnaire & interviews), comparison to petrol-fuelled vehicles in the Bath fleet 14 Acceptance Acceptance of demonstration. Method as above 15 Spatial accessibility Perception of convenience/ease of access to hybrid vehicles. Method as above 16 Economic accessibility Perception of value for money for hybrid vehicles. Method as above 17 Perception of security Perception of security for access/use of hybrid vehicles. Method as above 18 Accuracy of time keeping Perception of availability of hybrid vehicles. Method as above 19 Quality of service Perception of ‘quality of service’ by user of hybrid vehicles. Method as above

Cost Benefit Analysis approach Analysis assumptions: Comparison of vehicle operating costs (VOC) of hybrid (Toyota Prius) vs conventionally fuelled vehicle (the ‘BAU’ vehicle, Vauxhall Astra). No change in distance travelled. Only fuel costs will be saved. General inputs: Interest rate for discounting: median SDR 3.5% (sensitivity analysis with SDR = 2% and 5.5%) Social costs of emissions (sensitivity analysis: -20% +20%). In the UK, these costs are available at the following Government website: http://www.defra.gov.uk/environment/quality/air/air-quality/economic/damage Identified benefits: Savings in fuel costs Reduction in emissions Required inputs: Fuel consumption (per mile), Fuel cost (per mile) excluding VAT Vehicle lease costs (indexed as commercially sensitive data) Average annual mileage Emissions per mile (manufacturer’s data): CO2, CO, NOx, PM

Survey of City Car Club members Online survey of all Bath CCC members 108 respondents (over 25% response rate) Quantitative assessment of Indicators 12-19 (comparison of perceptions of hybrids - Toyota Prius - with perceptions of existing vehicles - Ford Fiesta) In-depth telephone interviews with a cross-section of questionnaire respondents 16 participants Qualitative assessment of Indicators 12-19 In-depth understanding of trip purposes, travel behaviour change, car ownership change and analysis of ‘what if’ scenarios (e.g. absence of CCC)

No Indicator Key headline results from survey of Bath CCC members 12 Noise perception 92% rated Toyota Prius GOOD/VERY GOOD, compared to 53% in the case of Ford Fiesta. 13 Awareness 69% aware of a change in the provision of new cars since April 2010, in particular availability of hybrid cars. 14 Acceptance 76% support the introduction of hybrid vehicles into the fleet On-street survey in 2011: 71 out of 97 respondents support the hybrid car CIVITAS demonstration. 15 Spatial accessibility Availability of conveniently located bays: 28 respondents rated it same as before demonstration, 31 improved and 10 worsened. 16 Economic accessibility Car hourly cost: 43% rated Toyota Prius GOOD/VERY GOOD, compared to 63% in the case of Ford Fiesta. Value for money of service provided: 40 respondents rated it same as before demonstration, 11 improved and 16 worsened. 17 Perception of security 73% rated Toyota Prius GOOD/VERY GOOD, compared to 61% in the case of Ford Fiesta. 18 Accuracy of time keeping Availability of car when needing to travel: 33 same as before demonstration, 30 improved, 5 worsened. 19 Quality of service Performance: 72% rated Toyota Prius GOOD/VERY GOOD, compared to 51% in the case of Ford Fiesta. Availability of suitable car type: 30 respondents rated it same as before demonstration , 16 improved, 12 worsened.

Survey of Bath City Car Club members Sample characteristics (N=108) Half of the sample aged up to 45 years More male respondents (66%) Most respondents employed or self-employed (over 75%) Car club membership encourages reduction in car ownership

Usage of City Car Club vehicles Occasional users make up almost 70% of the sample Toyota Prius used slightly more than the Ford Fiesta

Awareness of the demonstration (from survey of CCC members) Although most respondents were not aware of the project ‘CIVITAS Renaissance’ before the survey, the majority of the sample (69%) had realised that a change in the provision of vehicles in the CCC fleet had taken place in Bath, in particular the availability of hybrid cars

Awareness of the demonstration (from survey of CCC members) New bays: Sydney Place St James Square Bathwick Hill St Michaels Road Sydney Buildings The Circus In terms of awareness of new parking bays, the new bay at the Circus is clearly well known, as it is located in a landmark place in Bath. Other new bays are also well known, e.g. St James Sq and Bathwick Hill. In term of use, St James Sq is the most used among the new bays, closely followed by the Circus and Bathwick Hill.

Awareness of the demonstration (from survey of CCC members)

Awareness of the demonstration (from on-street survey) Although the majority of both samples had no awareness of the City Car Club, the 2011 results indicate that, one year into the demonstration, more Bath residents seem to be aware of the City Car Club, with 11.6% more respondents in 2011 claiming to be aware (39.8% in 2011 compared to 28.2% in 2010). This might be due to the scale of the demonstration, which doubled the number of cars in the fleet and placed some of the vehicles in landmark locations, for example the Circus. Also, the City Car Club carried out an extensive marketing campaign across a range of online and printed media to promote the CIVITAS-sponsored new vehicles in the fleet, which might have also increased awareness levels among Bath residents. In 2011 more Bath residents seem to be aware of the City Car Club, with 12% more respondents in 2011 claiming to be aware (40% in 2011 compared to 28% in 2010). Of these, only a couple of respondents were also members of CCC. In 2011, of the 97 respondents aware of CCC, only 22 are aware of the demonstration

Acceptance of the demonstration (from survey of CCC members)

Acceptance of the demonstration (from on-street survey) In 2011, of the 97 respondents aware of CCC, the large majority support the introduction of hybrid vehicles (71 respondents) This question was only asked in the 2011 (ex-post) on-street survey in Bath (a large-scale survey addressing most measures and gathering public perceptions across a whole range of issues around the public realm, congestion, air quality, public transport etc.). Only Bath residents were asked the questions, and only if they had claimed that they were aware of the City Car Club.

Noise perception Toyota Prius Ford Fiesta

Quality of service Ease fo use on first booking Toyota Prius Ford Fiesta According to City Car Club experience, the Prius is not an easy car to use for first time users. This can explain the finding that most respondents found the Fiesta easier to use on first booking

Quality of service Starting the car Toyota Prius Ford Fiesta Again, the Fiesta is rated better for the same reasons as before, as the Prius is very different in the way it starts.

Quality of service Braking Toyota Prius Ford Fiesta Here we have similar responses

Quality of service: Manoeuverability Toyota Prius Ford Fiesta

Quality of service: Refuelling Toyota Prius Ford Fiesta Again as before, here we have similar responses

Quality of service Performance Toyota Prius Ford Fiesta Despite scoring less on other aspects, the Prius scores better in overall performance.

Economic accessibility Car hourly rate cost Toyota Prius Ford Fiesta The Toyota Prius scores less well on perception of costs. This can be a result of the pricing structure. At the start of the demonstration (April 2010) the Prius was priced £5.95 per hour in Bath (the Fiesta cost £4.95 per hour), while in other cities the Prius was priced £6.95 per hour. In March 2011 all prices for all vehicles everywhere in the UK were increased, so at the time of the survey members would pay £5.20 per hour for the Fiesta and £6.20 for the Prius (elesewhere the Prius would be £7.20).

Perception of security Toyota Prius Ford Fiesta

Measure 7.2: Freight Consolidation Ongoing evaluation work on awareness and acceptance of the consolidation centre Online survey of non-participating retailers Interviews with participating retailers Observations/feedback from contractor CBA

Cost-Benefit Analysis approach Analysis assumptions: Computation of annual freight costs before consolidation scheme. Cordon around Bath city centre with average radius (x2) travelled within Bath. No change in total distance travelled. Bristol-Bath run for electric DHL lorries assumed to be negligible. No calculation of electric charging costs. ‘Average Lorry’ computed from freight fleet distribution according to DHL survey averages Pre change logistic model: 1 delivery per 1 ‘average’ lorry. Zero economic cost for freight consolidation. Retailer payments discarded. Total costs saved are to be assumed as net benefits Identified benefits: Savings in fuel costs Reduction in emissions Required inputs Vehicle types Fuel consumption per vehicle type Fuel costs excluding VAT No. of daily deliveries

Final Thoughts... Evaluation in Civitas is not undertaken in ‘laboratory conditions’ Targets set before design of demonstration can have limited ‘fit’ with needs to remain responsive to the emerging demonstration Quantification useful but only one part of evaluation Qualitative study provides important, often unexpected insights Process evaluation most important for those seeking to implement elsewhere