1. CIVITAS PLUS Renaissance – BATH
Overview of Evaluation: Methods & Some Key Results
Prof Graham Parkhurst, Dr Miriam Ricci – UWE Bristol
Renaissance & CIVINET Conference, 13 March Bath

2. Evaluation – overview
Identify EU added value: potential for upscaling locally and exploitation elsewhere
Impact evaluation: to identify (and often to quantify) the range of impacts in relation to:
Economic and energy efficiency (CBA on selected demonstrations)
Congestion, Air quality and CO2 emissions
Public awareness & acceptance
Modal shift (stated preferences)
Process evaluation: to understand how demonstrations have been implemented and the underlying drivers and barriers

3. Impact evaluation - methodology
Before/After:
4.3: Wayfinding
5.4: St James Rampire
With/Without:
1.3: Hybrid bus trial (with CBA)
6.3: City Car Club (with CBA)
Case study:
3.4: Demand management
6.4: Bike-sharing
7.2: Freight Consolidation Centre (with CBA)

4. Impact evaluation – methods Public awareness, acceptance & modal shift
Demonstration
Method
Notes
All measures
On-street survey in central Bath
Before and after key demonstrations
1.3 – Hybrid bus trial
Survey of bus passengers and car park users
To compare passengers’ perceptions of hybrid bus to those of diesel bus; to discover perceptions of non-users; to examine stated-preferences for low C bus technology.
3.4 – Demand management
Survey of logistics/delivery providers
Evaluation to be conducted
4.3 - Wayfinding
On-street survey + qualitative wayfinding tasks
Mixed methods (quantitative + qualitative)
5.4 – St James Rampire
On-street survey + qualitative site walk-throughs

5. Impact evaluation – methods Public awareness, acceptance & modal shift
Demonstration
Method
Notes
6.3 – City Car Club
Online survey of Bath City Car Club members + in-depth qualitative interviews with selected survey respondents
Mixed methods (quantitative + qualitative). To compare and contrast subjective experience and perceptions of the Toyota Prius with conventional vehicles. To understand change in travel behaviours/car ownership.
6.4 – Bike-sharing
Survey of users of the scheme
Evaluation to be conducted
7.2 – Freight Consolidation Centre
Qualitative interviews with participating retailers + online survey of non-users of the centre
Evaluation in progress.

6. Measure 1.3: Hybrid bus trial
Commercial partner: FIRST Group
Vehicle: Wrightbus HEV diesel-electric parallel hybrid
Vauxhall 1.9l diesel engine (less than a quarter of those for conventional buses)
Ballard hybrid power system (37kW battery pack)
Vehicle capable of operation in “zero emission” mode (electric only) in the city centre
Commenced service on 1st Sept 2010 on all 3 P&R routes in Bath - variety of hilly and flat routes
Trial to last 18 months, until end of Feb 2012
Conventional P&R vehicles in Bath are EURO 3 VOLVO diesel buses

7. Indicators No. Impact Indicator Data used Comments 2B Costs
Operating costs
Cost per km
£ per km, quantitative, measured 3
Fuel Consumption
Vehicle fuel efficiency
Fuel used per vkm
MJ per km, quantitative, measured
8-11
Emissions
CO2, CO, NOx, and particulate emissions
Emissions per vkm by type
Gram per km, quantitative, derived or measured 12
Noise
Noise perception
Perception of noise;
Actual noise levels
Noise perception collected through passengers’ questionnaire survey;
Noise levels measured in-service 13
Awareness
Awareness level
Quantitative data from questionnaire survey
Assessed through passengers’ questionnaire survey 14
Acceptance
Acceptance level 18
Service Reliability
Accuracy of time keeping
Incidence of failure; availability for service; number and/or percentage of service arriving/departing on time; drivers’ perceptions
Service reliability assessed by combining measurements and survey data 19
Quality of service
Perception of quality of service including comfort, etc. 28
Modal split
Average Modal Split - trips
Stated preference of modal shift based on attractiveness of new bus
Assessed through passengers’ and car park users’ questionnaire survey

8. Cost-Benefit Analysis approach
Analysis assumptions: Comparison of same route with two separate vehicles:
Identified benefits:
savings in vehicle operating costs (VOC)
reduction in emissions
Required inputs:
Costs excluding tax:
Variable VOC (per mile or per hour) : fuel, oil, tires, maintenance;
Fixed VOC (annual cost translated to per mile or per hour): depreciation, insurance;
New vehicle value: cost of procurement of new vehicle excluding tax.
Scrap value: estimate of bus vehicle value at end of life (15 years).
Average annual millage for this route.
Emissions per mile or per hour: NOx, CO2.

9. Evaluation of hybrid bus trial
Survey methods:
Quantitative questionnaire survey on board the buses (N=1,106), administered by researchers
Quantitative questionnaire survey in the major city centre car parks (N=856), administered by researchers
Questions:
understanding P&R user profile,
motivations to use/not use P&R
perceptions of hybrid bus
attitudes towards cleaner, low C buses

10. Bus passenger survey – key findings

11. Bus passenger survey – key findings

12. Bus passenger survey – key findings

13. Bus passenger survey – key findings
Difference is not statistically significant (95% CL)

14. Bus passenger survey – key findings
Difference is statistically significant (95% CL)

15. Bus passenger survey – key findings
Difference is not statistically significant (95% CL)

16. Bus passenger survey – key findings
Difference is statistically significant (95% CL)

17. Bus passenger survey – key findings

18. Bus passenger survey – key findings

19. Bus passenger survey – key findings

20. Bus passenger survey – key findings

21. Measure 4.3: Wayfinding The old The new New system comprises:
A unique range of street furniture
A unique wayfinding and interpretation system
Public transport infrastructure including bespoke shelters and flags
Ancillary print and web services to aid visitors’/residents’ understanding and experience of Bath
A unique graphic identity including typography for communicating the above products and services
Design of a lifestyle marketing campaign to promote sustainable transport options in Bath
The new

22. Indicators No. Impact Indicator Data used Comments 2A Costs
Capital costs
Development cost of wayfinding system and implementing the pilot
Data supplied by B&NES 2B
Operating costs
Maintenance costs of pilot area 13
Awareness
Awareness Level
Attitudinal data from quantitative questionnaire survey. Complementary qualitative data.
On-street questionnaire survey conducted by UWE in May 2010 and repeated in May Qualitative data obtained through wayfinding tasks in March 2010, repeated in March 2011. 14
Acceptance
Acceptance Level 19
Quality of service
Quality of Service 28
Modal Split
Average Modal Split-trips
Percentage of trips per each mode in relevant areas of the city centre
Data are supplied by B&NES (pedestrian counts etc.)

23. Evaluation of new Wayfinding and Information system
Objective: to evaluate and compare residents’ and visitors’ experience of the new and the old wayfinding system (functionality, aesthetics etc)
Method:
Before/After quantitative on-street questionnaire survey (N=1,000+)
Qualitative wayfinding tasks, before and after measure implementation
Questions: understanding perceptions/change in perception of wayfinding system across different user profiles

24. Wayfinding – key evaluation results
Positive change, but not statistically significant

25. Wayfinding – key evaluation results
No change, overall positive public perceptions in both years

26. Wayfinding – key evaluation results
No change, overall positive public perceptions in both years

27. Wayfinding – key evaluation results
Positive change, it is statistically significant

28. Wayfinding – key evaluation results
Positive change, it is statistically significant
However, in both years perceptions are still mostly negative – perhaps awareness of walking distance/times provision in new wayfinding elements is still low

29. Wayfinding – key evaluation results
Positive change, it is statistically significant

30. Wayfinding – key evaluation results
Overall positive perceptions in both years, no statistically significant changes

31. Wayfinding – key evaluation results
Overall positive perceptions in both years, but decrease in strong agreement (statistically significant) in 2011

32. Wayfinding – key evaluation results

33. Wayfinding – key evaluation results

34. Measure 5.4: St James Rampire
Re-design of the central area of St James rampire, including:
Improved pedestrian area
New lighting solution
Wayfinding street elements
Testing paving solutions

35. Measure 5.4: Indicators (1)
No.
Impact
Indicator
Data used
Comments
2A/2B
Costs
Capital costs /Operating Costs
Costs related to the re-design and maintenance of the demonstration area (including new lighting system installed)
Data supplied by B&NES 3
Fuel Consumption
Energy Efficiency
Energy consumption figures
Data referring to the new lighting installed. Data supplied by B&NES 12
Noise
Noise Perception
Perception data from quantitative questionnaire survey. Complementary qualitative data.
On-street questionnaire survey conducted by UWE in May 2010 and repeated in May Qualitative data obtained through in-situ walk-throughs in March 2010, repeated in March 2011. 13
Awareness
Awareness Level
Attitudinal data from quantitative questionnaire survey. Complementary qualitative data.
On-street questionnaire survey conducted by UWE in May Qualitative data obtained through in-situ walk-throughs in March 2011. 14
Acceptance
Acceptance Level

36. Measure 5.4: Indicators (2)
No.
Impact
Indicator
Data used
Comments 15
Spatial Accessibility
Perception of Accessibility
Qualitative observations on walking, cycling and driving patterns before and after implementation
Qualitative data obtained through in-situ walk-throughs in March 2010, repeated in March 2011. 17
Security
Perception of security
Attitudinal data from quantitative questionnaire survey. Complementary qualitative data.
On-street questionnaire survey conducted by UWE in May 2010 and repeated in May Qualitative data obtained through in-situ walk-throughs in March 2010, repeated in March 2011. 19
Quality of service
Test of marking and damage in-situ at regular intervals.
Test of surface materials in different areas and affects of different levels of traffic. Data supplied by B&NES 20
Transport Safety
Safety – Skid resistance
Manufacturer and research test guidelines.
Data supplied by B&NES
Safety – Perceptions of safety 22
Traffic flow by vehicle type
Traffic levels
Classified traffic count of all junction movements
Classified junction counts were carried out in April 2009 prior to the junction remodelling and in April and October 2011, after the changes.

37. Measure 5.4: Evaluation
Positive change in perceptions

38. Measure 5.4: Evaluation
Positive change in perceptions

39. Measure 5.4: Evaluation
Positive change in perceptions

40. Measure 5.4: Evaluation
Positive change in perceptions

41. Measure 5.4: Evaluation
Positive change in perceptions

42. Measure 5.4: Evaluation
Positive change in perceptions

43. Measure 5.4: Evaluation

44. Measure 5.4: Evaluation

45. Measure 6.3: City Car Club
Six news parking bays have been introduced (see map, in red) in addition to those already existing in Bath
Expansion of the Bath City Car Club (CCC) fleet with 6 new hybrid vehicles (Toyota Prius) since April 2010, in addition to 6 Ford Fiestas

46. No Indicator Data used & methods 1 Operating revenues
CCC (indexed), comparison to conventional vehicles of similar class 2
Capital & operating costs
CCC (leasing costs & operating costs, indexed), comparison to conventional vehicles of similar class. Used for CBA
3-4
Fuel consumption
Manufacturer’s data and CCC, comparison to conventional vehicles of similar class. Used for CBA
8-11
Emissions (CO2, CO, NOx, PM)
Manufacturer’s data, comparison to conventional vehicles of similar class. Used for CBA 12
Noise perception
Survey of members (questionnaire & interviews), comparison to petrol-fuelled vehicles in the Bath fleet 13
Awareness
Awareness of demonstration. Survey of members (questionnaire & interviews), comparison to petrol-fuelled vehicles in the Bath fleet 14
Acceptance
Acceptance of demonstration. Method as above 15
Spatial accessibility
Perception of convenience/ease of access to hybrid vehicles. Method as above 16
Economic accessibility
Perception of value for money for hybrid vehicles. Method as above 17
Perception of security
Perception of security for access/use of hybrid vehicles. Method as above 18
Accuracy of time keeping
Perception of availability of hybrid vehicles. Method as above 19
Quality of service
Perception of ‘quality of service’ by user of hybrid vehicles. Method as above

47. Cost Benefit Analysis approach
Analysis assumptions:
Comparison of vehicle operating costs (VOC) of hybrid (Toyota Prius) vs conventionally fuelled vehicle (the ‘BAU’ vehicle, Vauxhall Astra). No change in distance travelled. Only fuel costs will be saved.
General inputs:
Interest rate for discounting: median SDR 3.5% (sensitivity analysis with SDR = 2% and 5.5%)
Social costs of emissions (sensitivity analysis: -20% +20%). In the UK, these costs are available at the following Government website:
Identified benefits:
Savings in fuel costs
Reduction in emissions
Required inputs:
Fuel consumption (per mile), Fuel cost (per mile) excluding VAT
Vehicle lease costs (indexed as commercially sensitive data)
Average annual mileage
Emissions per mile (manufacturer’s data): CO2, CO, NOx, PM

48. Survey of City Car Club members
Online survey of all Bath CCC members
108 respondents (over 25% response rate)
Quantitative assessment of Indicators (comparison of perceptions of hybrids - Toyota Prius - with perceptions of existing vehicles - Ford Fiesta)
In-depth telephone interviews with a cross-section of questionnaire respondents
16 participants
Qualitative assessment of Indicators 12-19
In-depth understanding of trip purposes, travel behaviour change, car ownership change and analysis of ‘what if’ scenarios (e.g. absence of CCC)

49. No Indicator Key headline results from survey of Bath CCC members 12
Noise perception
92% rated Toyota Prius GOOD/VERY GOOD, compared to 53% in the case of Ford Fiesta. 13
Awareness
69% aware of a change in the provision of new cars since April 2010, in particular availability of hybrid cars. 14
Acceptance
76% support the introduction of hybrid vehicles into the fleet
On-street survey in 2011: 71 out of 97 respondents support the hybrid car CIVITAS demonstration. 15
Spatial accessibility
Availability of conveniently located bays: 28 respondents rated it same as before demonstration, 31 improved and 10 worsened. 16
Economic accessibility
Car hourly cost: 43% rated Toyota Prius GOOD/VERY GOOD, compared to 63% in the case of Ford Fiesta.
Value for money of service provided: 40 respondents rated it same as before demonstration, 11 improved and 16 worsened. 17
Perception of security
73% rated Toyota Prius GOOD/VERY GOOD, compared to 61% in the case of Ford Fiesta. 18
Accuracy of time keeping
Availability of car when needing to travel: 33 same as before demonstration, 30 improved, 5 worsened. 19
Quality of service
Performance: 72% rated Toyota Prius GOOD/VERY GOOD, compared to 51% in the case of Ford Fiesta.
Availability of suitable car type: 30 respondents rated it same as before demonstration , 16 improved, 12 worsened.

50. Survey of Bath City Car Club members Sample characteristics (N=108)
Half of the sample aged up to 45 years
More male respondents (66%)
Most respondents employed or self-employed (over 75%)
Car club membership encourages reduction in car ownership

51. Usage of City Car Club vehicles
Occasional users make up almost 70% of the sample
Toyota Prius used slightly more than the Ford Fiesta

52. Awareness of the demonstration (from survey of CCC members)
Although most respondents were not aware of the project ‘CIVITAS Renaissance’ before the survey, the majority of the sample (69%) had realised that a change in the provision of vehicles in the CCC fleet had taken place in Bath, in particular the availability of hybrid cars

53. Awareness of the demonstration (from survey of CCC members)
New bays:
Sydney Place
St James Square
Bathwick Hill
St Michaels Road
Sydney Buildings
The Circus
In terms of awareness of new parking bays, the new bay at the Circus is clearly well known, as it is located in a landmark place in Bath. Other new bays are also well known, e.g. St James Sq and Bathwick Hill. In term of use, St James Sq is the most used among the new bays, closely followed by the Circus and Bathwick Hill.

54. Awareness of the demonstration (from survey of CCC members)

55. Awareness of the demonstration (from on-street survey)
Although the majority of both samples had no awareness of the City Car Club, the 2011 results indicate that, one year into the demonstration, more Bath residents seem to be aware of the City Car Club, with 11.6% more respondents in 2011 claiming to be aware (39.8% in 2011 compared to 28.2% in 2010). This might be due to the scale of the demonstration, which doubled the number of cars in the fleet and placed some of the vehicles in landmark locations, for example the Circus. Also, the City Car Club carried out an extensive marketing campaign across a range of online and printed media to promote the CIVITAS-sponsored new vehicles in the fleet, which might have also increased awareness levels among Bath residents.
In 2011 more Bath residents seem to be aware of the City Car Club, with 12% more respondents in 2011 claiming to be aware (40% in 2011 compared to 28% in 2010). Of these, only a couple of respondents were also members of CCC.
In 2011, of the 97 respondents aware of CCC, only 22 are aware of the demonstration

56. Acceptance of the demonstration (from survey of CCC members)

57. Acceptance of the demonstration (from on-street survey)
In 2011, of the 97 respondents aware of CCC, the large majority support the introduction of hybrid vehicles (71 respondents)
This question was only asked in the 2011 (ex-post) on-street survey in Bath (a large-scale survey addressing most measures and gathering public perceptions across a whole range of issues around the public realm, congestion, air quality, public transport etc.). Only Bath residents were asked the questions, and only if they had claimed that they were aware of the City Car Club.

58. Noise perception
Toyota Prius
Ford Fiesta

59. Quality of service Ease fo use on first booking
Toyota Prius
Ford Fiesta
According to City Car Club experience, the Prius is not an easy car to use for first time users. This can explain the finding that most respondents found the Fiesta easier to use on first booking

60. Quality of service Starting the car
Toyota Prius
Ford Fiesta
Again, the Fiesta is rated better for the same reasons as before, as the Prius is very different in the way it starts.

61. Quality of service Braking
Toyota Prius
Ford Fiesta
Here we have similar responses

62. Quality of service: Manoeuverability
Toyota Prius
Ford Fiesta

63. Quality of service: Refuelling
Toyota Prius
Ford Fiesta
Again as before, here we have similar responses

64. Quality of service Performance
Toyota Prius
Ford Fiesta
Despite scoring less on other aspects, the Prius scores better in overall performance.

65. Economic accessibility Car hourly rate cost
Toyota Prius
Ford Fiesta
The Toyota Prius scores less well on perception of costs. This can be a result of the pricing structure. At the start of the demonstration (April 2010) the Prius was priced £5.95 per hour in Bath (the Fiesta cost £4.95 per hour), while in other cities the Prius was priced £6.95 per hour. In March 2011 all prices for all vehicles everywhere in the UK were increased, so at the time of the survey members would pay £5.20 per hour for the Fiesta and £6.20 for the Prius (elesewhere the Prius would be £7.20).

66. Perception of security
Toyota Prius
Ford Fiesta

67. Measure 7.2: Freight Consolidation
Ongoing evaluation work on awareness and acceptance of the consolidation centre
Online survey of non-participating retailers
Interviews with participating retailers
Observations/feedback from contractor
CBA

68. Cost-Benefit Analysis approach
Analysis assumptions: Computation of annual freight costs before consolidation scheme.
Cordon around Bath city centre with average radius (x2) travelled within Bath.
No change in total distance travelled. Bristol-Bath run for electric DHL lorries assumed to be negligible. No calculation of electric charging costs.
‘Average Lorry’ computed from freight fleet distribution according to DHL survey averages
Pre change logistic model: 1 delivery per 1 ‘average’ lorry.
Zero economic cost for freight consolidation. Retailer payments discarded. Total costs saved are to be assumed as net benefits
Identified benefits:
Savings in fuel costs
Reduction in emissions
Required inputs
Vehicle types
Fuel consumption per vehicle type
Fuel costs excluding VAT
No. of daily deliveries

69. Final Thoughts...
Evaluation in Civitas is not undertaken in ‘laboratory conditions’
Targets set before design of demonstration can have limited ‘fit’ with
needs to remain responsive to the emerging demonstration
Quantification useful but only one part of evaluation
Qualitative study provides important, often unexpected insights
Process evaluation most important for those seeking to implement elsewhere