1. COMPLEMENTARY TEACHING MATERIALS
CABI TOURISM TEXTS
4th Edition Leisure, Sport and Tourism, Politics, Policy and Planning
A. J. Veal
COMPLEMENTARY TEACHING MATERIALS

2. Performance Evaluation
Leisure, Sport and Tourism, Politics, Policy and Planning, 4th Edition
CHAPTER 15
Performance Evaluation
Leisure, Sport and Tourism, Politics, Policy and Planning, 4th edition, Veal, 2017, CABI Tourism Texts

3. Outline Introduction Evaluation in context
Steps in the evaluation process
Approaches
Goals and performance indicators
Leisure, Sport and Tourism, Politics, Policy and Planning, 4th edition, Veal, 2017, CABI Tourism Texts

4. Evaluation in Context Economic evaluation – see Chapter 14
Managerialism and privatization
Forms of evaluation
Sustainability/carbon footprint/water conservation
Triple bottom line accounting
Effectiveness and efficiency
Leisure, Sport and Tourism, Politics, Policy and Planning, 4th edition, Veal, 2017, CABI Tourism Texts

5. Managerialism and privatization
Managerialism: the application of formal/rational management practices
Often used (typically negatively) in regard to applying private sector-style practices in the public sector
Arises particularly with privatization or ‘contracting out’ to the private sector – implying that only financial criteria are used and traditional public sector goals are being ignored
However:
Leisure, Sport and Tourism, Politics, Policy and Planning, 4th edition, Veal, 2017, CABI Tourism Texts

6. Managerialism and privatization (cont’d)
Contracts should include all the requirements of the public sector.
Even public sector organizations need to think about best use of capital.
Tourism enterprises, while commercial in nature, can also have wide community impacts.
When services are contracted out to private companies, the public agency (e.g. council) still has responsibilities regarding the quality of life of the community.
This also applies to single-purpose agencies (e.g. arts, sport) in their areas of concern.
Competitive tendering for public services can ‘concentrate the mind’ of the public agency to spell out its goals.
Leisure, Sport and Tourism, Politics, Policy and Planning, 4th edition, Veal, 2017, CABI Tourism Texts

7. Forms of evaluation Routine internal – e.g. annual
Strategic – relating to strategic planning
Accountability – inclusion in official annual reports
Ad hoc, one-off evaluations
Internal versus comparative
internal comparison with previous years or between units
comparative: comparison with external ‘benchmarks’
Leisure, Sport and Tourism, Politics, Policy and Planning, 4th edition, Veal, 2017, CABI Tourism Texts

8. Sustainability/carbon footprint/water conservation
Particular emphasis given to these items in recent years:
sustainability: no decline in quality of the basic resource
carbon footprint: minimisation of carbon-dioxide etc. emissions
water conservation: efficient use of water
Leisure, Sport and Tourism, Politics, Policy and Planning, 4th edition, Veal, 2017, CABI Tourism Texts

9. Triple bottom line
The idea that organizations should report annually, not just on finance, but on three criteria:
financial performance
social impact
environmental impact
Leisure, Sport and Tourism, Politics, Policy and Planning, 4th edition, Veal, 2017, CABI Tourism Texts

10. Effectiveness and efficiency
Effectiveness: the extent to which a project achieves what it is intended to achieve
Efficiency: is the cost (input) per unit of output
Leisure, Sport and Tourism, Politics, Policy and Planning, 4th edition, Veal, 2017, CABI Tourism Texts

11. Steps in the evaluation process (Table 15.1)
Identify goals
Specify objectives
Devise measures of effectiveness – performance indicators (PIs)
Devise measures of efficiency – performance indicators (PIs)
Specify data collection methods
Collect base-line PI data
Set targets
Collect PI data collection at specified times (e.g. weekly)
Identify and obtain external benchmark data
Compare base-line values of PIs with benchmarks
Deliver verdict
Consider of implications
Leisure, Sport and Tourism, Politics, Policy and Planning, 4th edition, Veal, 2017, CABI Tourism Texts

12. Goals/objectives and KPIs: leisure (Table 15.2)
Leisure participation
1. Goal
Increase participation
2. Objectives
Increase participation to target levels among specified socio-demographic groups and in specified planning zones
3. Effectiveness KPIs
Level of participation for target groups (A, B) and zones (C)
4. Efficiency KPIs
Cost per additional participant
5. Environmental KPIs
Amount of carbon emitted per 1000 customers
6. Data collection methods
Resident survey
Administrative
Special audit
7/8. KPI values
Overall participation:
Participation group A:
Participation group B:
Participation in zone C:
Average net cost/participant-session:
Carbon/participant-session:
Baseline (this year) Target, Year X
30% %
22% %
18% %
15% %
£ £2
1.5 kg kg
Leisure, Sport and Tourism, Politics, Policy and Planning, 4th edition, Veal, 2017, CABI Tourism Texts

13. Goals/objectives and KPIs: tourism (Table 15.2 cont’d)
Tourism development
1. Goal
Stimulated employment growth through tourism
2. Objectives
Increase number of tourists and tourist expenditure to create additional jobs in tourism and related sectors
3. Effectiveness PIs
Number of additional tourists
Number of jobs created
4. Efficiency PIs
Public sector cost per job created
Total cost per job created
5. Environmental PIs
Amount of carbon emitted per 1000 tourists
6. Data collection methods
3. Tourist survey and industry survey
4. Tourist survey, industry survey and administrative
5. Special audit
7/8. KPI values
Number of tourists: 100,000
Total tourist expenditure:
Jobs in tourism & sectors:
Council annual spend/job:
Carbon per tourist bed-night:
Baseline (this year) Target: year X
100, ,000
£25 million £30 million
£ £100
15kg kg
Leisure, Sport and Tourism, Politics, Policy and Planning, 4th edition, Veal, 2017, CABI Tourism Texts

14. Approaches to performance evaluation
Service quality
the CERM PI system (Australia)
the National Benchmarking Service (UK)
importance-performance analysis (generic)
Comprehensive Area Assessments
Arts, entertainment, cultural venues/events
Sport (UK)
Tourism (Australia)
Leisure, Sport and Tourism, Politics, Policy and Planning, 4th edition, Veal, 2017, CABI Tourism Texts

15. Service quality
Systems that evaluate quality of customer service, typically on an annual basis:
CERM PI system (Australia)
National Benchmarking Service (UK)
Importance-performance analysis (generic)
Leisure, Sport and Tourism, Politics, Policy and Planning, 4th edition, Veal, 2017, CABI Tourism Texts

16. Service quality: strategic vs operational goals (Fig. 15.1)
Leisure, Sport and Tourism, Politics, Policy and Planning, 4th edition, Veal, 2017, CABI Tourism Texts

17. CERM PI system (Box 15.1)
Centre for Environmental and Recreation Management (University of South Australia) Performance Indicators system
Version of SERVQUAL approach (Parasuraman et al., 1988)
compares customer service quality expectations with service quality received
Leisure facility managers who use the system collect standard annual data from customer surveys
CERM analyses data and presents results with comparison of averages for similar facilities (benchmarking)
Leisure, Sport and Tourism, Politics, Policy and Planning, 4th edition, Veal, 2017, CABI Tourism Texts

18. CERM PI system (cont’d)
A. Customer service: expectation
and performance
B. Organizational
1. Safe and secure parking
1. Expense recovery of operations (fee
income as % of expenditure)
2. Facility cleanliness
2. Promotion/marketing cost share
3. Value for money
3. Total visits per year
4. Suitable food and drink
4. Visits per square metre
5. Staff friendliness
5. Water costs per visit
6. Pool water cleanliness
6. Fit of socio-demographic profile of
centre users to that of the local
community
7. Behaviour of others
8. Problem resolution
9. Overall satisfaction
10. Behavioural intentions
Items rated using Likert-type scales
Leisure, Sport and Tourism, Politics, Policy and Planning, 4th edition, Veal, 2017, CABI Tourism Texts

19. National Benchmarking Service (Box 15.2)
UK Service similar to CERM PI sponsored by Sport England and operated by the Sport Industries Research Centre (SIRC) (Sheffield Hallam University)
Leisure, Sport and Tourism, Politics, Policy and Planning, 4th edition, Veal, 2017, CABI Tourism Texts

20. Analysis of data: importance-performance technique
Similar to SERVQUAL, but can be used:
in organizational decision making, as indicated by Harper and Balmer (1989);
in relation to perceived benefits of public leisure services, as a form of consumer consultation (Sieganthaler, 1994);
to measure customer satisfaction (Langer, 1997: 147); and
to analyse performance data.
Leisure, Sport and Tourism, Politics, Policy and Planning, 4th edition, Veal, 2017, CABI Tourism Texts

21. Importance-performance analysis: customer service evaluation example
8 service items (A–H) scored by customers of one facility in relation to:
importance/expectations
performance – service actually received
Results in Fig. 15.2
Leisure, Sport and Tourism, Politics, Policy and Planning, 4th edition, Veal, 2017, CABI Tourism Texts

22. Importance-performance analysis example (Fig 15.2)
Doing well
Wasted resources?
Cause for concern
Leisure, Sport and Tourism, Politics, Policy and Planning, 4th edition, Veal, 2017, CABI Tourism Texts

23. Importance-performance analysis: comparison example (Table 15.5)
The same facility over 4 years (4 cases): scored on 8 service quality items (could also be applied to 4 different facilities)
Different service quality items have different levels of importance/expectation: cored by management or users
Leisure, Sport and Tourism, Politics, Policy and Planning, 4th edition, Veal, 2017, CABI Tourism Texts

24. Importance-performance analysis: comparison (Table 15.5)
Service quality items
a. Importance scores
Case 1
Case 2
Case 3
Case 4
b. Performance scores A
4.3
1.2 2
2.7
4.5 B 3
4.8
4.9 C 4
4.2 D
2.5
3.5
3.7 E
1.5
1.6
1.7
1.9 F G H
4.7
Total score (sum of a x b)
78.9 84
89.6
104.3
% change
6.4
6.7
16.1
Attendances ('000s) 50
51.5
53.3
57.5
7.9

25. Comprehensive Area Assessments (CAAs)
Pre-2010 nationwide system of performance evaluation for local councils, including leisure, sport, arts
Abolished by Conservative/Liberal Democrat coalition to cut down ‘red tape’ etc.
Details included in book as example of an ambitious scheme
Legacy of national sport/arts participation surveys remains
Leisure, Sport and Tourism, Politics, Policy and Planning, 4th edition, Veal, 2017, CABI Tourism Texts

26. CAA: selected National Indicators (NIs) (Table 15.6)
Participation in sport and active recreation
Definition
% of the adult population (16+) in a local area who participate in sport and active recreation, at moderate intensity
Data source
The annual Active People Survey (Sport England)
NI9
Use of public libraries
% of the adult population who say they have used a public library service at least once in the last 12 months
As for NI8
NI10
Visits to museums and galleries
% of the adult population who visit a museum or gallery at least once in the last 12 months in leisure time
NI11
Engagement with the arts
% of the adult population who attended an arts event or participated in arts activity 3+ times in last year
Leisure, Sport and Tourism, Politics, Policy and Planning, 4th edition, Veal, 2017, CABI Tourism Texts

27. Other examples Arts, cultural venues and events Sport Tourism
arts items included in the CAA system (see above)
in 1991, the UK Audit Commission (now abolished) produced a system of PIs for the arts/cultural events (see Appendix 15A)
Sport
see the Sport England strategic plan in Appendix 15B
Tourism
see Tourism NSW Masterplan: Appendix 15C
Leisure, Sport and Tourism, Politics, Policy and Planning, 4th edition, Veal, 2017, CABI Tourism Texts

28. Goals and performance indicators
Throughout the book, numerous goals of public leisure/sport/tourism services are mentioned.
All can be expressed/evaluated in terms of performance indicators.
Appendix 15D
Leisure, Sport and Tourism, Politics, Policy and Planning, 4th edition, Veal, 2017, CABI Tourism Texts