COMPLEMENTARY TEACHING MATERIALS CABI TOURISM TEXTS 4th Edition Leisure, Sport and Tourism, Politics, Policy and Planning A. J. Veal COMPLEMENTARY TEACHING MATERIALS

Performance Evaluation Leisure, Sport and Tourism, Politics, Policy and Planning, 4th Edition CHAPTER 15 Performance Evaluation Leisure, Sport and Tourism, Politics, Policy and Planning, 4th edition, Veal, 2017, CABI Tourism Texts

Outline Introduction Evaluation in context Steps in the evaluation process Approaches Goals and performance indicators Leisure, Sport and Tourism, Politics, Policy and Planning, 4th edition, Veal, 2017, CABI Tourism Texts

Evaluation in Context Economic evaluation – see Chapter 14 Managerialism and privatization Forms of evaluation Sustainability/carbon footprint/water conservation Triple bottom line accounting Effectiveness and efficiency Leisure, Sport and Tourism, Politics, Policy and Planning, 4th edition, Veal, 2017, CABI Tourism Texts

Managerialism and privatization Managerialism: the application of formal/rational management practices Often used (typically negatively) in regard to applying private sector-style practices in the public sector Arises particularly with privatization or ‘contracting out’ to the private sector – implying that only financial criteria are used and traditional public sector goals are being ignored However: Leisure, Sport and Tourism, Politics, Policy and Planning, 4th edition, Veal, 2017, CABI Tourism Texts

Managerialism and privatization (cont’d) Contracts should include all the requirements of the public sector. Even public sector organizations need to think about best use of capital. Tourism enterprises, while commercial in nature, can also have wide community impacts. When services are contracted out to private companies, the public agency (e.g. council) still has responsibilities regarding the quality of life of the community. This also applies to single-purpose agencies (e.g. arts, sport) in their areas of concern. Competitive tendering for public services can ‘concentrate the mind’ of the public agency to spell out its goals. Leisure, Sport and Tourism, Politics, Policy and Planning, 4th edition, Veal, 2017, CABI Tourism Texts

Forms of evaluation Routine internal – e.g. annual Strategic – relating to strategic planning Accountability – inclusion in official annual reports Ad hoc, one-off evaluations Internal versus comparative internal comparison with previous years or between units comparative: comparison with external ‘benchmarks’ Leisure, Sport and Tourism, Politics, Policy and Planning, 4th edition, Veal, 2017, CABI Tourism Texts

Sustainability/carbon footprint/water conservation Particular emphasis given to these items in recent years: sustainability: no decline in quality of the basic resource carbon footprint: minimisation of carbon-dioxide etc. emissions water conservation: efficient use of water Leisure, Sport and Tourism, Politics, Policy and Planning, 4th edition, Veal, 2017, CABI Tourism Texts

Triple bottom line The idea that organizations should report annually, not just on finance, but on three criteria: financial performance social impact environmental impact Leisure, Sport and Tourism, Politics, Policy and Planning, 4th edition, Veal, 2017, CABI Tourism Texts

Effectiveness and efficiency Effectiveness: the extent to which a project achieves what it is intended to achieve Efficiency: is the cost (input) per unit of output Leisure, Sport and Tourism, Politics, Policy and Planning, 4th edition, Veal, 2017, CABI Tourism Texts

Steps in the evaluation process (Table 15.1) Identify goals Specify objectives Devise measures of effectiveness – performance indicators (PIs) Devise measures of efficiency – performance indicators (PIs) Specify data collection methods Collect base-line PI data Set targets Collect PI data collection at specified times (e.g. weekly) Identify and obtain external benchmark data Compare base-line values of PIs with benchmarks Deliver verdict Consider of implications Leisure, Sport and Tourism, Politics, Policy and Planning, 4th edition, Veal, 2017, CABI Tourism Texts

Goals/objectives and KPIs: leisure (Table 15.2) Leisure participation 1. Goal Increase participation 2. Objectives Increase participation to target levels among specified socio-demographic groups and in specified planning zones 3. Effectiveness KPIs Level of participation for target groups (A, B) and zones (C) 4. Efficiency KPIs Cost per additional participant 5. Environmental KPIs Amount of carbon emitted per 1000 customers 6. Data collection methods Resident survey Administrative Special audit 7/8. KPI values Overall participation: Participation group A: Participation group B: Participation in zone C: Average net cost/participant-session: Carbon/participant-session: Baseline (this year) Target, Year X 30% 32% 22% 30% 18% 30% 15% 30% £2 £2 1.5 kg 1.2 kg Leisure, Sport and Tourism, Politics, Policy and Planning, 4th edition, Veal, 2017, CABI Tourism Texts

Goals/objectives and KPIs: tourism (Table 15.2 cont’d) Tourism development 1. Goal Stimulated employment growth through tourism 2. Objectives Increase number of tourists and tourist expenditure to create additional jobs in tourism and related sectors 3. Effectiveness PIs Number of additional tourists Number of jobs created 4. Efficiency PIs Public sector cost per job created Total cost per job created 5. Environmental PIs Amount of carbon emitted per 1000 tourists 6. Data collection methods 3. Tourist survey and industry survey 4. Tourist survey, industry survey and administrative 5. Special audit 7/8. KPI values Number of tourists: 100,000 Total tourist expenditure: Jobs in tourism & sectors: Council annual spend/job: Carbon per tourist bed-night: Baseline (this year) Target: year X 100,000 120,000 £25 million £30 million 500 600 £100 £100 15kg 13 kg Leisure, Sport and Tourism, Politics, Policy and Planning, 4th edition, Veal, 2017, CABI Tourism Texts

Approaches to performance evaluation Service quality the CERM PI system (Australia) the National Benchmarking Service (UK) importance-performance analysis (generic) Comprehensive Area Assessments Arts, entertainment, cultural venues/events Sport (UK) Tourism (Australia) Leisure, Sport and Tourism, Politics, Policy and Planning, 4th edition, Veal, 2017, CABI Tourism Texts

Service quality Systems that evaluate quality of customer service, typically on an annual basis: CERM PI system (Australia) National Benchmarking Service (UK) Importance-performance analysis (generic) Leisure, Sport and Tourism, Politics, Policy and Planning, 4th edition, Veal, 2017, CABI Tourism Texts

Service quality: strategic vs operational goals (Fig. 15.1) Leisure, Sport and Tourism, Politics, Policy and Planning, 4th edition, Veal, 2017, CABI Tourism Texts

CERM PI system (Box 15.1) Centre for Environmental and Recreation Management (University of South Australia) Performance Indicators system Version of SERVQUAL approach (Parasuraman et al., 1988) compares customer service quality expectations with service quality received Leisure facility managers who use the system collect standard annual data from customer surveys CERM analyses data and presents results with comparison of averages for similar facilities (benchmarking) Leisure, Sport and Tourism, Politics, Policy and Planning, 4th edition, Veal, 2017, CABI Tourism Texts

CERM PI system (cont’d) A. Customer service: expectation and performance B. Organizational 1. Safe and secure parking 1. Expense recovery of operations (fee income as % of expenditure) 2. Facility cleanliness 2. Promotion/marketing cost share 3. Value for money 3. Total visits per year 4. Suitable food and drink 4. Visits per square metre 5. Staff friendliness 5. Water costs per visit 6. Pool water cleanliness 6. Fit of socio-demographic profile of centre users to that of the local community 7. Behaviour of others 8. Problem resolution 9. Overall satisfaction 10. Behavioural intentions Items rated using Likert-type scales Leisure, Sport and Tourism, Politics, Policy and Planning, 4th edition, Veal, 2017, CABI Tourism Texts

National Benchmarking Service (Box 15.2) UK Service similar to CERM PI sponsored by Sport England and operated by the Sport Industries Research Centre (SIRC) (Sheffield Hallam University) Leisure, Sport and Tourism, Politics, Policy and Planning, 4th edition, Veal, 2017, CABI Tourism Texts

Analysis of data: importance-performance technique Similar to SERVQUAL, but can be used: in organizational decision making, as indicated by Harper and Balmer (1989); in relation to perceived benefits of public leisure services, as a form of consumer consultation (Sieganthaler, 1994); to measure customer satisfaction (Langer, 1997: 147); and to analyse performance data. Leisure, Sport and Tourism, Politics, Policy and Planning, 4th edition, Veal, 2017, CABI Tourism Texts

Importance-performance analysis: customer service evaluation example 8 service items (A–H) scored by customers of one facility in relation to: importance/expectations performance – service actually received Results in Fig. 15.2 Leisure, Sport and Tourism, Politics, Policy and Planning, 4th edition, Veal, 2017, CABI Tourism Texts

Importance-performance analysis example (Fig 15.2) Doing well Wasted resources? Cause for concern Leisure, Sport and Tourism, Politics, Policy and Planning, 4th edition, Veal, 2017, CABI Tourism Texts

Importance-performance analysis: comparison example (Table 15.5) The same facility over 4 years (4 cases): scored on 8 service quality items (could also be applied to 4 different facilities) Different service quality items have different levels of importance/expectation: cored by management or users Leisure, Sport and Tourism, Politics, Policy and Planning, 4th edition, Veal, 2017, CABI Tourism Texts

Importance-performance analysis: comparison (Table 15.5) Service quality items a. Importance scores Case 1 Case 2 Case 3 Case 4 b. Performance scores A 4.3 1.2 2 2.7 4.5 B 3 4.8 4.9 C 4 4.2 D 2.5 3.5 3.7 E 1.5 1.6 1.7 1.9 F G H 4.7 Total score (sum of a x b) 78.9 84 89.6 104.3 % change   6.4 6.7 16.1 Attendances ('000s) 50 51.5 53.3 57.5 7.9

Comprehensive Area Assessments (CAAs) Pre-2010 nationwide system of performance evaluation for local councils, including leisure, sport, arts Abolished by Conservative/Liberal Democrat coalition to cut down ‘red tape’ etc. Details included in book as example of an ambitious scheme Legacy of national sport/arts participation surveys remains Leisure, Sport and Tourism, Politics, Policy and Planning, 4th edition, Veal, 2017, CABI Tourism Texts

CAA: selected National Indicators (NIs) (Table 15.6) Participation in sport and active recreation Definition % of the adult population (16+) in a local area who participate in sport and active recreation, at moderate intensity Data source The annual Active People Survey (Sport England) NI9 Use of public libraries % of the adult population who say they have used a public library service at least once in the last 12 months As for NI8 NI10 Visits to museums and galleries % of the adult population who visit a museum or gallery at least once in the last 12 months in leisure time NI11 Engagement with the arts % of the adult population who attended an arts event or participated in arts activity 3+ times in last year Leisure, Sport and Tourism, Politics, Policy and Planning, 4th edition, Veal, 2017, CABI Tourism Texts

Other examples Arts, cultural venues and events Sport Tourism arts items included in the CAA system (see above) in 1991, the UK Audit Commission (now abolished) produced a system of PIs for the arts/cultural events (see Appendix 15A) Sport see the Sport England strategic plan in Appendix 15B Tourism see Tourism NSW Masterplan: Appendix 15C Leisure, Sport and Tourism, Politics, Policy and Planning, 4th edition, Veal, 2017, CABI Tourism Texts

Goals and performance indicators Throughout the book, numerous goals of public leisure/sport/tourism services are mentioned. All can be expressed/evaluated in terms of performance indicators. Appendix 15D Leisure, Sport and Tourism, Politics, Policy and Planning, 4th edition, Veal, 2017, CABI Tourism Texts