27-28 May 2004: Andy Pringle leeds metropolitan university Local Exercise Action Pilots The National Evaluation General Lessons Learnt So Far! 27-28 May 2004: Andy Pringle leeds metropolitan university Andy Pringle Leeds Metropolitan University 1

Who Am I? Andy Pringle Carnegie Faculty of Sport & Education. Leeds Metropolitan University. Senior Lecturer in Active Lifestyles and Health. Project Manager for the National Evaluation of LEAP: Local Exercise Action Pilots. For further information: Andy Pringle Senior Lecturer: Active Lifestyles & Health Carnegie Faculty of Sport & Education Leeds Metropolitan University Fairfax 212, Beckett Park Campus LEEDS LS6 3QS. A.Pringle@LEEDSMET.AC.UK 0113 283 2600 Ext.....7409. WWW.LEEDSMET.AC.UK

Outline What is LEAP? General overview of the National Evaluation Framework for LEAP. General lessons that have been learnt in developing and managing the National Evaluation Framework. These are generic and apply to whatever population physical activity interventions are targeted including older people. Some suggestions of what we would do differently next time around. This presentation will provide an overview of LEAP, the National Evalaution Framework for LEAP and offer some general lessons that have been learnt in managing the evaluation,. These are relevant what ever group physical activity interventions are being based upon including that of older people. 2

What is LEAP? Local Exercise Action Pilots: This is a £2.6 million contract awarded by Department of Health, Sport England and The Countryside Agency in 2003 to 10 Primary Care Trusts (PCTs/sites) to deliver and evaluate physical activity interventions. There are 10 pilot sites from the North East to the South West of England using a variety of physical activity interventions which are being evaluated. LEAP will run for two years. LEAP is a £2.6 million contract awarded by Department of Health, Sport England and The Countryside Agency to 10 Primary Care Trusts (PCTs). In 2003 the Carnegie Faculty of Sport & Education at Leeds Metropolitan University was successful in obtaining a two-year contract to evaluate LEAP, Local Exercise Action Projects. The basic aim of the evaluation project is to establish what are the most effective interventions at increasing levels of physical activity by people from deprived communities. Interventions are being run by PCTs in collaboration with local partner agencies aimed at increasing the physical activity levels of a range of target groups including older people through a range of physical activity interventions. 1. To provide information that shows how well the interventions are working, so they may be locally sustained beyond the LEAP project through incorporation in mainstream PCT activity; 2.To facilitate the national evaluation that will provide an overall summary of the efficacy and effectiveness of the LEAP projects, which could lead to more central funding and support for physical activity interventions; 3.To understand what works well and why in different interventions so that good practice can be disseminated amongst those engaged in planning and delivering physical activity intervention programmes; and 4.Funding for LEAP projects is associated with the expectation that each PCT will provide an evaluation of the physical activity interventions that it delivers.

The Interventions The interventions follow 7 themes: 1. Exercise Referral 2. Classes and Groups 3. Peer Mentoring 4. Campaigns and Directories 5. Motivational Interviews 6. Outdoors and Transport 7. Training Leaders and Coordinators with a range of physical activity target groups. Including older people. The seven themes are: 1. Exercise Referral Schemes; 2. Peer Mentoring; 3. Campaigns and Directories; 4. Classes and Groups; 5. Leaders Co-ordinators and Training; 6. Outdoor Activity; and 7. Motivational Interviews. In some cases an intervention is only linked to one theme. In some cases the intervention may be linked to more than one theme (e.g. exercise referral and outdoor activity). These 7 themes and their associated evaluation questions will generate data that can be compared across similar interventions or parts of interventions Examples of interventions involving Older people include: Senior Peer Mentoring, Classes and groups, Motivational Interviewing in Primary Care Settings, the promotion of Walking in Parks.

What is the Role of leeds metropolitan university? The main roles and responsibilities of the partners within the monitoring and evaluation process can be summarized as follows: Leeds Metropolitan University ·Develop national evaluation framework. ·Support 10 PCTs in development and delivery of local evaluations. ·Integrate findings of MORI pre and post survey into national evaluation. ·Gather data for and integrate cost analysis conducted by MATRIX (LMU partner) into national evaluation. ·Conduct “what works, why and how” evaluation across 10 PCTs from perspective of providers. ·Develop and carry out selected case studies (in-depth focus groups, interviews, observation, and direct measurements of activity, lever in clinical data and other measurements). ·Provide interim and final reports on national evaluation to LEAP project management team.

What is the National Evaluation Framework? Leeds Met has the contract to evaluate LEAP working in partnership with other organizations and consultants. The basic aim of the National Evaluation Framework is to establish: Which community physical activity interventions are effective in getting people physically active? Also looking to find out other information as well. The NEF is the infrastructure that has been put in place to evaluate the effectiveness of LEAP.

The National Evaluation Framework Leeds Metropolitan University + Matrix MESO EVALUATION MACRO EVALUATION MICRO EVALUATION LEAP SURVEY MORI + BHF GENERIC QUESTIONS 1. Physical Activity 2. Cost Effectiveness 3. What works, why and how: providers perspective 4. Participants Perspective INTERVENTION THEMES 1. Exercise Referral 2. Classes and Groups 3. Peer Mentoring 4. Campaigns and Directories 5. Motivational Interviews 6. Outdoors and Transport 7. Training Leaders and Coordinators LOCAL EVALUATIONS CASE STUDIES EACH INTERVENTION Run by 10 PCTs Supported by Leeds Met Team Survey Pre (Nov 2003) and Post (Nov 2005) LEAP Interventions This document will provide an overview of the National Evaluation Framework. The national evaluation framework for LEAP is summarised in Figure 1. This figure shows how the elements of the evaluation link to each other to form the national evaluation. There are three elements to the evaluation: the MACRO, MESO and MICRO evaluations. PCTs are responsible for the development and delivery of a local evaluation (MICRO), both of which are supported by the national evaluation team. The national evaluation is dependent on data that is collected through each of the local evaluations, especially that which can be collected as part of the good practice of running effective physical activity interventions, and where there is direct contact with the participants accessing the various interventions. PCTS are located in the following regional areas of England North East North West West Yorkshire East Midland West Midland London South East South West 5 LEAP PCTs Dudley Great Yarmouth North Kirklees Wandsworth West Cornwall Comparison Group

Three Levels 1. MACRO 2. MESO 3. MICRO 1. The MACRO evaluation comprises two elements: 1. MORI Survey 2. Case Study data collection that addresses a set of generic questions common to the evaluation of all interventions. 2. The MESO evaluation comprises seven intervention themes where interventions across the 10 participating PCTs can be compared using data generated by addressing theme specific evaluation questions. 3. The MICRO evaluation comprises the 10 separate local evaluations incorporating a case study of each intervention. There are three parts to the National Evaluation Framework: The “MACRO” evaluation comprises two elements: A survey conducted before and after the two year period of LEAP interventions in five of the participating PCTs, together with data collection that addresses a set of generic questions common to the evaluation of all interventions (see Figure 1). The “MESO” evaluation comprises seven intervention themes where interventions across the 10 participating PCTs can be compared using data generated by addressing theme specific evaluation questions (see Figure 1). The “MICRO” evaluation comprises the 10 separate local evaluations incorporating a case study of each intervention (see Figure 1). These are the responsibility of the PCTs, with support given by the national evaluation team at Leeds Metropolitan University (“LMU”). The national evaluation of LEAP is therefore a combination of the three elements of evaluation as shown in Figure 1 (MACRO, MESO, and MICRO). Delivery of the national evaluation is dependent on the successful implementation of each of the three elements of the evaluation.

Macro 1. The MORI Survey Macro Level 1. An independent MORI survey. Telephone survey of 800 participants conducted in five of the PCTs before (November and December 2003), and after (November and December 2005), the two years of physical activity intervention. The macro level has two parts to it: 1. An independent MORI survey conducted in five of the PCTs before (November and December 2003) and after (November and December 2005), the two years of physical activity intervention. There is a control PCT that is not one of the 10 PCTs that is receiving LEAP funding.

The MORI Survey 10 minute telephone questionnaire. The survey instrument covers questions on: Physical activity category. Awareness of local physical activity campaigns. Knowledge and attitudes to physical activity. Social climate, intention to be active. Demographics.

Macro 2. A Case Study in the 10 PCTS/Sites The majority of the information for the N.E.F is being collected at the PCT level by PCTS and associates. A common set of questions across all interventions where appropriate. 1. Baseline and change in physical activity status. 2. Effect of the intervention on participant’s FITT of participation. 3. Who the participants are and where are they are from? Contact details & demographics. 4. Is the activity new or a replacement physical activity. 2. A set of common questions applied across all the 10 PCTs which include: Physical Activity associated with LEAP. Baseline physical; activity Change in physical activity status Influence of the LEAP intervention on the participants parameters of physical activity. Demographically information Cost Analysis. What works why and how? – the perspective of the physical activity providers. The participants’ perspective on the physical activity experience.

The National Evaluation Framework 5. Cost analysis of planning, designing and delivering the LEAP interventions Data collection on inputs (e.g.. personnel), outputs (e.g.. classes) and outcomes (e.g.. change in physical activity behavior). Working in collaboration with MATRIX CR. 6. Qualitative evaluation of planning, delivery and evaluation of the interventions: What works, why and how? Undertaken by the Leeds Met Field Researchers.

What are the lessons that have been learnt….so far? What action could be taken to prevent and manage these issues in the future?

Collaboration It is clear that there are some “first rate” examples of collaborative working between the PCTs/Pilot Sites and Leeds Met in the development of the Evaluation Framework. However there are examples of less enthusiasm for the evaluation (in its current context). The perception of what evaluation constitutes and how much work is needed: Resource base available. Skills available for evaluation. There are some excellent example of collaborative working between PCT and Leeds Met. However there remains some frustrations on behalf of the PCT and these are underpinned in part by Perception of what evaluation constitutes and how much work is required. The resource base available to undertake evaluation and how this might compare with that needed to develop the intervention. Tensions between resources spent on evaluation and delivery of the interventions.. Skills available in the PCT for evaluation.

Specific information There are examples of where the PCTS have under-estimated/been unclear of the size of the work commitment of the evaluation component for LEAP. Action: Provision of clear and specific information of what the PCTs commitment is at the outset is needed. Specifically what the magnitude of the task is? There are examples of where the PCTS have under-estimated/been unclear of the size of the work commitment of the evaluation component for LEAP. Provision of clearer and specific information of what the PCTs commitment is at the outset is needed. Specifically what the magnitude of the task is? This might help manage any expectations that the evaluation commitment is going to require “ alight touch”!

Clarity on Roles and Responsibilities The specific types of interventions to be evaluated. Finalizing the design of the evaluation framework (to Inc tools, dates of data capture. Leeds Met to be involved in the site selection.

Resources: Money & Time Resources will always be a contentious issue. Health Promotion writers talk of the need to ensure that evaluation is supported with the necessary resources (Green & Tones, 2004, Naidoo & Wills, 2000). It is important that evaluation frameworks are well supported with resources so that the the best possible job can be done.

Other Issues Choice of tools: Local Ethical Clearance. Particularly for older measuring physical activity levels in people. IPAQ: Recall of information. Where English is not spoken. Objective measures of physical activity: Anxiety of some of the sites. Local Ethical Clearance. Tensions between the job of evaluation and delivery of physical activity interventions: Clear Information.

For further information: Andy Pringle Senior Lecturer: Active Lifestyles & Health Carnegie Faculty of Sport & Education Leeds Metropolitan University Fairfax 212, Beckett Park Campus LEEDS LS6 3QS. A.Pringle@LEEDSMET.AC.UK 0113 283 2600 Ext.7409. WWW.LEEDSMET.AC.UK

leeds metropolitan university Andy Pringle Senior Lecturer: Active Lifestyles & Health