National Obesity Observatory The Standard Evaluation Framework National Obesity Observatory The Standard Evaluation Framework Kath Roberts

What is evaluation: bring me the evidence!

One question….. If evaluation was an animal… what would it be?

Scary?

Tricky to get hold of…

Prickly? OTHERS…

Does it work? Has it achieved its objectives? What happened? Has it worked? Could it work? What questionnaire should I use? Do I need a double blind randomised control trial? Was it effective? Project

Standard Evaluation Framework (SEF) for weight management interventions

Aim of the SEF: To support high quality, consistent evaluation of weight management interventions in order to increase the evidence base

Objectives of the SEF: Increase the proportion of weight management interventions that are evaluated Increase the quality of evaluations Develop a core dataset to increase the consistency and comparability of evaluations Provide wider evaluation support

Target audience for the SEF: PCT commissioners (to support service specifications) PCT & DH obesity leads Organisations running interventions External evaluators The NE region requires the SEF is incorporated as an essential component of all new weight management programmes?

Why should we use the SEF? Number of interventions Speculative interventions Evidence-based interventions Time

What interventions should it apply to? Any intervention that explicitly sets out to manage or reduce body weight as an intermediate or long-term health outcome (including the primary prevention of weight gain) Projects focusing on diet, physical activity, or both in combination Intended to be applicable to a range of approaches including interventions conducted with individuals on a one-to-one basis, or with people in groups; and in clinical or community settings.

The SEF is not intended for: Medical treatment e.g. bariatric surgery & prescribing Wider environmental interventions e.g. changes to the built environment (although the SEF could support the evaluation of such interventions alongside wider evaluation criteria)

Consultation Processes: Developed in consultation with academic experts, representatives from Public Health Observatories, Primary Care Trusts, Government Offices, Local Authorities and other relevant organisations and bodies Initial list of criteria consulted on using an online questionnaire - followed up by individual interviews Two consultation meetings to finalise criteria and guidance. NB the SEF itself will also be evaluated.

What is in the SEF document? An introduction to the concepts of evaluation A list of data collection criteria divided into ‘essential’ and ‘desirable’ A guide to using the criteria and collecting the data

Introduction to Evaluation: What is evaluation and why do it? What are the different types of evaluation? What are the differences between evaluation, audit and monitoring? What sorts of information need to be collected to carry out evaluations? What sorts of issues need to be considered as part of an evaluation?

SEF Criteria: 1.Programme details e.g. name of intervention, aims/objectives 2.Demographics of individual participants e.g. age, sex, ethnicity 3.Baseline data e.g. height/weight, dietary and physical activity behaviours 4.Follow-up data (impact and process evaluation) 5.Analysis and interpretation

Supporting SEF guidance: Details about why criteria have been categorised as ‘essential’ or ‘desirable’ What the data can and cannot tell you Standardised methods for collecting some of the data Suggested standardised definitions for data Suggested data collection tools Signposting to further guidance and useful information

Recommended Physical Activity Measurement Tools: 4 instruments met the following criteria –Relatively easy and practical to administer –Mean difference between energy estimated by self-report and doubly labelled water <40% –Instrument should have been used in a trial of exercise promotion and detected a significant difference between intervention and control participants –Population in the validity study generalisable to wider population Appendix to SEF by Dr Melvyn Hillsdon

Recommended Physical Activity Measurement Tools: International Physical Activity Questionnaire (IPAQ) long version –7day recall –Valid in year olds only Stanford 7-day recall –Self report for sleep, moderate, hard and very hard activities –Valid in adults only

Recommended Physical Activity Measurement Tools: 7-day physical activity diary –Self-completion diary ‘ticking’ 15min blocks of activity –Valid in adults only New Zealand Physical Activity Questionnaire –7day Self-completion recall, using prompt cards –Valid in middle aged adults only

Limitations: No existing physical activity measure perfectly meets all criteria Lack of suitable measures for children No gold standard for measuring sedentary behaviours Most measures have quite complex scoring systems

Future SEF products: Stand alone evaluation guide Stand alone criteria tables Example case studies Excel template for audit data Excel template for individual study data Online audit (currently taking place in the NE), national audit under development. Dietary tool guide Commissioners guide to weight management interventions Others…

Questions? Queries to: PHINE SEF support group:

Interactive Workshop Aims: –Share how you plan to use the SEF in your locality. –Discuss using the SEF for an intervention(s) running or due to run in your locality and identify:  Possible barriers/difficulties to using the SEF.  Possible solutions to these barriers and how NOO can best support these.  The advantages the SEF would bring to this intervention.

Workshop Feedback: Each group to report back: 1.How is the SEF going to be used in your locality? 2.What possible barriers are there to using the SEF? 3.How could these barriers be overcome? 4.What could NOO provide to support use of the SEF further?