1 Session C: Evaluation James Farnham 3 June 2009

2 Content Context for evaluation Principles of evaluation Stepped approach to evaluation

3 The context for evaluation

4 Evaluation benefits Continuous learning and improvement ‒ improved effectiveness of marketing programmes ‒ improved marketing efficiency Better knowledge and insight for how to influence consumer behaviour Accountability More informed budgeting process Better forecasting of outcomes ‒ better management of expectations re results ‒ input to ‘policy levers’ ‒ enhanced credibility of social marketing

5 Moving from a static, linear approach to a more dynamic, continuous improvement journey  Test-Learn-Refine (TLR) EvaluationPlanning EvaluationPlanning

6 Evaluation perspectives AccountabilityTest-Learn-Refine Social constructionistsScientific realists ‘What gets measured is what matters’ Practice Theory Vs ‘No single accepted best practice’

7 A ‘line of sight’ to behaviour change is required across a range of measures Output measures eg awareness Input measures eg toolkits, TVRs Behavioural change measures Reduced consumption Intermediate measures eg attitudes and response Outtake measures eg understanding & knowledge Outcome/impact measures Reduced ARHA

8 Principles of evaluation

9 Ten marketing evaluation principles Better evaluation will improve both our marketing results and our accountability – it is essential, not a ‘nice to have’ Five themes ‒ Culture ‒ Process ‒ Measurement ‒ Influencing behaviour ‒ Test-Learn-Refine

10 Ten marketing evaluation principles Culture 1. Hard wire evaluation into every stage of the marketing process, focusing on active learning and rapid application 2. Evaluate every piece of activity at a proportionate level - between 5% and 10% of total spend

11 3.Develop a formal evaluation plan at the outset of the marketing process, including clear objectives. Each plan will address the following questions: Ten marketing evaluation principles What will be evaluated? (i.e. what is the marketing activity about and in what Policy context does it exist?) What aspects of the marketing activity will be considered when judging its performance? What standards (i.e. type or level of performance) must be reached for the marketing activity to be considered successful? What evidence will be used to indicate how the marketing activity has performed? What relevant lessons from previous marketing activities can be applied? What conclusions regarding marketing activity performance are justified by comparing the available evidence to the selected standards? How will the lessons learned from the evaluation be used to improve public health effectiveness? Process

12 Ten marketing evaluation principles Measurement 4. Collect robust, consistent data on marketing activity 5. Ensure all marketing evaluations focus on a range (c. 10) of KPI measures that are critical to success – both in terms of behavioural influence and ROI/cost-effectiveness. Avoid using awareness as an isolated measure of success Influencing behaviour 6. Put behaviour change at the heart of the evaluation thinking, capitalising on existing behavioural theory

13 Ten marketing evaluation principles Test-learn-refine 7. Use a rigorous test-learn-refine process within and between channels 10. Question activities that cannot be evaluated 9. Evaluation should be used to encourage innovation, not create risk aversity 8. Try to test locally prior to full PCT exposure, establishing control areas if possible

14 Stepped approach to evaluation

15 Is it a step? NSMC Total Process Planning Model

16 PHASE 1: SCOPE - gaining early clarity Step 1 Form an Evaluation Team Step 2 Lessons learned from previous evaluations Step 3 Clarifying objectives Step 4 Setting the evaluation budget Step 5 Evaluation plan 1

17 PHASE 2: DEVELOP – working up the overall approach Step 1 Using behavioural theory Step 2 Defining the behavioural goals Step 3 Defining the KPIs Step 4 Writing the Evaluation Plan Step 5 Briefing the implementation agencies

18 PHASE 3: IMPLEMENT – iteratively designing the evaluation detail with all agencies Step 1 Evaluation development with implementation agencies Step 2 Designing and setting up the measurement approaches Step 3 Getting the most out of evaluation

19 PHASE 4: LEARN – short term and long term Step 1 Immediate post-launch reporting Step 3 Final Evaluation Report Step 2 Interim Evaluation Step 4 Sharing and capturing the learnings from the Evaluation Report