1. Sigried Caspar
European Commission, DG Employment & Social Affairs Moderator

2. Céline Mas
Occurrence communication research & consulting company, France

3. Workshop A7: how to evaluate your communication activities 2014-2020
December 10, 2013 – Brussels
Céline Mas
Occurrence est
certifiée ISO depuis 2004

4. John Wanamaker Inventor of mass retailing in the United States
"Half the money I spend on advertising is wasted; the trouble is I don’t know which half."
John Wanamaker Inventor of mass retailing in the United States
Occurrence - Workshop A7 - 10/12/2013

5. €  Key issue How much does it contribute? What does it contribute?
Communications are under pressure:
How much does it cost?
View it as an investment and not as an expense
Provide the resources to prove your effectiveness: 5% to evaluation.
"How much does overlooked inefficiency cost?“ : if you can not assess it, you can not improve it! €
How much does it contribute? 
What does it contribute? 5
Occurrence - Workshop A7 - 10/12/2013

6. Most of the time, evaluation is occasional or partial.
The missing link in a virtuous circle
Communication plan
A goals and resources « contract »  with institution or company management
Most of the time, evaluation is occasional or partial.
It rarely shapes dialogue between the communications team and the other decision makers.
Actions
A goals and resources « contract »  with company management
Assessment
Reporting on the achievement of the targets, or on the progress and effectiveness of the implementation
Occurrence - Workshop A7 - 10/12/2013

7. Behaving like any other function
Communications must not exclude itself from Quality and Operating Excellence systems
Communications is a job and a skill; it must include ongoing improvement procedures
The Deming Wheel
Ongoing improvement
Quality management system 7
Occurrence - Workshop A7 - 10/12/2013

8. The 4 main benefits of an evaluation
1. PROMOTING
2. MANAGING
Allocating resources in accordance with performance indicators
Identifying the most effective initiatives for achieving your various objectives
Circulating results and performances to other departments/teams
Achieving investment choices that are based on targets, and not on expenditures on resources
3. SAVING TIME
4. SHARING
Prioritizing/Sorting initiatives by order of effectiveness
Concentrating your efforts and budget on the most effective initiatives
Gathering all the activity and effectiveness data
Highlighting best practices
AND SPECIFIC TO THE PUBLIC SECTOR
Giving evidence of a sound use of public money
Reinforce citizen’s trust 8
Occurrence - Workshop A7 - 10/12/2013

9. 2 main categories of performance (KPI) indicators
Activity
including Resources
(What?)
e.g. Number of initiatives/tools,
type of initiatives/tools, assessment of the content issued
(Press releases, and internal communications),
Ressources: Who? How much? How long?
Effectiveness
including the Audiences
(For what purpose?)
e.g. Memorization, Understanding, Buy-in, Incentive, Transformation, Satisfaction, Improving the brand's image, and satisfying internal customers
Audience: How many people attended? How many Likes ? How many readers? …
A third, highly practical approach is possible: assessing the satisfaction of (internal) customers
Defining performance thresholds for each indicator 9
Occurrence - Workshop A7 - 10/12/2013

10. Process : global picture
Target audience
Communications objectives
Communications Dashboard communications plan
Segmentation
Opinions/perceptions to share
Do you agree with the following assertion ?

1. Define 2. Count
3. Contact details
- "ABC respects environment” - …
Surveys, research media analysis, observations  
Yes
Indicators
Yes : No : 
Don’t know :
x 1
[File qualification]
-- "ABC respects environment”

x N
[Sample survey]
 
60%
Current
score:
60 %
Targeted score : 90 %
Management
Allocate
Understand
Adjust
Maintain
Etc…
Gap between results and objectives OK ! !!
Occurrence - Workshop A7 - 10/12/2013

11. Advice
Efficiency means producing the desired effects on the desired target audiences
Therefore, you need to define the target audiences that you want to reach with which effect, prior to the initiative, and ideally to define the performance threshold
As a starting point: KISS (Keep It Simple, Stupid)
Keep things simple at the beginning
BUT assess them regularly
And don't change the assessment system for each evaluation
Do not restrict the evaluation to the activity, in order not to limit communications to initiatives and tools
Design SMART (Specific, Measurable, Achievable, Realistic, and Timed) tools and goals for each initiative
Share the results and the decisions they help to take in order to enhance the value added of evaluation
La Communication peut et doit s’évaluer
La seule contrainte est de définir des objectifs clairs et partagés en amont
L’évaluation nécessite un minimum d’effort : 5% mesure
L’évaluation avance sur deux pieds :
Les indicateurs d’activité (dont les ressources) moins coûteux à produire
Les indicateurs d’efficacité (dont les audiences) plus coûteux à produire
L’évaluation permet aux communicants de
Crédibiliser / légitimer / valoriser la fonction communication
Démontrer son rôle et sa création de valeur pour l’entreprise
Piloter de manière plus rationnelle les investissements (arbitrage éclairé)
Gagner du temps et du budget en concentrant ses moyens sur les actions les plus efficaces 11
Occurrence - Workshop A7 - 10/12/2013 11

12. Thank you Contact : Céline Mas Partner & Research Director Occurrence
Occurrence - Workshop A7 - 10/12/2013

13. PPT

15. Ben Ward
Head of Evaluation and Research, Coffey International Development, UK

16. Instructions for use Evaluating communication
This template has been built with custom styles and formats aligned to the new brand identity.
All charts, org charts, tables and smart graphics have the custom colour palette and font styles embedded. Choose the style that best suits the presentation of your information.
Inserting charts as images and including photography will increase the file size – include lower resolution files.
The Coffey orange should only be used in headers, sub headers and bullets – avoid using it in charts, tables and graphs.
Secondary colours are only used in charts, tables and graphs – avoid using as headers, sub headers and bullets.
Bolder tones are best used for charts – as they appear clearer on screen.
Please do not change the primary colour set ups.  
Headlines on pages: if they are one line in length please ensure they are presented in black.  If the headline/title splits across two lines – please retain the first line in black and put the second line in the Coffey Orange (select from custom colour palette).
Evaluating communication
Ben Ward December 9th 2013
IDEV Executive Workshop
4 November 2013

17. Evaluating communication activities
Contents
Who we are and what do we do?
Challenges faced by public sector communicators
Evaluation - hints and tips
Discussion
Telling the story conference
December 2013

18. Evaluating communication activities
Coffey Evaluation and Research
Example I&C assignments
Founded in 1990, acquired by Coffey International Development in 2009
Team of 30 international and multilingual M&E professionals
Work in EU, UK domestic, and international development evaluation markets
Evaluation on EU Regional Policy communication (DG REGIO)
Ex ante evaluation of campaign on violence against women (DG JUSTICE)
Evaluation of ESF communication activities (DG EMPL)
Evaluation of DG TRADE’s communication strategy
Ex-ante, Mid-Term and Ex-post
Telling the story conference
December 2013

19. Evaluating communication activities - Challenges
The communication challenge in the public sector is greater than for business communicators
(Adapted: Liu and Horsley, 2007)
Public sector comms
Greater resource constraints
Legal constraints
Political agendas
Extreme media scrutiny
Telling the story conference
December 2013

20. Evaluating communication activities - Challenges
Telling the story conference
December 2013

21. Evaluating communication activities – Hints and tips
Take an evidence based approach to communications design
Design of any communications campaign should be driven by good evidence
The evidence base:
Primary and secondary
Quantitative and qualitative
Research areas: - Build deep knowledge around what you are communicating – e.g. the problem itself - Defining target audience(s) - Appropriate communication tools and activities
Research primary and secondary: doesn’t have to be expensive Research areas: the issue itself – what are the facts?
Telling the story conference
December 2013

22. Evaluating communication activities – Hints and tips
Evaluation should start at the very beginning
Set the objective(s): What are you trying to achieve?
Develop an intervention logic and SMART indicators
Mapping how you will get to your desired outcomes (use an evidence base to inform this)
Workshop - communications team and other relevant stakeholders
“He who fails to plan is planning to fail”.
Communicators say to the product development team to the policy team…..comms need to be involved from the beginning
Evaluators say to communicators ….evaluation needs to be considered from the beginning
Telling the story conference
December 2013

23. Evaluating communication activities – Hints and tips
Effective evaluation: Focus on measuring the outcomes
Measure as much as you can with the resources available
Outputs and outtakes both have their place (So what?)
However, achieving outcomes ultimately help to justify the use of communications
Producing a lot of material, such as leaflets and posters, and getting out the right messages is no good if you fail to get the final result!!
“Measurement isn’t about simply monitoring outputs (i.e. the quantity of media coverage achieved), but instead effective measurement is about assessing outcomes (i.e what attitude shifts have occurred)
Outputs – just a case of counting the number of outputs which are the result of some form of communications activity. SO WHAT?
Out-takes – an out-take is something the public or your target group will take away as a result of a particular piece of communications activity. For example a key message, a perception of understanding of a concept or issue.
Outcomes – outcomes are quantifiable changes in attitudes, behaviours or opinions. With a campaign aimed to encourage greater recycling, an out-take could be an individual taking on board the messages, and able to recall hearing or seeing the message if asked. However, an outcome would be that the person has actually changed their recycling habits for the better as a result of the campaign. Fundamental to behaviour change is being able to use insight to understand your audience and the behaviours you are seeking to influence
RBM concept! In management and comms!
Telling the story conference
December 2013

24. Evaluating communication activities – Hints and tips
A few final tips……..
Recognise what you can and you cant do with your evaluation (but also with your comms!)
Take a proportionate approach to evaluation
Size of intervention
Repeatability / Scalability factors
Use mixed methods (Quant and Qual)
Limited resources (Time and money) : Recognise what you can and you cant do
Don’t promise the world!
Put yourself in a position to walk the talk! Give yourself a chance!
The what but also the how and the why?
Quant data showing crime in NY on decline not really understanding why?
Telling the story conference
December 2013

25. Evaluating communication activities
Thank you
Telling the story conference
December 2013

26. PPT