1. Evaluation of Effectiveness
Advertising Principles
and Practices
Evaluation of Effectiveness

2. Questions We’ll Answer
How well do you understand why and how advertising evaluation is conducted?
Can you list and explain the stages of message evaluation?
What are the key areas of media evaluation?
How are campaigns and IMC programs evaluated?

3. Outstanding in the Field
New Holland had to reinvent and reintroduce its tractor brand with sliding market share, a shrinking market, and limited budget.
The campaign positioned New Holland as the better choice of an empowered consumer, and drove traffic to its Web site.
Sales grew by 9%, market share by 21% and 36% for low and high horsepower categories, Web site visits were 68,000 over the campaign period. \
Visit the Site
Prentice Hall, © 2009
19-3

4. Impact: How does it work?
Many executive feel advertising is only successful if it produces sales.
Others feel advertising should emphasize long-term brand building.
If advertising delivers the desired communication effects, but sales don’t increase, was the advertising ineffective?
Video Snippet
AFLAC’s initial campaign created 89% brand recognition.

5. Evaluating Effectiveness
Intuitive analysis is based on an experienced managers judgment.
Measurement tracks consumer responses with structured feedback like response cards and calls.
Formal evaluation is necessary:
Financial stakes are high—production of :30 spot averages $200,000; national media costs several million.
Advertising optimization—reducing the risk failure through testing, analyzing, tracking performance, and making changes to increase performance.
Identify “best practices”—what works and what doesn’t, so brand advertising continues to improve.

6. Types of Evaluation Testing—to predict results
Sample ads are tested before they run.
Monitoring—to track performance
Performance is tracked to see if anything needs to be changed.
Measurement—to evaluate the results
The results, or actual effects, are measured after the campaign runs.

7. Stages of Evaluation Developmental research Concurrent research
Pretesting to see if an idea will work, or another is better.
Concurrent research
Tracking studies and test marketing to see how campaign is unfolding and how messages and media are working.
Posttesting research
Comparing the impact of campaign after it’s over against a benchmark, baseline, or other starting point.
Diagnostic research
Taking apart an ad to see what elements are working and which aren’t; examine frame by frame or piece by piece.

8. Facets: Measuring Responses
It’s difficult to measure advertising’s effect on sales:
Other factors affect sales (e.g., pricing, distribution, competition), making it hard to isolate impact.
Effects are delayed; it’s hard to link sales to advertising.
Communication effects an be measured as surrogate measures for sales impact:
Awareness of the advertising, purchase intention, preference, liking.
Principle:
Good evaluation plans, as well as effective promotional work, are guided by a model of how people respond to advertising.

9. Table 19:1 Effectiveness Research Questions
Perception
Awareness/Noticed
Attention
Recognition
(Aided )
Relevance
Emotion
Liking/Disliking
Desire
Cognition
Interest
Comprehension/
Confusion
Recall (Unaided)
Brand Recall/
Linkage
Differentiation
Research Questions
What ads do you remember seeing?
Which ads were noted?
What caught your attention?
Did the ad stand out among the other ads and content around it?
What stood out in the ad?
Have you seen this ad/this campaign?
Sort elements into piles of remember/don’t remember.
How important is the product message to you? Does it speak to your interest and aspirations?
What emotions did the ad stimulate?
How did it make you feel?
Do you like this brand? This story? The characters (and other ad elements)?
What did you like or dislike about the brand? The ad?
Do you want this product or brand?
Did you read/watch most of it? How much?
Did it engage your interest or curiosity?
Where did your interest shift away from the ad?
What thoughts came to you? Do you understand how it works? Is there anything in the ad you don’t understand? Do the claims/product attributes/benefits make sense? Do you have a need for this brand or can it fulfill a need for you?
What happened in the commercial? What is the main message? What is the point of the ad?
What brand is being advertised in the ad? (In open-ended responses, was the brand named?)
What’s the difference between Brand X and Y?

10. Table 19:1 Effectiveness Research Questions
Persuasion
Attitude
Preference
Intention
Argument/Counter
Argument
Believability/Conviction
Trust
Association
Personality/Image
Self Identification
Research Questions
Do you have a favorable or unfavorable opinion of the brand? The ad?
How excellent or weak is the brand? The ad?
Do you respect it?
In Category X (or product set), which brand would you choose? (Usually a pre- or posttest question).
What brand do you prefer?
Do you want to try or buy this product/brand?
Would you put it on your shopping list?
What are your reasons for buying it? Or for not buying it—or its competing brand(s)? How does it compare to competitor’s brands?
Do you argue back to the ad?
Do you believe the reasons, claims, or proof statements?
Are you convinced the message is true? The brand is best?
Do you have confidence in the brand?
When you think of this brand, what (products, qualities, attributes, people, lifestyles, etc.) do you connect with it?
Do you link this brand to positive experiences? What is the personality of the brand? Of whom does it remind you? Do you like this person/brand personality?
What is the brand image? What does it symbolize or stand for?
Can you see yourself or your friends using this brand? Do you connect personally with the brand image?

11. Message Evaluation Copy Testing Message Development Research
During Execution: Concurrent Testing
Posttesting: After Execution Research

12. Copy Testing
Companies that conduct research and perform diagnostic methods to identify an ad’s strong and weak points:
Ameritest: brand linkage, attention, motivation, communication, flow of attention and emotion through the commercial
ARS: persuasion, brand/ad recall, communication
Diagnostic Research: brand recall, main idea, attribute statements (importance, uniqueness, believability)
IPSOS-ASI: recall, attention, brand linkage, persuasion, (brand switch, purchase probability), communication
Mapes & Ross: brand preference change, ad/brand recall, idea communication, key message delivery, like/dislike, believability, comprehension, desire to take action, attribute communication
Millward Brown: branding, enjoyment, involvement, understanding, ad flow, brand integration, feelings to ad, main stand-out idea, likes/dislikes, impressions, persuasion, new news, believability, relevance
RoperASW: overall reaction, strengths and weaknesses, understanding, clutter-busting, attention, main message, relevance, appeal, persuasiveness, motivate trial, purchase intent

13. Ameritest
Ameritest is an international advertising research firm that studies thousands of ads for some of the world’s largest advertisers to discern which ones work, which ones don’t, and which are close—and why.
Visit the Site

14. Message Development Research
Concept Testing
Compares the effectiveness of various message strategies and their creative ideas (the big idea).
Pre-testing
Helps marketers make final go/no-go decisions about finished/nearly finished ads using photoboards or animatics.
Diagnostics
Designed to diagnose strengths and weaknesses of ideas to improve work still in development or to learn more in order to improve subsequent advertisements.
Principle:
Advertising effects are too complicated to be reduced to a single score.

15. During Execution: Concurrent Testing
Coincidental Surveys
In broadcast media, random calls to target market determine stations choices, ads they’ve seen/heard, brand perceptions.
Tracking Studies
Every 3 to 6 months, measure top-of-mind brand awareness.
Brand tracking tracks the performance of the brand.
Test Markets
Evaluate product variations, campaign or media elements.
Generally, two or more markets with markets as controls.

16. Posttesting: After Execution Research
Breakthrough: Attention—interest, enjoyability, liking
Engagement Tests—eye-tracking as readers scan ads
Memory Tests—recognition test, recall tests, unaided recall, aided recall
Emotion Test—MRI measures brain activity
Likeability Tests—relevant, important, enjoyable, entertaining, fun
Persuasion Tests—intention to buy, motivation
Inquiry Tests—measures number of responses to an ad
Scanner Research—tally up purchase and collect consumer buying info
Single-Source Research—advertising and brand purchase data come from the same households, linking advertising to sales

17. Media Evaluation Evaluating Audience Exposure
How did each media vehicle perform? Were reach and frequency objectives met?
Services include Simmons-Scarborough, Arbitron, MediaMark.
For outdoor, traffic counts don’t equal exposure.
For Web or Internet advertising, what is measured and how does it compare to traditional media: hits, click-throughs, minutes spent?
Alternative or guerilla marketing is even more difficult to equate to traditional media.

18. Arbitron
Arbitron Inc. measures radio, TV, cable, online radio and out-of-home media audiences; surveys the retail, media and product patterns of local market consumers; and provides software for analyzing media audience and marketing information data.
Visit the Site

19. Media Evaluation Advertising ROI and Media Efficiency
Return on investment (cost to sales ratio) is hard to calculate because many factors affect sales.
How do you determine if you’re overadvertising or underadvertising?
Wearout—recall stabilizes or declines and irritation increases until there’s no or less response (can be a combo of creative impact and media buying).
Media optimization—the goal is optimum media performance getting the most impact for the investment.

20. Evaluating Marketing Communications Campaigns
Last, and perhaps most important, stage in the development of a campaign plan
Determines whether the campaign’s message and media were effective
Measures the overall impact on the brand, but the pieces are still evaluated to determine their individual effectiveness.

21. Marcom Tools
Certain marketing communication functions such as public relations and sales promotion, do some things better than other areas.
An integrated plan uses the best tools to accomplish the desired effect.
Principle:
Advertising is particularly effective in accomplishing such objectives as creating exposure, awareness, and brand image, and delivering brand reminders.

22. Table 19:2 Message Effectiveness Factors
Key Message Surrogate
Effects Measures Communication Tools
Perception Exposure Adv Media; PR, PoP
Attention Adv; Sales Promo, Packaging; PoP
Interest Adv; SP; PR, Direct; PoP
Relevance Adv; PR; Direct; PoP
Recognition Adv; PR, Pkging; PoP, Specialties
Cognitive Understanding Adv; PR; Sales; Direct
Recall Adv; SP; PR, PoP, Specialties
Adv; PR; Pkging
Affective Emotions and Liking Adv; Sales Promo, Pkging; PoP
Appeals Adv; PR; Sales; Events/Spnsrshps
Resonate Adv; PR; Events/Spnsrshps
Persuasion Attitudes Adv; PR; Direct Preference/Intention Adv; PR; Sales; Sales Promo
Credibility PR
Conviction PR; Sales; Direct
Motivation Adv; PR; Sales; Sales Promo
Brand Assoc Brand Image Adv; PR, Events/ Spnsrshps
Action Trial SP; Sales; Direct, PoP
Purchase SP; Sales; Direct
Repeat Purchase Adv; SP; Sales; Direct, Specialties

23. Direct Response
The objective is to generate an immediate behavior response (transaction, buy)
Use toll-free numbers, mail-in coupons, Web site or address, an offer in the copy
Response is easy to measure in terms of effectiveness and ROI.
Total responses divided by total mailed = Response per thousand (RPM)

24. Sales Promotion
May be necessary to evaluate both trade and consumer promotions
Payout analysis compares the costs of a promotion to the expected sales
Breakeven analysis—finds the point at which the total cost of the promotion exceeds the total revenues

25. A Sales Promotion Breakeven Analysis
At the breakeven point, where 30,000 premiums are delivered at a cost of $45,000, the sales revenues exactly cover, but do not exceed, total costs. Below and to the left of the breakeven point (in the portion of the diagram marked off by dashed lines) the promotion operates at a loss. Above and to the right of the breakeven point, as more premiums are sold and sales revenues climb, the promotion makes a profit.
Prentice Hall, © 2009
19-25

26. Public Relations
Measure the success in getting out the message in terms of output and outcomes
Output: materials produced and distributed; how many press releases ran
Input: acceptance and impact of materials; changes in public opinion
Content analysis: Was coverage favorable?
Public opinion studies: Have attitudes, behaviors, or knowledge changed?

27. Common Measures of Output and Outcomes in Public Relations
Output Objectives Achieved
Production. Number of PR messages, such as news releases or brochures, generated.
Distribution. Number of media outlets (TV stations, newspapers) receiving PR products.
Coverage. Number and size of clips, column inches, or minutes of time or space.
Impressions. Media placements multiplied by circulation or broadcast reach.
Advertising value. Equivalent ad costs for time or space.
Content Analysis. Positive or valence (whether the story or mention seems to be more positive or negative), key messages (the idea in the story), sources, and prominence.
Outcome Objectives Achieved
Awareness. Aided and unaided recall.
Attitudes. Perceptions, preferences, and intent to buy.
Behavior. Did target do what you wanted them to do?

28. Web Site Evaluation Traffic volume Click-through rates Cost per lead
Page views
Site visitors
Click-through rates
Ads sold as pay-per click
Cost per lead
An attempt to measure ROI using a conversion rate (percent of visitors who complete desired action)

29. Special Advertising Situations
Retail advertising
B2B advertising
International advertising
Objective: Generate store traffic
Simple counts of people at promotions and events
Objective: Visibility
Participation counts at events, or “how-to” classes
Sign-up and fill-out forms
Objective: Loyalty
Participation in frequency clubs or loyalty programs

30. Special Advertising Situations
Retail advertising
B2B advertising
International advertising
Objective: Generate response/sales leads
Lead count based on calls, s, and cards returned to the advertiser
Objective: Conversion rates—number of leads who make a purchase

31. Special Advertising Situations
Retail advertising
B2B advertising
International advertising
Difficult to evaluate because of the number of markets, distance, cost and variety of cultures.
Evaluation should focus initially on pretesting to help correct big problems (due to unfamiliarity with the culture, language or consumer behavior) before they occur.

32. Campaign Evaluation
It’s difficult to evaluate and estimate the impact of synergy.
Brand tracking can measure campaign effectiveness by adding and taking away ingredients, and studying the effects of those changes
The challenge: look at the big picture rather than individual pieces and parts.
Advertisers seek an evaluation method that brings all the individual metrics together to efficiently and effectively evaluate and predict communication effectiveness.

33. ARS Group
ARS Group is a marketing research firm that helps clients evaluate and optimize the effectiveness of their advertising messages within individual touchpoints and across integrated marketing campaigns.
Visit the Site

34. Discussion Questions

35. Discussion Question 1
Many creative people feel that formal copy testing research doesn’t do justice to their ideas.
In particular, they feel that research results are designed to reward cognitive approaches and don’t do a good job of evaluation for brand image ads and emotional ads.
From what you have read in this chapter about copy testing, why do they feel that way?
Do you believe that is a legitimate criticism of copy testing?

36. Discussion Question 2
Most clients want a quick and easy answer to the question of whether the ad works.
Advertising professionals, however, tend to believe that a one-score approach to copy testing is not appropriate.
Why would they feel that way?
If you were helping an agency prepare for a presentation on its copy testing results, what would you suggest the agency say to explain away the idea that you can evaluate an ad with a single test?

37. Discussion Question 3
Three-minute debate: You are hiring a research consulting company to help a client evaluate the effectiveness of its advertisements. One of the consultants recommends using focus groups to evaluate their effectiveness. Another consultant suggests that focus groups aren’t very effective for posttesting and recommends other measures. Which viewpoint do you believe is most insightful?
In class, organize into small teams with pairs of teams taking one side or the other. Set up a series of three-minute debates with each side having half of that time to argue its position. Every team of debaters has to present new points not covered in the previous teams’ presentations until there are no arguments left to present. Then, the class votes as a group on the winning point of view.

38. Copyright © 2009 Pearson Education, Inc. Publishing as Prentice Hall
All rights reserved. No part of this publication may be reproduced, stored in a retrieval system, or transmitted, in any form or by any means, electronic, mechanical, photocopying, recording, or otherwise, without the prior written permission of the publisher. Printed in the United States of America.
Copyright © 2009 Pearson Education, Inc.  
Publishing as Prentice Hall