1. Testing Ads Note: Slides with red titles are used in the formal class lectures
Adapted from J. Scott Armstrong from
lectures related to Persuasive Advertising.
File: TestingAds-R17
In Lectures for AdPrin/Testing Ads
Updated March 23, 2016

2. Learning Diary
The lectures follow an experiential learning experience. To make this work properly:
Obtain a learning diary (paper). A 10 x 13 bound diary is suggested.
Keep it up to date.
Take the learning diary with you to all class sessions.
For self-learners, use the diary to track your learning progress for all of your learning activities.
Adprin.com

3. Objectives of this session
To understand and apply these principles and techniques (not to convince you). Ask for clarification as needed.
Set a goal for yourself on how many principles and techniques you plan to use by the end of this session. Even a goal of one will help you. Put this in your learning diary now.
Note: We will discuss only some of the slides. When you go through the lecture  on your own, view it in “Slide Show” and follow the experiential procedures.
It is typically difficult (and a bit upsetting) to students when they are asked to set objectives, so I go easy here.
Adprin.com

4. Procedure
Focus on understanding. Record questions in your learning diary that will help you to apply the techniques or principles, then, after you decide which ones you want to apply, try to answer these from the readings. If not clear, ask others for help.
Adprin.com

5. The most important word in the vocabulary of advertising is test (David Ogilvy)
Test your premise
Test your headlines and illustrations
Test your media
Test your level of expenditure
“Never stop testing and your advertising will never stop improving.” David Ogilvy
Adprin.com

6. How would you test this headline for Tylenol?
“When not playing is not an option.”
Adprin.com

7. Testing the effectiveness of advertisements
Assume that you were given 10 pairs of print ads and asked to identify which ad in each pair would be more persuasive. Each ad advertises a product with the same brand. In addition, you would like to suggest how to improve each of the ads. You have a budget of $20,000. Write different approaches in your learning diary. Spend at least five minutes if you already know how to solve this problem, and ten minutes if you do not know. If working in a group, exchange your solutions with a partner. When you finish, go to the next slide.
Assume that you were given 10 pairs of print ads and were asked to identify which ad in each pair would be more persuasive. Each ad advertises the same brand of a product. In addition, you would like to suggest how to improve each of the ads.
You have a budget of $20,000. Write different approaches. Spend at least five minutes if you already know how to solve this problem, and ten minutes if you do not know. If working in a group, exchange your solutions with a partner.
When you finish, go to the next slide.
Adprin.com

8. Five approaches to testing advertisements: Listed by Effectiveness
All of these methods involve comparisons of responses for two or more alternative advertisements:
Unaided ratings by novices (used for Super Bowl).
Unaided ratings by experts (used for Super Bowl).
Copy testing: Test which ads gain the best response.
Persuasion Principles Audit: Rate how well ad applies evidence-based persuasion principles. This is analogous to how we evaluate medical doctors and engineers.
Field experiment: Test direct-response ads to see which are most effective (e.g., as done by Bose radio ads).
Reference: Armstrong et al (2014)
All of these methods involve comparisons of responses for two or more alternative advertisements:
Independent unaided ratings by experts (at least 5, but better with 20). (This is the least valid approach.)
AdPrin Audit: Rate how well each ad applies evidence-based persuasion principles.
Copy testing: Which ads gain the best response.
Field experiment: Use alternative versions of direct-response ads and see which are most effective. (e.g., as done by Bose radios)
By using more approaches, one can obtain more valid predictions.
Adprin.com

9. Benefits of unaided judgment by non-experts*
Low cost
Fast
Combining independent ratings improves predictive validity
Adprin.com

10. Unaided judgments by experts
Experts are typically poor at forecasting effects of different approaches in complex uncertain situations. Tetlock Seer-sucker Theory Experts in consumer behavior were poorer than marketing practitioners and high-school students in predicting consumer behavior (Armstrong 1991).
Advertising experts typically poor at forecasting effects in complex uncertain situations.
Tetlock [This link opens to a word document that is still tracking changes. It would be good to make sure that It is in its final version]
Seer-sucker
Experts in consumer behavior were poorer than marketing practitioners and high-school students in predicting consumer behavior (Armstrong 1991).
Adprin.com

11. Persuasion Principles Audit
Freeware and training module available at AdPrin.com
Requires some knowledge of the principles (See AdPrin.com & Persuasive Advertising)
Requires good cognitive skills, logic, and patience
Creativity helps
Practice helps
People differ in their ability to do this task. I suspect that economists, lawyers, and engineers do well.
The audit leads to a Persuasion Principles Index (PPI)
Freeware is available at AdPrin.com along with a training module
Requires some knowledge of the principles, which are explained on AdPrin.com. More detail in Persuasive Advertising. [MAYBE PLACE HYPERLINK TO BOOK?]
Requires analysts who have good cognitive skills and who are logical and patient.
It also helps if the analysts are creative.
Adprin.com

12. Calculating the Persuasion Principles Index
Complete 90-minute AdPrin.com training module
Rate print ads in about 30 minutes, motion ads in 40.
Use five independent coders, then combine by using the mode
When ratings differ “substantially” for “relevant principles” obtain a new rater
for details see Predictive Validity of Evidence-based Advertising
Trained (or self-trained) non-experts (or experts) can rate print ads in about half an hour (sometimes longer for motion ads) after taking the training module (about two hours) at AdPrin.com and working on a few ads.
Use five independent coders.
Seek consensus for “relevant principles” on which ratings differ “substantially.” This involves the administrator sending those principles back to the raters for a second rating. (For details see Predictive Validity of Evidence-based Advertising) [HYPERLINK??]
Adprin.com

13. Benefits of the AdPrin Audit
Relatively low cost
Relatively fast
Leads to evidence-based improvements
It provides the most accurate predictions of effectiveness
Adprin.com

14. Copy testing: Selecting a representative sample
Who to test? • A convenience sample • Mall intercepts or student subjects, or Mechanical Turk at Amazon.
• Screening questions for similarity to the target market
2. How large a sample? Start small, increase as the client desires or as you think proper.
3. What approach? Experimental design, with random assignment of subjects to two or more treatments.
The next slide expands on testing the elements.
Who to test? Use convenience sample of people similar to the target market (mall studies and student subjects usually used). Do this by using screening questions.
How large a sample? Start with a small sample and increase as the client desires.
Experimental design: “Random assignment” of subjects to two or more treatments.
Adprin.com

15. Copy testing for advertisements
Compare responses to ads.
Ads do not have to be in finished form.
Rough copy generally produces realistic results on the relative magnitude of effects.
High face validity  convincing to clients.
Are ads appropriate for all groups?
Consider testing among members of interest groups.
To maintain security, ask subjects to role-play such interest groups.
1. Copy testing is useful for evaluating the comparative responses of two or more ads.
2. The ads do not have to be in finished form. Rough copy generally produces realistic results on the relative magnitude of effects of the ads.
3. Copy testing has high face validity, and thus is convincing to clients.
4. Testing can also assess whether the ads are appropriate to all interest groups. (Consider testing among members of interest groups, or, to maintain security, ask subjects to role-play such interest groups.)
Adprin.com

16. Testing procedure Prepare Test Package
Degree of finish of the ads (“finished” not critical)
Context (e.g., in a magazine)
Number and length of exposures
Design of the test (e.g., pre- and post- tests; within subjects v. across subjects).
Use multiple measures of effect
You should be aware of the proper procedures if you are a tester or a client
Adprin.com

17. Testing responses
Describe the subjects’ role (e.g., “Assume you are planning to buy a car”)
Focus on behavioral change (e.g., purchase intentions, not “liking”)
Use a scale for responses when possible (e.g., use “0 to 100%” for intentions)
Use alternative ways of asking the questions
Pretest the questions to ensure understanding
Test elements of the ads (e.g., picture alone versus headline alone)
Focus on behavioral change (e.g., purchase intentions, not “liking”)
Describe the role for the subjects (e.g., “Assume that you were planning to buy an automobile”
Use a scale for responses when possible (e.g., use “0 to 100%” for intentions)
Pretest questions to ensure understanding
Use alternative ways of asking the questions
Test elements of the ads (e.g., picture alone; headline alone).
Adprin.com

18. Test the headline
Adprin.com

19. Test the illustration: What is the product?

20. Complete ad

21. Administering copy tests
Written testing instruments and instructions.
Self-administration of tests by individuals.
Those administering the instrument should be unaware of the hypotheses if they interact with the subjects.
Prepare written instructions for administrators.
Take notes about departures from the script.
Testing instruments and all instructions must be written.
Tests should be self-administered by individuals.
If those administering the instrument interact with the subjects, they should not know the hypotheses.
Prepare written instructions for administration.
Take notes about departures from the script.
Adprin.com

22. What not to do for copy testing
Do not ask customers how they make decisions when purchasing products. They do not know even when making important decisions. (See meta-analysis by Nisbett & Wilson 1977). Even if they did know, they would not remember (Kaasa, Morris & Loftus 2011).
Consumers are not experts at advertising, so do not ask them how an ad might be improved or if they liked an ad.
Do not use focus groups.
Do not ask customers how they make decisions when purchasing products. They do not know, even when making important decisions. (See meta-analysis by Nisbett & Wilson 1977). And even if they did, they would not remember (Kaasa, Morris & Loftus 2011).
Consumers are not experts at advertising, so do not ask them how an ad might be improved, and do not ask them if they liked an ad.
Do not use focus groups.
Adprin.com

23. Review: Benefits of copy testing
Useful for comparing responses to ads.
The ads do not have to be in finished form. Rough copy generally produces realistic results on the relative magnitude of the ads’ effects.
Copy testing has high face validity and is thus convincing to clients.
Testing can also assess whether the ads are appropriate to all interest groups.
1. Copy testing is useful for evaluating the comparative responses of two or more ads.
2. The ads do not have to be in finished form. Rough copy generally produces realistic results on the relative magnitude of effects of the ads.
3. Copy testing has high face validity, and thus is convincing to clients.
4. Testing can also assess whether the ads are appropriate to various interest groups.
Adprin.com

24. Experimental Designs (Campbell and Stanley)
“BEFORE”
MEASURES
TREATMENT
“AFTER” MEASURES
“AFTER ONLY”
“BEFORE-AFTER”
“BEFORE-AFTER WITH CONTROL”
EXP.
CONTROL
“FOUR GROUP-
SIX STUDY
EXP I
EXP II
CONTROL I
CONTROL II

25. Provide Contingency Plans
After Only Design
Use time series
Prepare (obtain) predictions with ad campaign
Compare results with prediction P A
Provide Contingency Plans
Adprin.com

26. Laboratory or field experiments?
Laboratory experiments are typically less expensive and offer more control.
Field experiments are more realistic.
Findings from each method are comparable.
For a brief summary, see Persuasive Advertising, pp.7-13.
See Locke’s Generalizing from Laboratory to Field Settings
Laboratory experiments are typically less expensive and they offer more control.
Field experiments are more realistic.
Findings from each method are comparable.
See Locke’s Generalizing from Laboratory to Field Settings
Adprin.com

27. What would you do in this asbestos case?
492 X-rays submitted as court evidence in asbestos lawsuits. X-ray readers found lung damage in 96% of the cases. You have been hired as an expert witness for all cases. What would you do? Spend five minutes to answer this in your learning diary. If working in a group, share your ideas with a learning partner. When finished, go to the next slide.
Adprin.com

28. Resolving the asbestos case
Replicate: Ask independent experts to read the x-rays.
Result?
Only 4.5% of the X-rays were read as showing damage. (See Gitlin study)
The plaintiffs will claim that you are biased. How would you protect against that?
Click after you have written your answer in your learning diary.
Take steps to ensure that the independent experts do not know why they are being asked to do the readings.
Adprin.com

29. Reporting on your research
Bring in research as needed to support recommendations.
Use statistics to communicate information (See The Visual Display of Quantitative Information by E. Tufte).
Do not use statistical significance. Use prediction intervals instead.
Report on shortcomings of your research.
Provide full disclosure on data and methods.
See checklist for oral and written reports
Use research to support recommendations (bring in as needed)
Purpose of statistics is to communicate information (Tufte, E., The Visual Display of Quantitative Information)
Do not use statistical significance. Use confidence intervals instead.
Report on shortcomings of your research.
Provide full disclosure on data and methods.
See checklist for oral and written reports
Adprin.com

30. Applications: Tylenol ad
Apply material from this lecture to test this Tylenol headline. “When not playing is not an option.” If working in a group setting, exchange your applications with a learning partner. Then click for some answers
Johnson & Johnson did copy testing
Adprin.com

31. Tylenol headline
It did poorly on copy tests. Also does poorly on principles. For example: 1. No brand name 2. Two negative words 3. Contains a period 4. Social responsibility and legal issues* * For example, they might get sued if an unfortunate event occurs.
Adprin.com

32. Ideas for Applications of Techniques for Testing Ads
If you are in a class, test your advertisements for the “house ad” by using the techniques discussed in this lecture.
If you are a self-learner apply the testing techniques to advertisements in you organization, or to a cause that is of interest to you.
The study on checklists was done by Haynes et al.(2009). See short summary in PA, P 290: Would you want a surgical team to operate on you if they had no checklist? For example, a 2008 study evaluated the effects of the implementation of a 19-item checklist for hospital procedures. A before/after experimental design was used for thousands of patients in eight hospitals in eight cities around the world. In the month after the operations, the checklist led to a reduction in deaths from 1.5% to 0.8%, and in complications, from 11% to 7% (Haynes et al. 2009).
Adprin.com

33. Techniques
In your diary, describe the techniques that you were able to use to test your advertisement and rate your success (e.g., creativity, objective setting)
Adprin.com

34. Advice on learning techniques
One study found that fewer than 10% of students were successful in applying new knowledge.
This went to 20% if they actively applied what they were taught during a class session.
It went to 90% when they worked with a learning partner and coached each other.
Select techniques to apply
Adprin.com