1. Testing Ads Adapted from J. Scott Armstrong from lectures related to Persuasive Advertising. File: TestingAds-R16 In Lectures for AdPrin/Testing Ads Updated January 2015 1

2. Learning Diary The lectures follow an experiential learning experience. To make this work properly: 1.Obtain a learning diary (paper). A 10 x 13 bound diary is suggested. 2.Keep it up to date. 3.Take the learning diary with you to all class sessions. 4.For self-learners, use the diary to track your learning progress for all of your learning activities. 2 Adprin.com

3. Objectives of this session To understand and apply these principles and techniques (not to convince you). Ask for clarification as needed. Set a goal for yourself on how many principles and techniques you plan to use by the end of this session. Even a goal of one will help you. Put this in your learning diary now. Note: We will discuss only some of the slides. When you go through the lecture on your own, view it in “Slide Show” and follow the experiential procedures. Adprin.com

4. Procedure Focus on understanding. Record questions in your learning diary that will help you to apply the techniques or principles, then, after you decide which ones you want to apply, try to answer these from the readings. If not clear, ask others for help. 4 Adprin.com

5. The most important word in the vocabulary of advertising is test Test your premise Test your headlines and illustrations Test your media Test your level of expenditure “Never stop testing and your advertising will never stop improving.” David Ogilvy Adprin.com

6. How would you test this headline for Tylenol? “When not playing is not an option.” 6 Adprin.com

7. Testing the effectiveness of advertisements Assume that you were given 10 pairs of print ads and asked to identify which ad in each pair would be more persuasive. Each ad advertises a product with the same brand. In addition, you would like to suggest how to improve each of the ads. You have a budget of $20,000. Write different approaches in your learning diary. Spend at least five minutes if you already know how to solve this problem, and ten minutes if you do not know. If working in a group, exchange your solutions with a partner. When you finish, go to the next slide. 7 Adprin.com

8. Five approaches to testing advertisements: Listed by Effectiveness All of these methods involve comparisons of responses for two or more alternative advertisements: 1.Unaided ratings by novices (used for Super Bowl). 2.Unaided ratings by experts (used for Super Bowl). 3.Copy testing: Test which ads gain the best response. 4.Persuasion Principles Audit: Rate how well ad applies evidence- based persuasion principles. This is analogous to how we evaluate medical doctors and engineers. 5.Field experiment: Test direct-response ads to see which are most effective (e.g., as done by Bose radio ads). Reference: Armstrong et al (2014)Armstrong et al (2014) 8 Adprin.com

9. Benefits of unaided judgment by non-experts* 1.Very low cost 2.Very fast 3.Combining helps to improve validity 9 Adprin.com

10. Unaided judgments by experts Experts are typically poor at forecasting effects of different approaches in complex uncertain situations. Tetlock Seer-suckerSeer-suckerTheory Experts in consumer behavior were poorer than marketing practitioners and high-school students in predicting consumer behavior (Armstrong 1991).(Armstrong 1991) 10 Adprin.com

11. Persuasion Principles Audit Freeware and training module available at AdPrin.com Freeware Requires some knowledge of the principles (See AdPrin.com & Persuasive Advertising)Persuasive Advertising Requires good cognitive skills, logic, and patience Creativity helps Practice helps People differ in their ability to do this task. I suspect that economists, lawyers, and engineers do well. The audit leads to a Persuasion Principles Index (PPI)audit 11 Adprin.com

12. AdPrin Audit procedure Complete 90-minute AdPrin.com training moduletraining module Rate print ads in about 30 minutes, motion ads in 40. Use five independent coders, then combine by using the mode When ratings differ “substantially” for “relevant principles” obtain a new rater – for details see Predictive Validity of Evidence-based AdvertisingPredictive Validity of Evidence-based Advertising 12 Adprin.com

13. Benefits of the AdPrin Audit 1.Relatively low cost 2.Relatively fast 3.Leads to evidence-based improvements 13 Adprin.com

14. Copy testing: Selecting a representative sample 1.Who to test? A convenience sample Mall intercepts or student subjects, or Mechanical Turk at Amazon. Screening questions for similarity to the target market 2. How large a sample? Start small, increase as the client desires or as you think proper. 3. What approach? Experimental design, with random assignment of subjects to two or more treatments. 14 Adprin.com

15. Copy testing for advertisements 1.Compare responses to ads. 2.Ads do not have to be in finished form.  Rough copy generally produces realistic results on the relative magnitude of effects. 3.High face validity  convincing to clients. 4.Are ads appropriate for all groups? -Consider testing among members of interest groups. -To maintain security, ask subjects to role-play such interest groups. 15 Adprin.com

16. Testing procedure 1.Prepare Test Package – Degree of finish of the ads (“finished” not critical) – Context (e.g., in a magazine) – Number and length of exposures 2.Design of the test (e.g., pre- and post- tests; within subjects v. across subjects). 3.Use multiple measures of effect Adprin.com

17. Testing responses 1.Describe the subjects’ role (e.g., “Assume you are planning to buy a car”) 2.Focus on behavioral change (e.g., purchase intentions, not “liking”) 3.Use a scale for responses when possible (e.g., use “0 to 100%” for intentions) 4.Use alternative ways of asking the questions 5.Pretest the questions to ensure understanding 6.Test elements of the ads (e.g., picture alone versus headline alone) 17 Adprin.com

18. 18 Test the headline Adprin.com

19. 19 Test the illustration: What is the product?

20. 20 Complete ad

21. Administering copy tests 1.Written testing instruments and instructions. 2.Self-administration of tests by individuals. 3.Those administering the instrument should be unaware of the hypotheses if they interact with the subjects. 4.Prepare written instructions for administrators. 5.Take notes about departures from the script. 21 Adprin.com

22. What not to do for copy testing 1.Do not ask customers how they make decisions when purchasing products. They do not know even when making important decisions. (See meta-analysis by Nisbett & Wilson 1977). Even if they did know, they would not remember (Kaasa, Morris & Loftus 2011).Nisbett & Wilson 1977Kaasa, Morris & Loftus 2011 2.Consumers are not experts at advertising, so do not ask them how an ad might be improved or if they liked an ad. 3.Do not use focus groups.focus groups 22 Adprin.com

23. Review: Benefits of copy testing 1.Useful for comparing responses to ads. 2.The ads do not have to be in finished form. Rough copy generally produces realistic results on the relative magnitude of the ads’ effects. 3.Copy testing has high face validity and is thus convincing to clients. 4.Testing can also assess whether the ads are appropriate to all interest groups. 23 Adprin.com

24. Experimental Designs (Campbell and Stanley) “BEFORE” MEASURES TREATMENT “AFTER” MEASURES “AFTER ONLY” “BEFORE-AFTER” “BEFORE-AFTER WITH CONTROL” EXP. CONTROL “FOUR GROUP- SIX STUDY EXP I EXP II CONTROL I CONTROL II

25. After Only Design Use time series Prepare (obtain) predictions with ad campaign Compare results with prediction P A Provide Contingency Plans Adprin.com

26. Laboratory or field experiments? 1.Laboratory experiments are typically less expensive and offer more control. 2.Field experiments are more realistic. 3.Findings from each method are comparable. For a brief summary, see Persuasive Advertising, pp.7-13.Persuasive Advertising See Locke’s Generalizing from Laboratory to Field SettingsGeneralizing from Laboratory to Field Settings 26 Adprin.com

27. What constitutes “evidence” Reliability “Repeat the test on a similar population to determine if results are similar.” Replication: Other researchers get same result by same procedure. In Of 20 replications marketing journals, 12 gave substantially different results and only 3 were fully in agreement. Hubbard & Armstrong 1994 Hubbard & Armstrong 1994 Inter-rater (different raters agree) Across time (same rater) Validity “Test measures what it purports to measure.” Face validity (experts agree); Very weak test. Try asking them to predict the outcomes before seeing outcomes. Then pay attention to the differences. Construct validity (different approaches) For example, Apple’s 1984 ad, was not liked by the experts and it did poorly on copy testing, but it did well on implementing principles. Predictive validity. Are the predictions testable? (For example, is a cold winter evidence for or against global warming? What will the temperature be in one hundred years?) Adprin.com

28. What would you do in this asbestos case? 492 X-rays submitted as court evidence in asbestos lawsuits. X-ray readers found lung damage in 96% of the cases. You have been hired as an expert witness for all cases. What would you do? Spend five minutes to answer this in your learning diary. If working in a group, share your ideas with a learning partner. When finished, go to the next slide. 28 Adprin.com

29. Resolving the asbestos case Replicate: Ask independent experts to read the x-rays. Result? Only 4.5% of the X-rays were read as showing damage. (See Gitlin study)Gitlin The plaintiffs will claim that you are biased. How would you protect against that? Click after you have written your answer in your learning diary. Take steps to ensure that the independent experts do not know why they are being asked to do the readings. 29 Adprin.com

30. Reporting on your research 1.Bring in research as needed to support recommendations. 2.Use statistics to communicate information (See The Visual Display of Quantitative Information by E. Tufte). 3.Do not use statistical significance. Use prediction intervals instead.Do not use statistical significance 4.Report on shortcomings of your research. 5.Provide full disclosure on data and methods. See checklist for oral and written reportsoralwritten 30 Adprin.com

31. Applications: Tylenol ad Apply material from this lecture to test this Tylenol headline. “When not playing is not an option.” If working in a group setting, exchange your applications with a learning partner. Then click for some answers Adprin.com

32. Tylenol headline It did poorly on copy tests. Also does poorly on principles. For example: 1. No brand name 2. Two negative words 3. Contains a period 4. Social responsibility and legal issues* * For example, they might get sued if an unfortunate event occurs. 32 Adprin.com

33. Ideas for Applications of Techniques for Testing Ads If you are in a class, test your advertisements for the “house ad” by using the techniques discussed in this lecture.house ad If you are a self-learner apply the testing techniques to advertisements in you organization, or to a cause that is of interest to you. 33 Adprin.com

34. Techniques In your diary, describe the techniques that you were able to use to test your advertisement and rate your success (e.g., creativity, objective setting)rate your success 34 Adprin.com

35. Advice on learning techniques One study found that fewer than 10% of students were successful in applying new knowledge. This went to 20% if they actively applied what they were taught during a class session. It went to 90% when they worked with a learning partner and coached each other. Select techniques to applytechniques 35 Adprin.com