1. Evaluation Adapted from J. Scott Armstrong Updated March 2016
This is a short lecture. I usually do it in combination with the Creativity lecture
Adapted from J. Scott Armstrong
Updated March 2016
Evaluation-R19

2. Learning Diary
The lectures follow an experiential learning experience. To make this work properly:
Obtain a learning diary (paper). A 10 x 13 bound diary is suggested.
Keep it up to date.
Use the diary for all.
For self-learners, use the diary to track your learning progress for all of your learning activities.
Adprin.com

3. Objectives of this session
To understand and apply these principles and techniques (not to convince you). Ask for clarification as needed.
Set a goal for yourself on how many principles and techniques you plan to use by the end of this session. Even a goal of one will help you. Put this in your learning diary now.
Note: We will discuss only some of the slides. When you go through the lecture  on your own, view it in “Slide Show” and follow the experiential procedures.
It is typically difficult (and a bit upsetting) to students when they are asked to set objectives, so I go easy here.
Adprin.com

4. Procedure
Focus on understanding. Record questions in your learning diary that will help you to apply the techniques or principles, then, after you decide which ones you want to apply, try to answer these from the readings. If not clear, ask others for help.
Adprin.com

5. Evaluation Topics How can you select the best ad?
How can you improve an ad?
Adprin.com

6. A Horse Named Sam Write your answer on a small piece of paper.
“A man bought a horse named Sam for $60 and sold him for $70. Then he bought him back again for $80. Finally, he sold him for $90. How much money did he make in the horse business?”
Write your answer on a small piece of paper.
Usually half the people get this correct. How can you improve upon this?
After individual analysis, form groups, say 3-5 people, and discuss your answers with a focus on understanding different viewpoints. Then seek consensus. This nearly always leads to the correct answer to this problem.
Ref: Maier & Solem (1952) Click for answer.
The answer is $20. Some people say $10, and $30 comes up also. But in a group someone nearly always has the right answer and discussion typically leads to correct solutions for groups. Nearly all groups get the correct answer.
Adprin.com

7. Parallel processing for analysis
Groups are not good at analysis.
Use parallel processing
two or more people do the same analysis– (e.g., structuring a problem, or doing math-- either prior to a meeting or in a “time-out.”)
The answer is $20. Group discussion typically leads to correct solutions for this problem.
Write another way to solve this problem. Then click for another approach.
That Deduct 10 points etc., was a quote from Russ Ackoff. I have seen no empirical evidence on this, but it seems about right to me.
Adprin.com

8. Devil’s Advocate (DA) approach
Group arbitrarily selects a member to find things that are wrong with a proposal.
The DA meets with the proposal team to describe what is wrong.
In 1587, the Roman Catholic Pope instituted the DA to evaluate if someone should be canonized.
Adprin.com 8 8

9. Do you see as the possible defects in the Devil’s Advocate?
Write your ideas in your learning diary.
Then click this slide for the evidence-based findings.
Adprin.com

10. Evidence on the value of DA
Devil’s Advocate not effective (review by Nemeth & Nemeth-Brown 2003).
Why? Presenters believe:
There is no hope of convincing someone who has been instructed to be negative (the DA).
Thus, they reinforce their current solution instead of using the criticisms for change.
Adprin.com

11. Alternatives to the Devil’s Advocate
Authentic dissent - people argue for what they believe.
The group realizes that an authentic dissenter might change, whereas there is no hope for this with the DA.
The role does not protect the people playing as
Devil’s Advocates – they became unpopular.
Authentic dissenters are also unpopular, but people
respect them for their courage.
Adprin.com 11 11

12. Authentic dissent
Those with authentic beliefs argue their positions better than do those assigned beliefs.
Authentic dissenters were able to get people to focus more on opposing thoughts than supporting ones - thus causing more change. (Experiments by Nemeth et al. 2001)
However, groups are annoyed by dissenters. They are not considered to be team players. Suggest a solution. Use problem storming and write your answers in you learning diary. Then click here.
Ask everyone to act as a dissenter– anonymously.
Adprin.com

13. Multiple Anonymous Authentic Dissent (MAAD)
Multiple Anonymous Authentic Dissent (MAAD) involves asking experts (e.g., all project group members) to act as dissenters.
1. Each person independently writes* all perceived defects in a proposal. Everyone sends them to an “assembler” who distributes a written report.
2. Each person assumes that each objection has merit, and writes ways to improve the proposals.
3. The suggested improvements are summarized in writing and appropriate revisions are made in the advertising proposal.
* This procedure cannot be done by simply thinking about the steps.
MAAD is my name for the process. There are a number of procedures in MAAD that have been independently validated.
Adprin.com 13 13

14. On the importance of “peer review”
To produce an effective proposal, you must do two things:
Start writing the report early – i.e., when you start a project. To do this, you must make a written promise on when you will start. Write one now. Then exchange with someone to get peer review.
Obtain independent peer review on the report as you proceed.
On a given report, you would need about ten good reviewers to find about 95% of the errors (Schroter et at 2008).
Scientific peer reviewers have been provided with paper that have intentional errors that are substantial. In one such study, on average each reviewer found less than 30%. Thus, it is important to have many people looking for errors, and doing so independently.
Schroter, et al “What errors do peer reviewers detect, and does training improve their ability to detect them?” J R Soc Med, 101: found the 30% chance of finding the errors in their experiment and other experiments have found similar results. The reviews were all done by experts in the field.
Adprin.com

15. Groups often rush to a decision. They then resist changes
Groups often rush to a decision. They then resist changes. How can you avoid this?
Second solution technique
Assume that you cannot use your currently favored solution.
Brainwrite alternatives for five minutes.
Relax the constraint and use either the first or second solution.
The second solution typically comes in less time than the first, and groups often think it is better.
The-is techniques was proposed and tested by Norman Maier. See Maier, N. R. F., & Hoffman, N, L. R. Quality of first and second solution on group problem solving, Journal of Applied Psychology , 1960, 44,
Adprin.com 15 15

16. Pre-testing ads
How can you pre-test ads to determine which will be more effective? Write 3 ways to test ads before they are run. When you complete this, go to the next slide.
Adprin.com

17. How can you evaluate the effectiveness of advertisements?
Combined expert judgments: Ask experts (5 to 20) to rate the effectiveness of the ads and combine.
Adherence to principles: Rate how well the ads use principles for persuasive advertising (Use 5 raters.) See the Persuasion Principles Audit .
Research
Copy-test the effectiveness of alternative ads.
Copy-test the effectiveness of components of ads (e.g., the illustration, headline, copy, sound track)
Adprin.com

18. Adherence to Persuasion Principles
Persuasion principles can be used not only for creating, but also for evaluating ads. They can identify problems and suggest how to fix them.
The principles are formulated as checklists. Checklists can improve decision making in complex situations. Such as a reduction of 47% in deaths due to surgery in hospitals
(Haynes et al. 2006)
A 2008 study evaluated the effects of the implementation of a 19-item checklist for hospital procedures. A before/after experimental design was used for thousands of patients in eight hospitals in eight cities around the world. In the month after the operations, the checklist led to a reduction in deaths from 1.5% to 0.8%, and in complications, from 11% to 7% (Haynes et al. 2009).
Adprin.com

19. How could you use consumers to evaluate ads?
Which of the following approaches are useful?
1. Focus groups to discuss their reactions to an ad.
2. Surveys to ask people how much they like an ad.
3. Surveys that ask people about their intentions to purchase when presented with different ads for the same product.
Click for the evidence-based answer
Adprin.com

20. Evidence-based recommendations
Avoid asking consumers if an ad is effective. They are not experts in persuasion.
In particular, avoid focus groups, an expensive and inappropriate method for evaluation. Focus groups tend to result in bias based on the
behavior of the group leader
wording of the questions and suggestions
reluctance to put forth unusual ideas
(Armstrong 2004)
2. Do not use “liking” as a criterion, whether by consumers or experts.
Liking is not an objective (Persuasive Advertising p 289)
Focus on persuasiveness
3. Use experiments. Assess how people are affected when presented with different ads (some get ad A while others get ad B)
Adprin.com

21. Why 195 principles? Why not the top ten?
Advertising is complex-- like medicine. Would you like to be restricted to the top ten medicines?
We use the “Index method” to determine how closely the ads adhere to principles (Armstrong & Graefe 2011).
Regression analysis does not work in such a situation as there are too many variables.
Adprin.com

22. Develop action steps for applications
Write a plan in your learning diary for applying at least one technique from this session. Be explicit about what you will do. Set a time deadline for completing the application. Start your applications report for this course. Describe your plan and, later, how it turned out.
Adprin.com

23. Techniques Used in this Lecture
Problem storming
Devil’s advocate approach
Authentic dissent and MAAD
Index method
Second solution
Adprin.com

24. Advice on learning
One study found that fewer than 10% of students were successful in applying new knowledge.
This went to 20% if they actively applied what they were taught during a class session.
It went to 90% when they worked with a learning partner and coached each other.
Adprin.com

25. Follow-up: Complete prior to next session
1. ___ Go through this lecture on your own (It is on
the Educational Materials page)
___Study Persuasive Advertising pages and record your reading time in your learning diary. Highlight techniques that you want to apply in yellow.
___ Complete the End of Chapter Questions for “Evaluating Advertisements” and check your answers against PA.
Adprin.com