Experimental Design: Part I MAR 6648: Marketing Research February 3, 2010

Overview What are the basic features of an experiment? How do those features get implemented in a real experiment? How do we adapt experiments to meet our goals and resources?

1.Experimentation is the conscious manipulation of one or more variables by the experimenter in such a way that its effect on one or more variables can be measured. 2.The variable being manipulated is called the independent variable (a.k.a. cause). 3.The variable being measured is called the dependent variable (a.k.a. effect). 4.Elimination of other possible causal factors: i.e., the research design should rule out the other factors (exogenous variables) as potentially causal ones. 5.This is typically done through random assignment to condition

An example of an experiment Suppose you want to know whether commercials make people enjoy TV shows less This means you’ll want to have some shows without commercials, and some shows with them – Therefore, commercials (or not) is the independent variable And you’ll want to measure enjoyment of the TV shows they watch – Therefore, enjoyment is the dependent variable

Conditions Not in terms of what you can and can’t do… Each independent variable (or combination of IVs) is called a condition Condition 1Condition 2

Hypotheses Experimentation is essentially the process of trying to determine which of two hypotheses is not false The null hypothesis: – H 0 : Usually that there are no differences between conditions The alternative hypothesis: – H 1 : usually that there is a difference between conditions P-values in stats essentially represent the likelihood that we found evidence for H 1 by chance alone

Confirmation Bias We are inclined to confirm our beliefs but less inclined (or able) to disconfirm them A real world example: – Business managers don’t keep track of those they don’t hire Why? – Theories lead to unwarranted confidence – Inability to search out disconfirmation – Fixation or mental set

Control condition Control conditions allow us to see that our manipulations caused (or didn’t cause) a change in the dependent variable Usually a control condition is just no manipulation – This is sometimes done by adjusting when you run your manipulation Sometimes, though, you want to compare your new manipulation to what’s typically done now – The control condition may be the standard or default

Random Assignment This essentially means that any one participant is equally likely to be in any condition – Usually you put your conditions in random order, and assign participants in the order that they “arrive” – Computers now allow you to assign people on the spot Randomizer.org or random.org are good sources

An example of an experiment The hallmark of an experiment is random assignment to conditions – Let’s say the groups (the commercial watchers and the people who watch it straight through) now look different! – Random assignment means that the two groups should not have differed systematically at the start – It also means that only your independent variable was different between groups Random assignment and manipulation of the IV mean that you can infer that the IV causes a change in the DV

An example of an experiment Question: do commercials make you enjoy a TV show less? Do people correctly predict this? Randomly assign your participants to groups – Half will predict how they enjoy a TV show with or without them, half will actually experience it and report how they feel – Half will watch a TV show with commercials, half will watch the same show without them Measure enjoyment or predicted enjoyment

An example of an experiment Nelson, Meyvis, & Galak, 2009

Example Objective: GAP wishes to gauge whether new more aggressive sales techniques employed by store assistants increase sales What is the best experimental design?

Experiment 1 Design: – 50 stores are sampled at random and assistants are trained in the new approach Metric = MINUS Average sales for the 50 stores in the next six months Average sales for the 50 stores in the next six months Average sales for the 50 stores in the prior six months Average sales for the 50 stores in the prior six months

Notation X = Exposure of a sample to the independent variable (i.e., what we manipulate – “treatment”) O = Observation of measurement of the dependent variable (i.e., what we measure / want to affect) Movement through time is represented by the horizontal arrangement of Xs and Os from left to right.

Experiment 1: One group – before after Causal Effect of X = O 2 -O 1

Problems with this design? History or maturation Defensiveness Mortality Instrumentation

Experiment 2 Design: – 50 stores are sampled at random and the assistants are trained in the new approach – Another 50 stores are sampled at random as control Metric = MINUS Average sales for the 50 test stores in the next six months Average sales for the 50 test stores in the next six months Average sales for the 50 control stores in the next six months Average sales for the 50 control stores in the next six months

Experiment 2: Two group – only after Causal Effect of X = O 2 -O 1

Problems with this design?

Experiment 3 Design: – 50 stores are sampled at random and assistants are trained in the new approach – Another 50 stores are sampled at random as control Metric = MINU S Average sales for the 50 test stores in the next six months Average sales for the 50 test stores in the prior six months Average sales for the 50 control stores in the next six months Average sales for the 50 control stores in the prior six months

Experiment 3: Two group – before after Causal Effect of X = O 4 -O 3 – (O 2 -O 1 )

More Advanced Experiments We have so far mainly looked at simple experiments But often we need to test several variables When deciding on a marketing plan for a new product there are many factors involved

Factorial Design Suppose we wish to test both product price and web-design for an e-business $9.99$14.99$19.99 Design 1 Design 2 Price Design Full Factorial Design! Full Factorial Design!

Interactions and main effects

Factorial Design What do we do if we have many factors and levels? Example: – 5 prices, 4 product designs, 3 ad-copies  5*4*3 = 60 experimental cells! Solution: Use a fractional factorial design – Only use a subset of all 60 cells in experiment – Rely on regression analysis to extrapolate

Latin Squares 1 st ad2 nd ad3 rd ad4 th ad Group #1 Positive ad, Male speaker Positive ad, Female speaker Negative ad, Female speaker Negative ad, Male speaker Group #2 Positive ad, Female speaker Negative ad, Male speaker Positive ad, Male speaker Negative ad, Female speaker Group #3 Negative ad, Male speaker Negative ad, Female speaker Positive ad, Female speaker Positive ad, Male speaker Group #4 Negative ad, Female speaker Positive ad, Male speaker Negative ad, Male speaker Positive ad, Female speaker

Latin Squares 1 st ad2 nd ad3 rd ad4 th ad Group #1 αβδγ Group #2 βγαδ Group #3 γδβα Group #4 δ αγβ

An experiment? Steve was interested to see how much labels on wine bottles affect how much people enjoy the wine inside them. At a party, he served the wines like normal, leaving the bottles out for people to pour from, labels still on. He asked everyone to indicate which wine they liked the best. At the next party he threw, he poured the wine into decanters, so that his guests couldn’t see the labels when they poured the wine. They again indicate which wine they liked best, and they had different preferences from the last party.

An experiment? The owner of two McDonalds franchises here in Gainesville wants to see if transactions run more quickly if he uses both drive-thru windows or only one. He picks one restaurant to use both windows at all times for a month, and the other he has closed at all times for a month. He finds that the drive-thru that uses both windows has notably faster service times.

An experiment?

Summary Experiments are very useful for determining causality – The main hallmarks of experiments are random assignment to condition, manipulation of the independent variable, and a control group – There are many different types of experiments, which vary largely on whether they are run within or between subjects (or both), when the manipulation is run, and how many conditions are used