7 Experimentation and Test Markets

What is an Experiment? Experiment: research approach in which one variable (called an experimental, treatment, independent, or explanatory variable) is manipulated and the effect on another variable (referred to as a dependent variable) is observed. Example: the effect of advertising on sales or the effect of price on sales. – Advertising/price: experimental variable – Sales: dependent variable

Demonstrating Causation Causal research: research designed to determine whether a change in one variable likely caused an observed change in another. To demonstrate causation (that A likely caused B), one must be able to show 3 things: 1.Correlation or concomitant variation 2.Appropriate time order of occurrence 3.Elimination of other possible causal factors

Demonstrating Causation Concomitant variation – To provide evidence that a change in A caused a particular change in B, one must first show that there is correlation between A and B; in other words, A and B must vary together in some predictable fashion. – This might be a positive (i.e. advertising and sales) or inverse relationship (i.e. price and sales).

Demonstrating Causation Appropriate time order of occurrence – To demonstrate that A caused B, one must be able to show that A occurred before B occurred. – Example: to demonstrate that a price change had an effect on sales, you must be able to show that the price change occurred before the change in sales was observed.

Demonstrating Causation Elimination of other possible causal factors – The most difficult thing to demonstrate in marketing experiments is that the change in B was not caused by some other than A. – Example: suppose a company increased its advertising expenditures and observed an increase in sales. It is possible that the observed change in sales is due to some factor other than increase in advertising: a major competitor might have decreased advertising expenditures, or increased price, or pulled out of the market, so one or a combination of other factors may have influenced sales.

Experimental Setting Experiments can be conducted in a laboratory or in a field setting. Most experiments in the physical sciences are conducted in a laboratory setting; while many marketing experiments are field experiments. Laboratory experiments: experiments conducted in a controlled setting. Field experiments: tests conducted outside the laboratory in an actual environment, such as a marketplace.

Experiment Validity Validity: the degree to which an experiment actually measures what the researcher was trying to measure. The validity of a measure depends on the extent to which the measure is free from both systematic and random error. – Internal validity: extent to which competing explanations for the experimental results observed can be ruled out. – External validity: extent to which causal relationships measured in an experiment can be generalized to outside persons, settings, and times. Field experiments offer a higher degree of external validity and a lower degree of internal validity than do laboratory experiments.

Extraneous Variables Examples of extraneous factors/variables that pose a threat to experiment validity are: History: intervention, between the beginning and end of an experiment, of outside variables or events that might change the dependent variable. – Example: reading potential sales of Prego whie Ragu is having its promotional campaign may not get the accurate sales reading. Maturation: changes in subjects occurring during the experiment that are not related to the experiment (a function of time) but that may affect subjects’ response to the treatment factor. It includes getting older, hungrier, more tired, and etc.

Extraneous Variables Examples of extraneous factors/variables that pose a threat to experiment validity are: Instrument variation: changes in measurement instruments (e.g. interviewers or observers) that might affect measurements. Selection bias: systematic differences between the test group and the control group due to a biased selection process. Mortality: loss of test units or subjects during the course of an experiment, which may result in a nonrepresentativeness. Example: in a study of music preferences of the population, if nearly all the subjects under the age of 25 were lost during the course of the experiment, then the researcher would likely get a biased picture of music preferences at the end of the experiment. As a result, the finding would lack external validity.

Extraneous Variables Examples of extraneous factors/variables that pose a threat to experiment validity are: Testing effect: effect that is a by-product of the research process itself. Example: measuring attitude toward a product before exposing subjects to an ad may act as a treatment variable, influencing perception of the ad. Regression to the mean: tendency of subjects with extreme behavior to move toward the average for that behavior during the course of an experiment. Example: the researcher might have chosen people for an experiment group because they were extremely heavy users of a particular product/service. In such situations, their tendency to move toward the average behavior may be interpreted as having been caused by the treatment variable when in fact it has nothing to do with the treatment variable.

Extraneous Variables Controlling Extraneous Variables Causal factors that threaten validity must be controlled in some manner to establish a clear picture of the effect of the manipulated variable on dependent variable. 4 basic approaches are used to control extraneous factors: randomization: random assignments of subjects to treatment conditions to ensure equal representation of subject characteristics. Physical control: holding constant the value or level of extraneous variables (i.e. age, income, lifestyle) throughout the course of an experiment. Design control: use of the experimental design to control extraneous causal factors. Statistical control: adjusting for the effects of confounded variables (extraneous causal factors) by statistically adjusting the value of the dependent variable for each treatment condition.

Experimental Design, Treatment, and Effects Experimental design: test in which the researcher has control over and manipulates one or more independent variables. An experimental design includes 4 elements: 1.The treatment, or experimental, variable (independent variable) that is manipulated 2.The subjects who participate in the experiment 3.A dependent variable that is measured 4.Some plan or procedure for dealing with extraneous causal factors

Experimental Design, Treatment, and Effects Treatment variable: independent variable that is manipulated in an experiment. – Manipulation: a process in which the researcher sets the levels of the independent variable to test a particular causal relationship. – Example: to test the relationship between price (independent v.) and sales (dependent v.), a researcher might expose subjects to 3 different levels of price and record the level of purchases at each price level. As a variable that is manipulated, price is the single treatment variable, with 3 treatment conditions/ levels.

Experimental Design, Treatment, and Effects Experimental effect: effect of the treatment variable on the dependent variable. Example: 3 different markets are selected to test 3 different prices (treatment conditions) for 3 months. MarketPrice Experimental effect (change in sales)  12% lower  increased less than 1%  24% lower  increased by 3%  36% lower  increased by 5%

Limitations of Experimental Research High cost of experiments Security issues Competitors can decide whether and how to respond or stolen concepts that were being tested in the marketplace. Implementation problems Example: a regional marketing manager might be very reluctant to permit her market area to be used as a test market for a higher price that might lower sales for the area.

Selected Experimental Designs Pre-experiment designs Designs that offer little or no control over extraneous factors. Studies using this designs are often difficult to interpret because they offer little or no control over the influence of extraneous factors. – One-shot case study design: no pretest observations, no control group, and an after measurement only (O1) XO1 – One-group pretest-posttest design: pre (O1)- and postmeasurements (O2) but no control group O1XO2 X: the exposure of an individual/a group to an experimental treatment O (observation): the process of taking measurements on the test units (individuals, groups of individuals, or entities whose response to the experimental treatments is being tested: individual/group of consumers, retail stores, total markets, and etc.)

Selected Experimental Designs True experimental designs Research using an experimental group an a control group, to which test units are randomly assigned (R). These experimental designs are superior because randomization takes care of many extraneous variables. – Before and after with control group design: involves random assignment of subjects or test units to experimental and control groups and pre – and postmeasurements of both groups. Experimental group:(R)O1 X O2 Control group:(R)O3 X O4 – After-only with control group design: involves random assignment of subjects or test units to experimental and control groups, but no premeasurement of the dependent variable. Experimental group:(R)X O1 Control group:(R) O2

Selected Experimental Designs Quasi-experiments Studies in which the researcher lacks complete control over the scheduling of treatments or must assign respondents to treatments in a nonrandom manner. These designs frequently are used in marketing research studies because cost and field constraints often do not permit the researcher to exert direct control over the scheduling of treatments and the randomization of respondents. – Interrupted time-series design: research in which repeated measurement of an effect “interrupts” previous data patterns. Ex: the use of consumer purchase panels to make periodic measurements of consumer purchase activity (Os) with introducing a new promotional campaign (X). O1 O2 O3 O4 X O5 O6 O7 O8 – Multiple time-series design: interrupted time-series design with a control group, then researchers can be more certain in their interpretation of the treatment effect. Experimental group:O1 O2 O3 O4 O5 O6 Control group:O1 O2 O3 O4 O5 O6

Test Markets Test market: real-world testing of a new product or some element of the marketing mix using an experimental or quasi-experimental design. Test-market studies are designed to provide info. regarding to the following issues: – Estimates of market share and volume. – The effects that the new product will have on sales of similar products (if any) already marketed by the company. (cannibalization rate) – Characteristics of consumers who buy the product. – The behavior of competitors during the test.

Test Markets Types of test markets Traditional (standard) test market: testing the product and other elements of the marketing mix through regular channels of distribution which normally take 6 months or more. Scanner (electronic) test market: research firms have panels of consumers who carry scannable cards for use in buying particular products, esp. those sold through grocery stores. These panels permit us to analyze the characteristics of those consumers who buy and don’t buy the test products such as demographic data, pas purchase history, or even media viewing habits.

Test Markets Types of test markets Controlled test markets are managed by research suppliers who ensure that the product is distributed through the agreed upon types and numbers of outlets. The research suppliers pay distributors to provide the required amount of shelf space for test products and carefully monitor sales of the product in these controlled test markets. Simulated test markets (STMs): simulations of the types of test markets noted above. They can be conducted more quickly than other approaches, at a lower cost, and can produce results that are highly predictive of what will actually happen.

Test Markets Decision to conduct test marketing 1.Weigh the cost and risk of failure against the probability of success and associated profits 2.Consider the likelihood and speed with which competitors can copy your product and introduce it on a national basis. 3.Consider the investment required to produce the product for the test market vs. the investment required to produce the product in the quantities necessary for a national rollout. 4.Consider how much damage an unsuccessful new product launch would inflict on the firm’s reputation.

Test Markets Steps in a test market study 1.Define the objective of the test 2.Select a basic approach (type of test market) 3.Develop detailed test procedures (basic positioning approach, the actual commercials to be used, pricing strategy, media plan, and etc.) 4.Select test markets 5.Execute the plan 6.Analyze the test results throughout the test period (purchase data, awareness data, competitive response, source of sales)