1. Marketing Experiments II
This module builds on Marketing Experiments I and provides guidance for how one might extrapolate the results from the various experiments to the total market for the product/service. We will consider three contexts: geography, channels, and seasonality. This module presumes knowledge already covered in Basic Margins, Cannibalization, and Marketing Experiments I.
Authors: Raj Venkatesan and Stu James
© 2018 Raj Venkatesan, Stu James, and Management by the Numbers, Inc.

2. Quick Review of Marketing Experiments I
In Marketing Experiments I, we learned how to design an experiment to determine the lift relative to a base case for various changes in marketing activities. In this module, we will explore how to extrapolate the results from the experiment to estimate the net economic impact of a change of a marketing variable. First, however, let’s review how to calculate the lift from a basic marketing experiment.
Definitions: Basic Experiment Design
Lift (Units) = Test Group Sales (Units) – Control Group Sales (Units)
Lift (%) = Test Group Sales (Units) / Control Group Sales (Units)
Net Lift (%) = Lift (Units) / Control Group Sales (Units)
MBTN | Management by the Numbers

3. Applying Experiment Results
Designing and analyzing marketing experiments is only the first half of the managerial decision process. Once the preferred outcome is identified, the manager must next determine the net economic impact of that choice.
In order to examine the broader economic impact, one must extrapolate the results of the experiment to the broader market. This is, of course, an imperfect science, but let’s describe some techniques that can at least serve as a starting point.
We’ll use three contexts to illustrate simple extrapolation approaches: geography, channels, and seasonality.
MBTN | Management by the Numbers

4. Applying Experiment Results
Experiments, by design, are done on a limited subset of a larger market. So there are two key assumptions to applying the results to the broader market:
The percentage of the market that the subset represents is measureable and known.
The broader market can be expected to react to the change in the marketing element(s) similarly to the experimental group.
Both of these issues should be addressed as best as possible in the experimental design. In our examples, we will presume that both of these assumptions are valid.
As an example that illustrates the importance of these assumptions, consider a test market in Sweden of a new type of all season tire designed and marketed for snowy conditions and applying those results to Italy/Egypt. Similar concerns could be raised for a January test period vs. year-round sales.
MBTN | Management by the Numbers

5. Estimating the Size of the Total Market
Let’s use these simple examples to establish the basis for extrapolating test results from a limited geographic, channel, and/or timeframe to a broader market.
Question: What is the size of the total market given the following information?
Metropolis represents 1% of the total market and sales are 2,000 units yearly.
SpreadAround distributors represents 5% of the total market and sales through this channel are 10,000 units yearly.
Total market sales in April are 5,000 units and April sales represent 2.5% of the total market.
Answers:
2,000 / .01 = 200,000 units
10,000 / .05 = 200,000 units
5,000 / .025 = 200,000 units
MBTN | Management by the Numbers

6. Estimating Lift MBTN | Management by the Numbers Estimating Lift
Now let’s apply this same approach to an experiment using lift. An advertising experiment was run in April in Metropolis where baseline sales were 50 units, but the experiment yielded 80 units.
Recall that:
Metropolis represents 1% of the total market and sales are 2,000 units yearly.
Total market sales in April are 5,000 units and April sales represent 2.5% of the total market.
Question: What is the lift (units), % lift, net lift, and what can we expect the increase in sales to be for the market as a whole?
MBTN | Management by the Numbers

7. Estimating Lift MBTN | Management by the Numbers Estimating Lift
Answers:
Lift (units) = Test group sales - control group sales
Lift (units) = = 30 units
Lift % = Test group / control group sales
Lift % = 80 / 50 = 1.60 or 160%
Net Lift % = Lift (units) / Control Group Units
Net Lift % = 30 / 50 = 60%
To calculate the increase for the market as a whole apply the lift (units) and adjust for market size and timing.
Projected increase = 30 / .01 = 3000 units (for April)
Projected increase = 3000 / .025 = 120,000 units
Quick check: 120,000 / 200,000 = 60% = Net Lift %
MBTN | Management by the Numbers

8. Estimating Lift MBTN | Management by the Numbers Estimating Lift
Insight
Remember this approach works best when the geographic, channel, and/or time period is representative of the market as a whole. Recall snow tires in Sweden (test) vs. Italy (target market) or January (test) vs. annual (goal).
SpreadAround distributors was given a 10% price discount as a test in April. Control group sales were 250 units and the test group sales were 300 units.
Recall that:
Sales through SpreadAround accounted for 5% of the total market and sales are 10,000 units yearly.
Total market sales in April are 5,000 units and April sales represent 2.5% of the total market.
Question: What is the lift (units), % lift, net lift, and what can we expect the increase in sales to be for the market as a whole?
MBTN | Management by the Numbers

9. Estimating Lift MBTN | Management by the Numbers Estimating Lift
Answers:
Lift (units) = Test group sales - control group sales
Lift (units) = = 50 units
Lift % = Test group / control group sales
Lift % = 300 / 250 = 1.20 or 120%
Net Lift % = Lift (units) / Control Group Units
Net Lift % = 50 / 250 = 20%
To calculate the increase for the market as a whole apply the lift (units) and adjust for market size and timing.
Projected increase = 50 / .05 = 1000 units (for April)
Projected increase = 1000 / .025 = 40,000 units
Quick check: 40,000 / 200,000 = 20% = Net Lift %
Question: Finally, which of the two marketing activities should the firm pursue on a nationwide basis – the advertisement or the channel promotion?
MBTN | Management by the Numbers

10. Estimating Lift MBTN | Management by the Numbers Estimating Lift
Answer:
An initial thought might be that the advertisement would be the better approach given the higher lift rate. However, we haven’t taken into consideration any of the economic aspects of the decision, only the market response estimates. Now let’s bring margins and costs into the discussion!
Insight
There is a temptation in a sales-oriented organization to focus primarily on positive response functions that increase sales. This is especially evident when incentive compensation is oriented around sales and/or commissions. However, as a manager, your job is to understand the economic impact of these choices; which is even more complicated when considering competitive dynamics and short vs. long term impacts.
MBTN | Management by the Numbers

11. Estimating Economic Impact
Now let’s add the necessary economic information to help answer the question of which marketing activity is likely to be more profitable for the company.
The standard price to distributors is $200 / unit.
The total variable cost of the unit is $150 / unit.
The advertising budget for the national campaign for the year is $5 million.
And recall that the promotional discount to distributors is 10% per unit (price discount).
Question: What is the estimated net economic impact of each of the two marketing activities if the marketing activity is used for the entire year nationwide?
MBTN | Management by the Numbers

12. Estimating Economic Impact
Answer: Advertisement
Recall that expected lift was 120,000 units
Margin = $200 - $150 = $50 / unit
Total Contribution Value of Lift
= $50 * 120,000 units = $6,000,000
Net Contribution Value of Advertisement = Contribution Value of Lift – Marketing Cost
= $6,000,000 - $5,000,000 = $1,000,000
Answer: Promotional Discount
Recall that expected lift was 40,000 units
Margin = Price less Promotional Discount – Cost
= 200 * ( ) - $150 = $180 - $150 = $30 / unit
= $30 * 40,000 units = $1,200,000
$1,200,000 > $1,000,000 so the distributor promotion discount appears to be the better option.
MBTN | Management by the Numbers

13. Estimating Economic Impact
Insight
It is worth considering whether running the advertisement or promotion for the entire year would yield the same results proportionally as running it for a month. It might be, for example, that people purchased the units in that single month because of the lowered price, “stealing” units from future sales. These are the types of dynamics that are more difficult to quantify.
Question: Aliyah, the manager, thought it might be more effective to run the promotion or advertisement during peak holiday season, which historically accounted for 40% of their sales. The advertising budget would be 10% of the annual budget for only advertising during the peak holiday season. Given this change in duration and season, what would be the likely net economic impact of the advertisement and promotion?
MBTN | Management by the Numbers

14. Estimating Economic Impact
Answer: Advertisement
Lift = 120,000 *.40 = 48,000 units (40% of yearly lift)
Margin = $200 - $150 = $50 / unit
Total Contribution Value of Lift
= $50 * 48,000 units = $2,400,000
Net Contribution Value of Advertisement = Contribution Value of Lift – Marketing Cost (10% of yearly cost)
= $2,400,000 - $500,000 = $1,900,000
Answer: Promotional Discount
Lift = 40,000 * .40 = 16,000 (40% of yearly lift)
Margin = Price less Promotional Discount – Cost
= 200 * ( ) - $150 = $180 - $150 = $30 / unit
= $30 * 16,000 units = $480,000
$1,900,000 > $480,000 so the advertisement appears to be the better option. It is also worth noting that the holiday timing is better than either year long option.
MBTN | Management by the Numbers

15. Estimating Net Economic Impact
Let’s go through one last example from start to finish where we’ll add three additional considerations – a new product introduction, control methodology, and cannibalization concerns.
McFriQueen, a fast food chain and home of the famous Triangle Burger (Tri Burger), wants to estimate the potential yearly contribution of its new product idea, the Square Burger Deluxe (SBD). To test this concept, they introduced it in limited test markets with local advertising support. The test markets represent 2% of the total market potential nationwide.
In a 4 week period (historically, typical of the year as a whole), they sold 20,000 SBDs in the test markets at a price of $5.00 each. It costs McFriQueen $2.00 per burger in variable costs. For a national introduction, they have budgeted $20,000,000 in advertising support.
Question: Based on this information alone, what is the anticipated net economic impact of introducing the SBD nationwide?
MBTN | Management by the Numbers

16. Estimating Economic Impact
Answer
Total market potential = Test Market Sales / Test Market % of Total
= 20,000 / .02 = 1,000,000 SBDs in a 4 week period nationwide.
= 1,000,000 / (4 wks / 52 wks per year) = 13 Million SBDs in a year
Annual Contribution Value of New Product Introduction
= Margin * Units Sold = ($5 - $2) * 13 million = $39 Million
Net Economic Impact = Contribution Value of Prod. Intro. – Marketing Cost
= $39,000,000 - $20,000,000 = $19,000,000
Harold, who oversaw restaurant operations, wasn’t convinced. He had two concerns with the methodology. First was that in one of the test markets, a hurricane had disrupted restaurant operations for at least a week. Second was that he was concerned that in the test markets, sales of the Tri Burger had decreased and he thought that was likely due to the SBD introduction. Mary Jo, the marketing manager of the SBD, asked for additional data to analyze these concerns.
MBTN | Management by the Numbers

17. Tri Burger Sales (Prev 4 Weeks)
Estimating Economic Impact
Estimating Economic Impact
SBD
Tri Burger
Tri Burger Sales (Prev 4 Weeks)
Store Sales %
Week 1
4,800
26,000
30,100
2.045%
Week 2
5,400
27,000
29,700
2.090%
Week 3 (Hurricane)
4,200
24,200
31,000
1.825%
Week 4
5,600
26,800
29,200
2.075%
Total
20,000
104,000
120,000
In reviewing this additional data, Mary Jo understood Harold’s concerns. First, she decided she wanted to address the potential impact of the hurricane on the test market data. She thought there were two ways she could do this. One was to only consider the 3 weeks of test market data that weren’t impacted by the hurricane. Two was to recalculate week 3’s results adjusting for the impact of the hurricane using expected sales rates based on the test market % of total sales. Let’s recalculate total market potential and net economic impact using these two adjusted approaches.
MBTN | Management by the Numbers

18. Estimating Economic Impact
Answer (Removing week 3 data)
Total market potential = Test Market Sales / Test Market % of Total
= (20,000 – 4,200) / .02 = 790,000 SBDs in a 3 week period nationwide
= 790,000 / (3 wks / 52 wks per year) = 13.7 Million SBDs in a year
Annual Contribution Value of New Product Introduction
= Margin * Units Sold = ($5 - $2) * 13.7 million = $41 Million
Net Economic Impact = Contribution Value of Prod. Intro. – Marketing Cost
= $41,000,000 - $20,000,000 = $21,000,000 ($2 Million Additional)
Answer (Adjusting hurricane week using expected sales in test markets)
Adjusted Sales = Hurricane Week Sales / (Ratio of Non-Hurricane Week Sales to Hurricane Week Sales) = 4,200 * (2.07% / 1.825%) = 4,764
Total Market Potential = (20,000 – 4, ,764) / .02 = 1,028,000 SBDs
= 1,028,000 / (4 wks / 52 wks per year) = Million SBDs in a year
= Margin * Units Sold = ($5 - $2) * million = $40.1 Million
= $40,100,000 - $20,000,000 = $20,100,000 ($1+ Million Additional)
MBTN | Management by the Numbers

19. Estimating Economic Impact
Insight
Both of these methodologies are improvements over not considering the impact of the hurricane. The important point is that these adjustments can easily move forecasts by millions of dollars. Basically, “errors” and adjustments are amplified by the same ratio of the test market to the entire market!
Mary Jo was pleased with the first round of adjustments as the argument for introducing the SBD was even stronger now. Now she wanted to try to take into consideration the issue of potential cannibalization. On the Triangle Burger, the contribution margin was $2.50 instead of $3.00/burger for SBD, so she figured every sale of a SBD was an improvement of $.50, so this would probably be a positive adjustment as well. But Harold reminded her that currently every sale of a SBD was a new sale rather than a replacement sale. Let’s see!
MBTN | Management by the Numbers

20. Estimating Economic Impact
Answer (Cannibalization)
First, let’s estimate the units of Triangle Burgers cannibalized by the sales of the SBD. For our purposes here, let’s use the non-hurricane weeks to do so.
Avg weekly sales of Tri Burgers (pre SBD) = 120,000 / 4 = 30,000
Avg weekly sales of Tri Burgers (post SBD) = (104,000 – 24,200) / 3 = 26,600
So, it appears that 30, ,600 = 3,400 of the sales of the Tri Burger were cannibalized by the SBD in the test markets. Annually, for the entire market, this equates to (3400 / .02) / (1 / 52) = 8,840,000 Tri Burgers cannibalized.
Net Economic Impact Including Cannibalization + Hurricane
Value of Cannibalized Tri Burger Sales
= 8,840,000 * $2.50 = $22,100,000
Previous Calculated Net Economic Impact of SBD = $21,000,000
Adjusted Net Economic Impact
= $21,000,000 - $22,100,000 = -$1,100,000
MBTN | Management by the Numbers

21. Estimating Economic Impact
Insight
It appears that cannibalization is one of the main drivers of whether launching the SBD is a good decision or not. Failing to consider its impact would be a costly error – and perhaps more research is warranted to better estimate the impact of cannibalization. Almost as importantly, it is now easy to see how those $1 million swings in assumptions can move it from a positive to a negative net economic impact (in fact, if you rerun the values using Tri Burger sales from weeks 1, 2, and 4 only, the net impact would turn positive again – try it on your own).
But that shouldn’t be the end of the discussion. Some further research might consider the following:
Are the new SBD buyers also purchasing other items (fries, drink, etc.)?
Do new SBD buyers return to the restaurant again in future?
Are the new SBD buyers switching from a competitive brand?
Are SBD buyers who previously purchased Tri-Burgers just trying it once or will they continue to buy it?
All of these questions affect net economic impact and are the types of questions that make marketing a fascinating and challenging science!
MBTN | Management by the Numbers

22. Further Reference For Further Reference:
Marketing Metrics by Farris, Bendle, Pfeifer and Reibstein, 2nd edition, chapter 8.
MBTN Modules: Marketing Experiments I, Cannibalization, Promotion Profitability, and Marketing ROI.
MBTN | Management by the Numbers