1. Concept evaluation and testing
Innovation Management 2012
Stefan Wuyts

3. Agenda
Concept evaluation
Concept testing
The full screen

4. Concept evaluation

5. The overall evaluation system

6. Element of cost: Cumulative expenditures curve
Many high-tech
products
Many consumer
products
Time

7. Element of risk: Risk/Payoff matrix
Cells AA and BB are “correct” decisions.
Cells BA and AB are errors, but they have different cost and probability dimensions.
Usually BA (the “go” error) is much more costly – but don’t forget opportunity costs!
Consider how “new-to-the-world” the product is as that has an impact on the risk level

8. Real options logic as response to risk
Source of real options approach: early discussions on exploration versus exploitation (Schumpeter 1934; March 1991)
Exploitation: refinement, choice, efficiency, selection, implementation, execution  Clear results, short-term
Exploration: search, variation, risk, experimenting, flexibility, discovery  More uncertainty regarding results, long-term
Problem exploitation: suboptimal (stable) equilibria;
Problems with exploration: insufficiently developed ideas, not reaping the benefits of experimentation (more search than application), no distinctive competence;
Problems of both: they are self-reinforcing.
Challenge: find the right balance.
Real options logic: postpone the decision until uncertainty is reduced

9. Probability distribution
Imagine two investment alternatives (A1 and A2), for which the probability of payoff (in $) is initially unknown. The decision maker can either:
Gather extra information, postpone choice  improving future returns;
Use current information, choose  improving current returns.
Probability distribution A2
After further exploration in A2 (scenario 1) A2
After further exploration in A2 (scenario 2)
Before further exploration in A2 A2 A1
Pay-off ($) -9 -6 -3 3 6 9

10. Real options logic summarized:
Bet on different horses: option on >1 strategic alternatives;
For now, invest in multiple ideas until there is more clarity as to their market potential;
Investing in a real option provides the right, but not the obligation to further invest at a later point in time (or to ‘abandon’ the option at a later point in time).
Appropriate if:
Uncertainty regarding link between investments and results;
Decisions today determine opportunities tomorrow (time dependence, path-dependence)
Possibility for abandoning/executing the option.
Applied increasingly in high-tech environments where these three conditions are oftentimes met (e.g. Philips)

11. Planning the evaluation system
Keep in mind:
Everything is tentative
Financial analysis
Marketing
Avoid potholes
Damaging problems
FDA approval, customer skills, manufacturing cost
You are dealing with people
Example: in the NPD literature, it has been shown to be very difficult to pull the plug for unsuccessful projects.
versus

12. Example of Potholes in new product development: Complexity
IT IS ROCKET SCIENCE
PHILIPS ADMITS WHAT EVERYONE KNOWS: DIGITAL GADGETS ARE WAY TOO COMPLICATED FOR THE AVERAGE CONSUMER (Newsweek 2004)
NEWSWEEK: Is it true you ran a test, giving 100 of your top managers one weekend at home to get various Philips gadgets operating?
KLEISTERLEE (CEO Philips): Yes, we did. And indeed, a number failed, returned frustrated and some even angry; another group that succeeded returned quite proud. It strengthened our conviction that we must start making things easier for consumers or we will never see the real promise of the digital revolution come to life. And we must do it now.

13. Example of Dealing with People: Hard to pull the plug.
Why? One of the most-cited reasons: “Escalation of Commitment”, the tendency to continue investing in a strategy, despite negative feedback, continuation in a failing course of action.
Solutions: How can we reduce escalation of commitment bias? (Boulding, Morgan & Staelin 1997)
More information? Problem of information distortion
Emphasize uncertainty or external causes of failure?
Calculate Net Present Value of continuing to invest versus NPV of not continuing to invest?
Establish a rule a priori that determines when to stop
Sequential decision decoupling (people who decide on continuation ≠ people who are strongly involved)
If at first you don’t succeed, try, try again. Then quit. No use being a damn fool about it. -- W.C. Fields

14. Quantifying evaluation: the A-T-A-R model
Profits = Units Sold x Profit Per Unit
Units Sold = Number of buying units
x % Aware of product
x % who would Try product if they can get it
x % to whom product is Available
x Repeat measure (average # units bought per person per year x # units repeaters buy in a year)
Profit Per Unit = Revenue per unit - cost per unit

15. Concept testing

16. Why concept testing? We are not starting from scratch:
We already had a Product Innovation Charter (PIC) approved as a guide to select ideas;
We already have gained first insight into the market area singled out by the PIC;
Concepts always undergo some initial reaction by management (based on heuristics or method like A-T-A-R).
How should we proceed? Concept testing.
Concept testing takes evaluation one step further, but it precedes technical work (not the same as prototype testing). It is intended to (1) remove poor concepts, (2) get first idea of purchasing likelihood, and (3) make the concept more concrete (attributes)

17. Concept testing is not feasible if…
…prime benefit is personal sense (e.g. taste).
…concept involves new art and entertainment.
…concept embodies new technology that users cannot visualize or address needs customers can’t articulate yet (e.g. first microwave).

18. Concept testing requires clear concept statement (e. g
Concept testing requires clear concept statement (e.g. new electric razor whose screen is so thin it can cut closer than any other electric razor on the market) Necessary steps include the selection of the concept test format; preliminary indication of price; specification of respondent group and response situation; preparation of interviews

19. Application Example Nestlé Refrigerated Foods (Contadina Pasta)
24% “definitely buy” + 51% probably “would buy”
Adjusted trial, rule of thumb: 80% of the “definitely” + 30% of the “probably” will actually buy, or: (0.8 x 24%) + (0.3 x 51%) = 34.5%
Assuming 48% awareness and 70% availability, we get :
AW x T x AV = 0.48 x 34.5% x 0.70 = 11.6%
Target households x trial rate = 77.4 million x 11.6% = 9 million
Repeat for similar products = 39%; average customer repeat = 2.5 times; No. of units bought per repeat purchase occasion = 1.4
Additional sales because of repeats: 39% x 2.5 x 1.4 = 136.5%
Hence, the “R” in the A-T-A-R model = 1 + (.39 x 2.5 x 1.4) = 2.365
A-T-A-R Sales prediction: 9 million x = million.
Note that this deviates from the calculations in the book! Good exercise to identify and think about the differences in calculations.

20. The full screen

21. Toward more informed allocation decisions
Refine and rank-order the concepts (on basis of mathematical models or checklists)
Forces feasibility evaluation along technical and commercial dimensions, and summarizes what must be done.
Cross-functional: involve major functions (marketing, technical, operations, finance), new products managers, staff specialists (IT, distribution, procurement, PR, HR)
 Decide whether or not to allocate further resources to each concept

22. Example (simple) mathematical model: profile analysis
Profile analysis: I = T*C*P/D
I = index of attractiveness
T = probability of successful technological development
C = prob. of commercial success if technological success
P = Likely profit
D = development cost
Keep it simple;
Need for qualitative input (accuracy, error);
Pay attention to deliberate underestimation of costs;
Select several ideas => higher returns.

23. Example: checklist + scoring model
Cooper’s NewProd study identified must-meet and should- meet factors across many product successes and failures
Must-Meet Criteria (rated yes/no)
Should-Meet Criteria (rated on scales)
Strategic alignment
Strategic (alignment and importance)
Product advantage
Product advantage (unique benefits, meets customer needs, provides value for money)
Existence of market need
Market attractiveness (size, growth rate)
Likelihood of technical feasibility
Technical feasibility (complexity, uncertainty)
Return versus risk
Return versus risk (NPV, ROI)
Environmental health and safety policies
Synergies (marketing, distribution, technical, manufacturing expertise)
Show stoppers (“killer” variables)