AMA Marketing Effectiveness Online Seminar Series Lynette Rowlands American Marketing Association

Proprietary and Confidential, Maritz Inc. © A wealth of information is available for marketing professionals at The #1 marketing site on the web

Proprietary and Confidential, Maritz Inc. © Commonly Asked Questions 1. Will I be able to get copies of the slides after the event? 2.Is this web seminar being taped so I or others can view it after the fact?

Proprietary and Confidential, Maritz Inc. © Commonly Asked Questions Yes 1. Will I be able to get copies of the slides after the event? 2.Is this web seminar being taped so I or others can view it after the fact? Yes

Proprietary and Confidential, Maritz Inc. © Introducing Today’s Speaker Using Importance Measurement to Drive Product Improvements Keith Chrzan Vice President, Marketing Sciences Maritz Research

Proprietary and Confidential, Maritz Inc. © Agenda  Introduction  Stated Importance Methods  Derived Importance Methods  Summary Introduction

Proprietary and Confidential, Maritz Inc. © Why Measure Importance?  Products or services can be thought of as bundles of attributes (properties, features, benefits, etc.)  Marketers usually cannot afford to optimize all attributes at once, so they must prioritize  In order to prioritize, marketers must know the relative importance of the attributes, hence the need for importance measurement  May be part of customer satisfaction, image/positioning, brand choice, loyalty, concept testing or other studies Introduction

Proprietary and Confidential, Maritz Inc. © Myers and Alpert’s Clarification  Salience – attribute is easily brought to mind  Importance – attribute is important  Determinance – attribute is important and brands differ on it, so that it “determines” preference or choice Introduction

Proprietary and Confidential, Maritz Inc. © Stated Importance  Direct questioning methods  e.g. Open-ends, rating, ranking, sorts, etc.  Respondents’ answers directly tell us what attributes are more important than others Introduction

Proprietary and Confidential, Maritz Inc. © Derived Importance  Indirect questions  Respondent describes her actual experience  Respondent evaluates that experience  We infer importance by relating descriptions to the evaluations  We apply statistical predictive models to respondent-supplied attribute and evaluative judgments and use statistical outputs as measures of importance Introduction

Proprietary and Confidential, Maritz Inc. © Measuring Attribute Importance  Introduction  Stated Importance Methods  Unconstrained methods  Constrained methods  Derived Importance Methods  Summary Stated importance

Proprietary and Confidential, Maritz Inc. © Objectives  As a result of this section, you will be able to  Describe two broad classes of stated importance measures  List seven specific kinds of stated importance measurement  Identify the strengths and weaknesses of the various kinds of stated importance measurement  Determine the best stated importance measure for your project using a decision tree Stated importance

Proprietary and Confidential, Maritz Inc. © Measuring Attribute Importance  Introduction  Stated Importance Methods  Unconstrained methods  Open-end questions  Importance ratings  Constrained methods  Derived Importance Methods  Summary Stated importance

Proprietary and Confidential, Maritz Inc. © Open-End Questions  Example: What attributes are important when choosing a ?  Advantages  Simple to ask in any survey modality (mail, phone, Web, etc.)  Question does not force response categories on respondent – researcher may learn about new attributes  Disadvantages  Importance and recall are confounded  Open-ends may measure salience more than importance  Recommendation: Use for exploratory research, not for importance measurement Stated importance

Proprietary and Confidential, Maritz Inc. © Importance Ratings  Example: When choosing a how important is— Not at all Extremely Important Fast service12345 Wide variety12345 Reliability12345 Ease of use12345 Country of origin12345 Package color12345 Stated importance

Proprietary and Confidential, Maritz Inc. © Importance Ratings  Advantages  Easy to ask in any survey modality  Respondents are familiar and comfortable answering importance ratings  Clients are familiar and comfortable receiving importance ratings Stated importance

Proprietary and Confidential, Maritz Inc. © Importance Ratings  Disadvantages  Respondents tend to rate most attributes very positively – this reduces ability to discriminate among them and hampers subsequent analyses  Scale use heterogeneity  Some respondents use high part of the scale others use low part – positional heterogeneity  Some respondents use a wider range of the scale than do others – heteroskedasticity  These response effects hampers interpretation and multivariate analyses and harms cross-cultural comparisons  Recommendation: Use as a last resort Stated importance

Proprietary and Confidential, Maritz Inc. © Importance Ratings  Empirical evidence against importance ratings  Discriminant analysis with brand used as dependent variable and importance ratings as predictors seldom makes sense, and it should, if importances are meaningful  Derived importance works like this: You model some dependent variable (Y) as a linear function of some performance measures (X) and you calculate coefficients that are proxies for importance  If you do the reverse, modeling Y as a function of importance ratings, the coefficients should be measures of performance, but they usually are uncorrelated with actual measures of performance  Importance ratings are next to worthless

Proprietary and Confidential, Maritz Inc. © Measuring Attribute Importance  Introduction  Stated Importance Methods  Unconstrained methods  Constrained methods  Rank order  Q-sort  Constant sum  Method of paired comparisons  Maximum difference scaling  Derived Importance Methods  Summary Stated importance

Proprietary and Confidential, Maritz Inc. © Rank Ordering  Example: Please rank these 11 features of. Give the most important feature a “1,” the next most important feature a “2,” and so on until the least important feature has a rank of “11.”  Advantages  Works well on mail, Web or in-person surveys if attribute list is short  Response distribution is constrained so rank information is standardized for use in cross-cultural comparisons  All respondents’ scores have the same mean  All respondents’ scores have the same variance Stated importance

Proprietary and Confidential, Maritz Inc. © Rank Ordering  Disadvantages  Difficult to administer in phone surveys  Difficult to administer if attribute list is long  Non-parametric statistical tests for rank orders are relatively weak, so less discriminating than even importance ratings  Recommendation: Do not use Stated importance

Proprietary and Confidential, Maritz Inc. © Q-Sort  Example: If 17 attributes are to be evaluated, respondent is instructed to sort them so that–  1 is in a pile for the most important attribute  3 are in a pile for the next most important attributes  1 is in a pile for the least important attribute  3 are in a pile for the next least important attributes  9 are implicitly sorted into the pile of middle importance  Resulting distribution is 1 : 3 : 9 : 3 : 1 Stated importance

Proprietary and Confidential, Maritz Inc. © Q-Sort  Advantages  Forced distribution standardizes responses, making this a viable technique for cross-cultural comparisons  Discriminating  Works in face-to-face, mail and Web surveys  Disadvantages  Will not work in phone interviews  Time consuming task if the number of attributes is large  Recommendation: May use if there are not too many attributes and data collection is not phone Stated importance

Proprietary and Confidential, Maritz Inc. © Constant Sum Allocation  Example: Please assign 11 points to these attributes according to how important they are to you when choosing a. You can assign some, none, or all 11 points to a given attribute, as long as the total number of points you assign is 11. Fast service_____ Wide variety_____ Reliability_____ Ease of use_____ Package color_____ Country of origin_____ TOTAL 11 Stated importance

Proprietary and Confidential, Maritz Inc. © Constant Sum Allocation  Advantages  Ratio measurement of importance  Discriminating – trade-off prevents all attributes from being important  Disadvantages  Difficult to do in telephone interviews unless attribute list is very short  Difficult to administer at all if attribute list is long  Unknown scale use heterogeneity  Unknown ability to standardize cross-cultural studies  Recommendation: Perhaps use with short attribute lists Stated importance

Proprietary and Confidential, Maritz Inc. © Method of Paired Comparisons  From Thurstone (1927)  Updated design theory by David (1988)  Updated analysis via hierarchical Bayesian analysis  Ask attributes two at a time, forcing respondent to choose which is more important  Ask 1.5 times as many pairs as there are attributes Stated importance

Proprietary and Confidential, Maritz Inc. © Method of Paired Comparisons  Example:  Is ease of use or reliability more important when choosing a ?  Is reliability or package color more important when choosing a ?  Is variety or fast service more important when choosing a ? Stated importance

Proprietary and Confidential, Maritz Inc. © Method of Paired Comparisons  Advantages  Easy to administer in any survey modality  VERY discriminating  Automatically standardized for cross-cultural comparisons (no scale-use bias)  HB analysis produces individual level importances  Ideal for input to needs-based segmentation  If well balanced (all attributes occur equally often with each other) analysis is simple: importance is proportional to percentage of times an attribute is chosen when it is available Stated importance

Proprietary and Confidential, Maritz Inc. © Method of Paired Comparisons  Disadvantages  A paired comparison question takes about 50% longer for respondent to answer in a phone survey, so that a 2 minute importance rating battery becomes a 3 minute paired comparison battery, all else being equal  Recommendation: Good method, use when possible Stated importance

Proprietary and Confidential, Maritz Inc. © Maximum Difference Scaling  Multiple choice extension of MPC  3+ attributes per question; respondent picks most and least important  Example: Which of the following is least important when you buy a widget and which is most important? LeastMost [ ]Ease of use[ ] [ ]Country of origin[ ] [ ] Package color[ ] [ ]Reliability[ ] Stated importance

Proprietary and Confidential, Maritz Inc. © Maximum Difference Scaling  Advantages  Handles more attributes with fewer questions than MPC  HB analysis produces individual level importances  Even MORE discriminating than MPC  Automatically standardized for cross-cultural comparisons  Ideal for input to needs-based segmentation  If well balanced, analysis is simple: importance is the log of the number of times chosen divided by the number of times available Stated importance

Proprietary and Confidential, Maritz Inc. © Maximum Difference Scaling  Disadvantages  Requires visual presentation of stimuli (i.e. paper and pencil or Web survey)  May require analysis to produce importances  Recommendation: Good method, use when possible Stated importance

Proprietary and Confidential, Maritz Inc. © Stated Importance Decision Tree Stated importance

Proprietary and Confidential, Maritz Inc. © Measuring Attribute Importance  Introduction  Stated Importance Methods  Derived Importance Methods  Summary Derived importance

Proprietary and Confidential, Maritz Inc. © Objectives  Following this section, you will be able to  Identify four types of derived importance models, in two broad classes  Use a decision tree to settle on a derived importance model for your project  Identify the strengths and weaknesses of the various kinds of derived importance measurement Derived importance

Proprietary and Confidential, Maritz Inc. © Overview - Derived Importance Measures  Respondent evaluates a brand/product using  A rating (satisfaction, performance, liking, purchase intent)  A share (market share, share of preference, an allocation)  A choice of one brand/product/therapy over others  Respondent rates product on multiple attributes  Predictive statistical model relates attributes to the evaluation  Model yields coefficients as proxies for importance  Correlation and choice-based methods Derived importance

Proprietary and Confidential, Maritz Inc. © Measuring Attribute Importance  Introduction  Stated Importance Methods  Derived Importance Methods  Correlation-based Methods  Correlation  Regression  True Driver Analysis  Choice-based Methods  Summary Derived importance

Proprietary and Confidential, Maritz Inc. © Variance  Correlation-based methods (correlation, regression, True Driver Analysis) depend on variance patterns  Importance just means “shared variance with the dependent variable”  If an attribute doesn’t share variance with a dependent variable, it can’t be important  Attributes which do not vary, or that vary only randomly, cannot share variance with the dependent variable, and so cannot be important  e.g. ‘cost of entry’ attributes like “4 wheels on a car” or “airline safety”

Proprietary and Confidential, Maritz Inc. © Correlation  Description: Bivariate correlations of evaluation with each attribute in turn  Advantages  Easy to do  Coefficients are unaffected by multicollinearity  Disadvantages  Lack of statistical control because attributes are analyzed in isolation  Importance can be double (or triple or more) counted to the extent attributes are correlated with one another  No composition rule for coefficients, so model cannot support simulation  Scale use heterogeneity can spuriously inflate all correlations Derived importance

Proprietary and Confidential, Maritz Inc. © Correlation  What correlation does Derived importance

Proprietary and Confidential, Maritz Inc. © Multiple Regression  Description: All attributes are used simultaneously to predict the evaluation; coefficients or standardized coefficients are importances  Advantages  Accessible technique learned in school that clients are familiar with  Easy to do  Disadvantage - multicollinearity (intercorrelation of independent variables) is omnipresent in survey research and has a pernicious effect on regression coefficients  It can distort the size and even the sign of coefficients  Makes coefficients unstable Derived importance

Proprietary and Confidential, Maritz Inc. © Multiple Regression  What regression does Derived importance

Proprietary and Confidential, Maritz Inc. © Multiple Regression  Why multicollinearity is bad news for regression Derived importance

Proprietary and Confidential, Maritz Inc. © A Middle Ground  Kruskal came up with a way of computing importance by doing regression analysis and “averaging over orderings”  Theil added an information-theoretic framework for this averaging over orderings  This idea works so well to solve the shortcomings of both correlation and regression that we’ve built it into a family of techniques for importance measurement that we call True Driver Analysis Derived importance

Proprietary and Confidential, Maritz Inc. © True Driver Analysis (TDA)  Advantages  Immune to multicollinearity  Importances are...  Additive  Ratio scaled  Intrinsically meaningful  Disadvantages  Complex programming Derived importance

Proprietary and Confidential, Maritz Inc. © Measuring Attribute Importance  Introduction  Stated Importance Methods  Derived Importance Methods  Correlation-based Methods  Choice-based Methods  MNL  Summary Derived importance

Proprietary and Confidential, Maritz Inc. © Logit vs Regression  In regression, the unit of analysis is a brand and its ratings: Obs.OverallAtt 1Att 2Att 3Att

Proprietary and Confidential, Maritz Inc. © Logit vs Regression  In MNL, the unit of analysis is a choice, so we collect multiple brands’ ratings: Obs.ChosenAtt 1Att 2Att 3Att

Proprietary and Confidential, Maritz Inc. © Multinomial Logit (MNL)  Description  Evaluation is brand preferred over others (choice or share)  Models this relative preference as a function of attributes of all competing brands  Between-brand differences drive the model (as they drive choice in the real world)  MNL coefficients are significant when differences between brand ratings relate to brand choice – i.e. it measures determinance Derived importance

Proprietary and Confidential, Maritz Inc. © Multinomial Logit (MNL)  Advantages  Model built from between-brand differences in attribute ratings, so scale use heterogeneity is not a problem  For the same reason, less likely to be affected by multicollinearity  Explicitly includes competitive context  Specifically answers the oft-asked question “Why do customers choose one product over another” (or “competitors’ products over mine,” or “mine over theirs”) Derived importance

Proprietary and Confidential, Maritz Inc. © Multinomial Logit (MNL)  Disadvantages  Because we need to ask every respondent about several brands’ attributes, questionnaire can get long if there’s a lot more in it Derived importance

Proprietary and Confidential, Maritz Inc. © Derived Importance Decision Tree

Proprietary and Confidential, Maritz Inc. © Measuring Attribute Importance  Introduction  Stated Importance Methods  Derived Importance Methods  Summary Derived importance

Proprietary and Confidential, Maritz Inc. © Stated Vs. Derived Importance  Fairly often, stated and derived measures give conflicting views of what is “important”

Proprietary and Confidential, Maritz Inc. © Back to Myers and Alpert  One reason stated and derived methods may not agree is that they are not even measuring the same thing  Open-end measures confound the measurement of salience and importance  Other stated methods may measure importance, though there is good reason to believe attribute importance ratings do not do so very well  MNL measures determinance, NOT just importance  Regression-based derived importance methods measure a cross-sectional version of determinance

Proprietary and Confidential, Maritz Inc. © Stated Vs. Derived Importance  Among stated importance methods—  Simpler approaches, like importance ratings and rankings, have serious shortcomings  More complex methods (MPC, MaxDiff) overcome at least some of the shortcomings  MPC for phone surveys  MaxDiff for mail, in-person or Web-based surveys

Proprietary and Confidential, Maritz Inc. © Stated Vs. Derived Importance  Among derived importance methods—  Simple approaches (correlations, regression) just aren’t good enough  More complex methods (TDA, MNL) fix a lot of the shortcomings  TDA for customer satisfaction and concept testing  MNL for brand choice/brand preference

Proprietary and Confidential, Maritz Inc. © Q & A

Proprietary and Confidential, Maritz Inc. © Thanks for your time and participation today! To replay this webcast (available September 10): go to For copies of today’s presentation: or (877) 4 MARITZ To contact today’s speaker: Keith Chrzan Questions for AMA: Lynette Rowlands