1. Frederick C. Van Bennekom, Dr.B.A. Helping Clients Listen Better Survey Program Training, Development & Targeted Advice Customer Service Strategic Positioning 421 Main Street, Bolton, MA 01740  contact_us “at” GreatBrook.com +1 (978) 779-6312  www.greatbrook.com Survey Mode Impact Upon Responses and Net Promoter Scores Frederick C. Van Bennekom Northeastern University, Great Brook Sam Klaidman Middlesex Consulting

2. © Frederick C. Van Bennekom, Great Brook, Slide 2.com Summary q Surveys – heavily used gauge of customer centricity –Mixed mode used to increase response rates q Net Promoter Score (NPS) is common summary attitudinal indicator q However, businesses are unaware of how changes in survey practices can affect responses q Research findings on survey mode: –Telephone survey mode elicits higher scores on the response scale than web-form survey mode –NPS’s threshold effect amplify the mode effects

3. © Frederick C. Van Bennekom, Great Brook, Slide 3.com Previous Research – Non-Measurement Error q Survey Mode impacts: –Non-measurement error: Whether the person responds –Measurement error: How the person responds q Non-measurement error –Non-response bias Telephone mode garners higher response rates (Groves 1989, Nunley, 2013) –Composition bias Response rates will differ by demographic groups for each mode

4. © Frederick C. Van Bennekom, Great Brook, Slide 4.com Previous Research – Measurement Error q Measurement Error –Response effects are greater with telephone mode yielding higher, more positive scores Acquiescence – yes saying – (Bowling 2005) Social Desirability (Bowling 2005) Primacy & Recency (Christian et al. 2007, Bethlehem 2012, Kreuter 2008) –Scale truncation effect Our belief is that the telephone presentation of the scale with endpoint anchors as the only information guides leads to the tendency toward extreme responses

5. © Frederick C. Van Bennekom, Great Brook, Slide 5.com Previous Reasearch – Net Promoter Score q New statistic applied to the Likelihood to Recommend question –Reichheld (2003), based on Sambandam and Hausser (1998) q Reichheld’s argument: –Best summary measure to indicate likelihood of future profitability

6. © Frederick C. Van Bennekom, Great Brook, Slide 6.com Why the NPS Statistic? It’s one number  Rather than report both Top Box and Bottom Box People “get” percentages Highly responsive to changes  Real or artifacts of the survey process Provides focus to the low end of response scale However, highly controversial due to lack o f reproducibility (Mo r gan and Rego 2008)

7. © Frederick C. Van Bennekom, Great Brook, Slide 7.com Research Venue q Large, anonymous capital goods manufacturer with after-sales service q Transactional survey conducted in both telephone & web-form modes after service events –Web-form if email address captured by field –Telephone if otherwise q Three question survey –Likelihood of recommendation: NPS question –Overall satisfaction with XXX as a service provider –Overall satisfaction with the last visit q Access to December 2011 data

8. © Frederick C. Van Bennekom, Great Brook, Slide 8.com From a Research Design Perspective Good News, Bad News… q Bad news –Not a controlled experiment q Good news –Real company data used, not a convenience sample of college students

9. © Frederick C. Van Bennekom, Great Brook, Slide 9.com Survey Statistics Of 133 districts, 48 had no web-form responses These districts eliminated due to possibility of confounding factor Telephone survey scores tested between districts with and without email address. p =0.65 Item non-response led to Unusable Responses

10. © Frederick C. Van Bennekom, Great Brook, Slide 10.com Frequency Distribution – Recommendation Question

11. © Frederick C. Van Bennekom, Great Brook, Slide 11.com Frequency Distribution & Chi Square Test Results Phone1.960.150.010.831.012.83 email21.421.590.099.1011.0630.99 Chi square values

12. © Frederick C. Van Bennekom, Great Brook, Slide 12.com Additional Testing q Recommendation question: p = 5.09 E-16 q Overall satisfaction question: p = 2.29 E-16 q Visit satisfaction question: p = 0.0035 –Less difference in scores since high for both modes

13. © Frederick C. Van Bennekom, Great Brook, Slide 13.com Mode Measurement Error Combined with Composition Effect on NPS Survey Method Top Box Promoters (9s+10s) Passives (7s+8s) Bottom Box Detractors (0s to 6s) Net Score (Promoters – Detractors) Phone67.8%22.5% 9.7%58.1% Web-Form44.4%29.2%26.4%18.0% Combined Phone & Web65.8%23.1%11.1%54.7% Combined with 10% shift to Web63.5%23.7%12.8%50.7%

14. © Frederick C. Van Bennekom, Great Brook, Slide 14.com Implications for Surveying Practices q Develop mode adjustment factors –Complicated, expensive, not transparent q Change survey delivery practices –Fully anchored 10-point scales not practical for phone q Track modes separately –Simple, but how to summarize for a district q Discontinue mixed-mode surveying q Need to educate consumers of survey data about the many errors present, especially when comparing data across companies