© Fred Van Bennekom, Great Brook, 2001Page 1 The Art & Science of Designing a Survey Instrument Frederick C. Van Bennekom, Dr.B.A. Great Brook Consulting Enhancing Organizational Improvement Through Customer Feedback 421 Main Street  Bolton, MA (978)  (877) GreatBr Toll Free 

© Fred Van Bennekom, Great Brook, 2001Page 2 Art versus Science q None – ignorance q Know a good outcome from bad q Know the characteristics of a quality outcome q Prioritization of these quality characteristics q Know the variables that lead to these outcomes q Know the impact of individual variables q Know the interaction effects among variables q Able to measure the variables q Able to control process to achieve quality outcomes – repeatedly & consistently Art Science

© Fred Van Bennekom, Great Brook, 2001Page 3 Art and Science? q The Art –Crafting the wording of the questions q The Science –The design process –Design of scales –Not to mention the survey administration

© Fred Van Bennekom, Great Brook, 2001Page 4 x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x r r r r r r r x x x x xx x xx r Population n Surveying a Sample is More Efficient Than a Full Census Sample r r r r r r r r What is a Survey? Design & Administer Instrument to a Sample Generalize Results to the Population Instrument Validity + Administration Accuracy = Reliability

© Fred Van Bennekom, Great Brook, 2001Page 5 The Role of Surveying in Achieving Loyal Behavior Value added chain Design... Replication… Sales… Service... Good service delivery Continuous improvement Problem solicitation Effective problem handling

© Fred Van Bennekom, Great Brook, 2001Page 6 A Rigorous Instrument Design Process 1) Interview management 2) Identify questions to ask 3) Draft survey instrument 4) Review by project team 6) Conduct pilot 7) Redraft & finalize instrument 5) Revision Iterations Science

© Fred Van Bennekom, Great Brook, 2001Page 7 Identifying Questions to Ask q Attributes of Service Delivery –that need to be understood & tracked q Attitudinal Outcomes –driven by perceptions of service delivery performance q Demographic Segmentations –for data analysis

© Fred Van Bennekom, Great Brook, 2001Page 8 Identify the Attributes q Draw a Service Blueprint –Process flow diagram –Highlights the Moments of Truth = where we “touch” the customer q Review complaint data, conduct focus groups, interviews, or other critical incident studies –What are critical service attributes? –What are customers major concerns? Art

© Fred Van Bennekom, Great Brook, 2001Page 9 Classify the Attributes – Service Quality Dimensions q Reliability: Delivering on promises q Responsiveness: Being willing to help q Assurance: Inspiring trust and confidence q Empathy: Treating customers as individuals q Tangibles: Representing the service physically A useful framework for thinking about the instrument design – and analyzing the data Science

© Fred Van Bennekom, Great Brook, 2001Page 10 Instrument Design – Attitudinal Outcomes q Perception of service delivery leads to attitudes –Likelihood of repurchase –Willingness to provide reference –Overall satisfaction –Any others? q Use of attitudinal measures –Summary measure for the survey –Dependent variable for regression tests Link attributes to true behavioral outcomes if data are available

© Fred Van Bennekom, Great Brook, 2001Page 11 Drafting the Survey Instrument q Overall form of the survey instrument q Issues with the construction of the questions q Selecting a scale q Question formats q Question sequencing

© Fred Van Bennekom, Great Brook, 2001Page 12 Overall Form of the Instrument q Pre-Administration Announcement Letter –Letter or from senior executive q Survey Introduction –Set the mental state & be consistent –Define critical terms q Initiation - First Questions –Engage the respondent –Get the respondent thinking q Instructions –Even if it seems silly... With each contact, motivate the respondent! Science

© Fred Van Bennekom, Great Brook, 2001Page 13 The Need for Instructions – (Need we say more?)

© Fred Van Bennekom, Great Brook, 2001Page 14 Overall Form of the Instrument q Grouping strategies –By topic, by scale, by chronology q Conditional branching – “Skip & Hit” q Routine – a response rut –Long series of questions that read in a rhythm –Respondents just give the same answer q Fatigue – caused by long list of choices –Leads to choosing first or last item –Especially important for telephone surveys Art

© Fred Van Bennekom, Great Brook, 2001Page 15 Issues with the Construction of Questions q Critical Criterion: Common Interpretation Otherwise... You’re Asking the Respondents Different Questions Focus BrevityClarity 3 Key Attributes o Control for Instrumentation Bias Response Bias

© Fred Van Bennekom, Great Brook, 2001Page 16 Avoiding Instrumentation Bias q Clearly Stated Criteria for Evaluation q Question Must Apply to Respondent q Examples Should Not Lead Response q Reasonable Recall Expectations q Unambiguous Word Choice q Ask One Question at a Time q Don’t Ask Leading or Loaded Questions 4 Bias Introduced by the Survey Instrument

© Fred Van Bennekom, Great Brook, 2001Page 17 Scale Anchoring Options q Fully Anchored Extremely Extremely SatisfiedSatisfiedUndecidedDissatisfied Dissatisfied Is this an interval scale or just an ordinal scale?? q Endpoint Anchored Extremely Extremely Satisfied Dissatisfied 12345

© Fred Van Bennekom, Great Brook, 2001Page 18 How Should I Solicit a Response? q Unstructured –Free-form or open-ended response – “Please describe...” –“Is there anything else...” q Structured –Response on pre-determined list or scale –“Check all that apply...” –“Please rate...” 4 Remember the Objective of a Survey: ò Maximize Information Gained while... ò Minimizing Respondent Burden

© Fred Van Bennekom, Great Brook, 2001Page 19 Question Formats & Types

© Fred Van Bennekom, Great Brook, 2001Page 20 Question Format: Unstructured q Advantages – Response Not Constrained to Predetermined Categories –May Uncover Unexpected Answers q Disadvantages –Very Long to Complete Respondent Burden Cost to Administer –Textual Data Difficult to Analyze and Summarize 4 Free-Form or Open-Ended Response

© Fred Van Bennekom, Great Brook, 2001Page 21 Question Format: Structured q Advantages –Clearer responses –Easy to summarize & analyze –Easy to administer q Disadvantages –Limits responses –May bias responses –Requires more investment in question design 4 Coded Response Multiple Choice & Scaled Data

© Fred Van Bennekom, Great Brook, 2001Page 22 Interval Rating Scales – Elements Listed below are several statements. Please indicate your agreement with each by selecting a number from 1 to 5 where 1 represents Strongly Disagree and 5 represents Strongly Agree. Strongly Strongly Disagree Agree I was on hold for a short time N/A12345 Question Item Scale Anchors Instructions

© Fred Van Bennekom, Great Brook, 2001Page 23 Clearly Stated Criteria for Evaluation Wrong q How would you rate the ability of the project team to define business requirements? Right q Compared to other projects done for you, how would you rate the ability of the project team to define business requirements?

© Fred Van Bennekom, Great Brook, 2001Page 24 Applicability to Respondent Wrong q How effective did you find the FAX-Back support system? Right q If you used the FAX-Back support system, how effective did you find it? 4Include a “not applicable” response choice 4Multiple NAs may lead to non-response. Use skip & hit.

© Fred Van Bennekom, Great Brook, 2001Page 25 Do Not Lead With Examples Wrong q What aspect of our service is most critical to you, for example, the speed of response? Right q What aspect of our service is most critical to you? 4Most critical with open-ended question format

© Fred Van Bennekom, Great Brook, 2001Page 26 Reasonable Recall Expectations Wrong q In your support calls over the past year, how many minutes was it before the phone was answered? Right q During the past three months, has the time for a support representative to answer the phone been reasonable?

© Fred Van Bennekom, Great Brook, 2001Page 27 Unambiguous Wording Wrong q In your last support call, was the response time reasonable? Right q Consider your last request for support. How reasonable was the time from when you called until you spoke with a support representative? 4Major source of construction flaws 4Avoid jargon

© Fred Van Bennekom, Great Brook, 2001Page 28 Examples of Ambiguous Phrasing –The ability of the help desk to resolve problems on the first try –The promptness with which you received the Service Engineer’s estimated time of arrival… –Satisfaction with the functionality of the equipment –Responsiveness of the Customer Support Personnel –Have you received service of consistent quality? –Was your call answered promptly?

© Fred Van Bennekom, Great Brook, 2001Page 29 Ask One Question at a Time Wrong q Was the staff technically competent and courteous? Right q Was the staff member who handled your issue technically competent? q Was the staff member who handled your issue courteous?

© Fred Van Bennekom, Great Brook, 2001Page 30 Avoid Loaded & Leading Wording Wrong q How did our interest in you, our customer, match your expectations? Right q To what extent did our concern for you match your expectations?

© Fred Van Bennekom, Great Brook, 2001Page 31 Thanks for Attending Any Questions??