1 1 Web and Paper Questionnaires seen from the Business Respondent’s Perspective Gustav Haraldsen, Statistics Norway Jacqui Jones, UK Office of National Statistics Third International Conference on Establishment surveys. Montréal 2007

2 Outline A quality driven approach to response burden The effect of survey design, external conditions and personal characteristics on perceived response burden The impact of perceived response burden on data collection and data quality (The perceived response burden of web versus paper questionnaires)

3 Stakeholder(s) Survey organisation Survey Design PRB Respondent(s) Availability Motivation Competence Response quality Response environment Survey Requirements Questionnaire Length Structure Content Layout Procedure Contact mode(s) Persuasion strategies Management Confidentiality Sample Specification Frame Sampling procedure Size Release Origins and Effects of Perceived Response Burden in Establishment Surveys Context: Encoding/record formation Respondent selection/identification Priorities Revision

4 The quality driven approach Response burden is mainly a problem as long as it affects the response quality It is the respondents’ response burdens that matters Perceived response burden is a more relevant variable than completion time The response burden may be affected by external conditions or personal characteristics of the respondents as well as by the survey design. Perceived response burden questions should identify why questionnaires are burdensome

5 Core PRB Questions Do you think it was quick or time consuming to collect the necessary information to complete the questionnaire? Very quick Quite quick Neither quick nor time consumingThese were asked to pick the main reason(s) from a list or specify other reason(s) Quite time consuming Very time consuming Did you find it easy or burdensome to complete the questionnaire? Very easy Quite easy Neither easy nor burdensomeThese were asked to pick the main reason(s) from a list or specify other reason(s) Quite burdensome Very burdensome

6 Sources of perceived response burden Survey Design Factors –Characteristics of 6 web questionnaires directed to different industries –(The time it took to fill in the questionnaire) External conditions –Size: The number of employees –Information sources: Documentation problems that made the information collection time consuming –Division of labour: The number of persons who took part in the response process Personal Characteristics –Availability: How easy or difficult it was to set aside time for the questionnaire –Motivation: How useful for the business and society the respondent felt that statistics produced from the data were –Competence: If he/she was a first time or experienced respondent

Web Response and PRB Response for Annual Business Surveys Web responsePRB response Hotels and restaurants31 %43 % Transport & communication 31 %40 % Sea transport41 %30 % Service industries33 %45 % Domestic trade33 %44 % Construction31 %43 %

8 Perceptions of the Information Collection and Completion of the 2006 Questionnaires IndexMinutes 0,01 0,

9 Sources of Time Consuming Preparations Different sources Need for help Waiting for information Many questionnaires Other reasons

10 Sources of Response Burdens Mismatch Calculations Response categories No. of questions Terms Layout Other E-technical E-usability

11 Information collection and Perceived response burden index: Industries and completion time IC-indexPRB - index Construction Sea transport0.04(0.02) Transport and communication Hotels & restaurants Domestic trade Service industry Completion time0 – 25 minutes0,32 26 – 50 minutes0,11 51 minutes +-0,16

12 PRB-index: External conditions PRB- index Size of0 – 9 employees0,11 establishment10 – 19 employees(0,13) 20 – 49 employees0,06 50 or more employees0,10 DocumentationNone0,15 problems1(-0,14) 2((-0,23)) 3-0,45 No of10,15 Respondents2 or more((-0,01))

13 PRB-index: Personal characteristics PRB- index Available timeVery easy to find0,68 Quite easy to find0,41 Neither easy nor difficult0,06 Quite difficult-0,09 Very difficult-0,37 Perception ofVery useful((0,22)) usefulnessQuite useful0,30 Neither useful nor useless0,18 Quite useless0,09 Very useless-0,11 ExperienceNo0,04 Yes0,11

14 The effect of survey design, external conditions and personal characteristics on PRB Calculations and mismatch between questions and available information are general problems in business surveys Sufficient time available for the questionnaire is essential for the perceived response burden. The number of questions was probably the most important difference between questionnaires Poor layout and usability problems were often mentioned by those who felt that the questionnaire was very burdensome The quality of documentation systems is probably the most important difference between industries The perception of uselessness goes together with perceived response burden

15 Web Response Patterns for Construction and Service Industry Questionnaires Construction Service

16 Questionnaires corrected Construction80,9 % Sea transport53,3 % Transport and communication63,3 % Hotels & restaurants79,7 % Domestic trade70,1 % Service industry33,4 %

17 The impact of perceived response burden on data collection and data quality Generally in business surveys.. –the data collection time is very long and –the number of questionnaires that are corrected is very high When there is a pronounced difference in perceived response burden between questionnaires, there also seems to be a difference in… –How quickly the data are received –The number of questionnaires that need to be corrected Problems with collecting the information asked for may also have an effect on these quality indicators

18 Perceived response burden of the paper and web versions of the Annual Manufacturing Industry surveys (2004 and 2005) PRB- index 0,01 -0,08 0,39 0,32

19 Sources of response burden in paper and web versions of the Manufacturing Industry Surveys One businessMulti business PaperWebDiffPaperWebDiff Mismatch Calculations Terms No of questions Response categories Layout E-technical/usability Other SUM

20 (The perceived response burden of web versus paper questionnaires) When the Manufacturing Industry Survey changed from paper to web… –The perceived response burden apparently went down –While the number of problems reported by those who felt the questionnaire burdensome seemed to go up A quote for reflection: “It takes 100 men to do as much harm as a computer can make within 1/100 of a second”