The Effect of Questionnaire Length on Response Rates and Level of Estimates in the German Job Vacancy Survey Hans Kiesl Institute for Employment Research (IAB), Germany Q2008 – European Conference on Quality in Official Statistics Rome July 9-11, 2008

2 Background Regulation (EC) No. 453/2008 of the European Parliament and of the Council of 23 April 2008 on quarterly statistics on Community job vacancies Member states have to provide  quarterly data on job vacancies (broken down to NACE section level)  quality reports In Germany, the data will be provided by the IAB.

3 Background (2) Information on job vacancies in Germany  Business units might report job vacancies to the Federal Employment Agency  Federal Employment Agency publishes monthly statistics on number of registered job vacancies (by NACE-sector)  Since 1989, IAB conducts a yearly (4 th quarter) sample survey among business units to estimate number of job vacancies (registered or not) and to get additional information (e.g. about recruiting strategies)  Mail questionnaire (8 pages in length); voluntary  CATI interviews in quarters 1 - 3

4 Basic sampling design  stratified SRS; 16 sectors  7 size classes x West/East  sampling rates and sample sizes in different size classes (IV/2006): WestEast ratesample sizeratesample size < 100.9% % % % % % % % % % % %395 >= % %193 total2.4% %32305

5 Highly skewed distribution of job vacancies

% of Zeros:

7 Problem: extremely low response rates

8 Reasons for non-response After fall 2007 survey, CATI sub-sample of non-respondents to find out main reasons for non-response  sample of 1292 business units  6% no/wrong telephone number or late respondents  15% not willing to respond  79% respondents; their reasons for non-response in job vac. survey:  54%: no time; too much work (72% for largest units)  20%: no job vacancies (25% for smallest units)  4%: no relevant topic (44% for smallest units)  7%: take part in surveys only if mandatory  7%: never take part in surveys

9 Impact of length of questionnaire (1)  During the 4 th quarter of 2006 (at the same time of the regular survey with 8 pages) a separate survey was conducted (1 page, basic infos, e.g. number of job vacancies; 9,450 sampling units).  Research questions:  Has length of questionnaire significant impact on response rates? (Prediction: yes)  If so, do different response rates lead to different estimates of number of job vacancies? (Prediction: yes)

10 Impact of length of questionnaire (2)  Response rates by size of business units: 8 pages1 pageabs. diff.rel. diff. < %27.3% + 6.3%+ 29.9% %30.7%+ 13.7%+ 80.1% %28.2% + 9.3%+ 49.1% %28.6% + 5.4%+ 23.2% %27.1% + 9.0%+ 49.9% %28.5%+ 10.1%+ 54.9% >= %

11 Impact of length of questionnaire (3)  Response rates by sector: 8 pages1 pageabs. diff.rel. diff. Agriculture, farming, forestry, fishing24.8%36.5%+ 11.6%+ 46.9% Manufacturing19.6%29.8%+ 10.2%+ 52.3% Construction21.5%28.8%+ 7.3%+ 34.1% Sale, trade, hotels and restaurants, transport, communications14.9%23.3%+ 8.4%+ 56.1% Financial intermediation, insurance, renting and business activities17.3%24.9%+ 7.6%+ 44.0% Public administration, education, health, social work, other service activities23.0%

12 Weighting and estimation  Calibrate Horvitz-Thompson-estimator to totals from auxiliary data:  sampling frame out of date (> 1 year)  up-to-date estimates on number of units by size and by sector (no cross-classification)  up-to-date estimates on number of employees by size and sector (no cross-classification)  registered number of job vacancies by sector  Linear GREG, using CLAN (Statistics Sweden)  Results quite robust with regard to underlying model e.g. constant variance or variance proportional to (1 + number of registered vacancies)

13 Impact of length of questionnaire on estimated number of job vacancies (1) 8 pages estimated standard errorestimated cv 1 page West 791,00027,0003.4% 843,00051,0006.0% East 212,00010,0004.7% 229,00032, % Germany1,003,00030,0003.0% 1,072,00060,0005.6% (Note: differences to officially published results due to slightly different weighting)

14 Impact of length of questionnaire on estimated number of job vacancies (2)

15 Impact of length of questionnaire on estimated number of job vacancies (2)

16 Impact of length of questionnaire on estimated number of job vacancies (3) 8 pages1 page (higher response) weighting withoutnumber of registered vacancies 211, ,000 registered jobnumber of vacancies 529, ,000 vacanciesshare of registered vacancies40%38% number of units with vacancies 166, ,000 weighting with registered jobnumber of registered vacancies465,000 vacanciesnumber of vacancies 791, ,000 share of registered vacancies59%55% number of units with vacancies 172, ,000

17 Conclusions  Length of questionnaire has considerable effect on response rates (as expected).  Change in response rates seem to have only little effect on main survey estimates (job vacancies) (not as expected).  Indication that non-response bias might be only modest.

Thank you very much for your attention! Q2008 – European Conference on Quality in Official Statistics Rome July 9-11, 2008