1. Measuring nonresponse bias in a cross-country enterprise survey
Małgorzata Osiewicz
Co-authors: Katarzyna Bankowska
Sébastien Pérez-Duarte
Vienna, 4th June 2014
The opinions of the authors do not necessarily reflect the views of the ECB or the Eurosystem

2. Outline The Survey on the Access to Finance of Enterprises (SAFE)
Results for the measures of representativity in SAFE
Definition of R-indicators
Indicators across survey rounds and during the fieldwork
Indicators for successfully matched data with business register Amadeus
Conclusions and recommendations

3. Main characteristics of SAFE
ECB
European Commission
Sponsors
Quota: 30% each of micro, small and medium; 10% large firms
Part of the sample - rotational panel
Sample design
Surveys in March (ECB wave) and September (joint wave)
Results published in one month
Timeliness
7,500 for ECB waves - limited euro area
15,000 for joint waves - extended EU
Sample size
Since 2010, 11 largest euro area countries
Limited coverage EE, CY, LU, LV, MT, SI, SK
Representativeness
Telephone
Web – as from September 2014 wave
Mode
Sponsors
ECB (DG-S/DG-E)
EU Commission (DG-ENTR)
Frequency
Every 6 months: ECB part (only euro area countries)
Every 2 years, since September 2013 every year: full survey with Commission (whole EU)
Timeliness:
c.a. 1 month  2013H2 fieldwork finished 24 Mar, published 30 Apr 2014
Sample size (part of the sample: rotational panel)
Euro area (ECB): 7,500-8,000 firms
EU (ECB/Commission): 15,000 firms
Representativeness at
Firm size: micro / small / medium / large firms (control group)
Oversampling of small, medium and large in terms of # of firms
Geography: representativeness euro area + 11 largest countries;
 limited coverage other euro area countries (<3%)
Activity: industry, construction, trade, services 3

4. Outcome rates for SAFE from 8th to 10th survey round
Auxiliary information:
country
size
sector
panel
where I – Interview, P – Partial interview, R – Refusal, NC – Non-contact, O – Other contact (non-refusals), U – Unknown if firm, e- the estimated proportion of cases of unknown eligibility that are eligible.
Standard AAPOR definitions:
response rate 3: I/((I+P) + (R+NC+O)+ e*U),
cooperation rate 3: I/(I+P+R)
refusal rate 2: R/((I+P)+(R+NC+O) + e*U),
contact rate 2: ((I+P)+R+O) / ((I+P)+R+O+NC+ e*U)
e: (I+P+R+NC+O)/(I+P+R+NC+O+NE)

5. Definition of R-indicator
Following Schouten, B., Shlomo, N. & Skinner, C. (2011):
𝑅=1−2𝑆 𝜌 =1−2 1 𝑁−1 𝑖=1 𝑛 𝑑 𝑖 ( 𝜌 𝑖 − 𝜌 ) 2
where:
𝑑 𝑖 are the design weights,
𝜌 = 1 𝑁 𝑖=1 𝑛 𝑑 𝑖 𝜌 𝑖 is the weighted sample mean of the estimated response propensities
N is the size of the population

6. Partial R-indicators 𝑆 𝑏𝑒𝑡𝑤𝑒𝑒𝑛 2 𝜌 𝑆 𝑤𝑖𝑡ℎ𝑖𝑛 2 𝜌
Unconditional
Conditional
𝑆 2 𝜌 =
𝑆 𝑏𝑒𝑡𝑤𝑒𝑒𝑛 2 𝜌
𝑆 𝑤𝑖𝑡ℎ𝑖𝑛 2 𝜌
Variable level
𝑃 𝑈 ( 𝑋 𝑘 )= 1 𝑁 ℎ=1 𝐻 𝑛 ℎ ( 𝜌 ℎ − 𝜌 ) 2
𝑃 𝐶 ( 𝑋 𝑘 )= 1 𝑁 𝑙=1 𝐿 𝑖∈ 𝑈 𝑙 𝑑 𝑖 ( 𝜌 𝑖 − 𝜌 ) 2
Category level
𝑃 𝑈 ( 𝑋 𝑘 ,ℎ)= 𝑛 ℎ 𝑁 ( 𝜌 ℎ − 𝜌 )
𝑃 𝐶 ( 𝑋 𝑘 ,ℎ)= 1 𝑁 𝑙=1 𝐿 𝑖∈ 𝑈 𝑙 𝑑 𝑖 ∆ ℎ,𝑖 ( 𝜌 𝑖 − 𝜌 𝑙 ) 2
Notation
𝑋 𝑘 is a categorical variable with H categories and it is a component of the vector X
𝑛 ℎ = 𝑖=1 𝑛 𝑑 𝑖 ∆ ℎ,𝑖 is the weighted sample size in the category h, where ∆ ℎ,𝑖 is a 0-1 dummy variable for sample unit i being a member of stratum h
𝑈 𝑙 is a cell in the cross-classification of all model variables except 𝑋 𝑘

7. R-indicators and other associated information for the survey rounds 8 to 109 10
Response
Contact
Total sample
91,528
66,026
80,219
Response rate 3 / contact 2
13.5%
16.8%
11.1%
72.1%
68.9%
52.1%
R-indicator
0.841
0.717
0.849
0.805
0.783
0.697
Standard error
0.003
0.006
0.004
0.005
Average propensity
0.082
0.114
0.094
0.658
0.665
0.518
Maximum bias
0.973
1.245
0.807
0.148
0.164
0.293
Lower bound for R
0.451
0.365
0.417
0.051
0.056
0.001

8. Partial indicators for response in 8th survey round
Variable level:
Conditional and unconditional partial indicators
Category level:
Conditional partial indicators

9. Partial indicators for contact in 10th survey round
Variable level:
Conditional and unconditional partial indicators
Category level:
Conditional partial indicators

10. R-indicators for the response for each quartile of the fieldwork (8th survey round)
Up to 1st quartile
Up to 2nd quartile
Up to 3rd quartile
Full fieldwork
Total sample
91,528
R-indicator
0.917
0.865
0.845
0.841
Standard error
0.003
Average response propensity
0.023
0.048
0.069
0.082
Maximum bias
1.784
1.425
1.129
0.973
Lower bound for R
0.698
0.574
0.494
0.451

11. Matching of SAFE with Amadeus database
Company-level financial information, e.g. turnover, value added, loans outstanding
Matching preserving confidentiality of the sampled companies on tax id, company name, street, postcode, city and country
Successful matching rate:
Overall – 80% in the 8th survey round
Country - from 43% in GR to over 90% in BE, ES, FR, NL

12. R-indicators for the availability of information on loans, value added and turnover (8th survey round, respondents)
Loans
Value added
Turnover
Total sample
7,510
R-indicator*
0.744
0.805
0.741
Standard error
0.003
0.002
Average propensity
0.605
0.403
0.539
Maximum bias
0.211
0.242
0.240
Lower bound for R
0.022
0.019

13. Conclusions … and recommendations
In the SAFE sample, the country variation contributes mostly to the loss in representativity
In the Amadeus subsample also size class plays a role with the evident underrepresentation of the micro firms
… and recommendations
increase efforts to enhance the quality of the sample contact information
fully harmonise the use of the outcome codes across countries and interviewers
collect more detailed information from the fieldwork useful for the monitoring of the data collection, i.e. outcome codes for each attempt and possibly interviewers’ performance and experience.
monitor representativity of different modes (telephone, web).

14. Representativity of the sample frame with respect to the population
Further research
Representativity of the sample frame with respect to the population
Sensitivity analysis using different weighting schemes
Representativity for the quota samples

15. Annex

16. Distribution of unconditional contact propensities for categories of variable country in wave 8 and 10

17. Lower bound of the R-indicator 𝑅≥1−2 𝜌 (1− 𝜌 )
lower bound of the R-indicator (see [8], p.104) depends on the response rate: R≥1-2ρ(1-ρ).
Lower bound of the R-indicator 𝑅≥1−2 𝜌 (1− 𝜌 )
Maximal relative absolute relative bias 𝐵 𝑚 𝑋 = 1−𝑅 𝜌 2 𝜌 ≤1− 𝜌

18. Dissemination of results
ECB website
Press releases + summary reports
Aggregate tables with all variables / breakdowns
Questionnaire
Anonymised microdata provided to researchers
Confidentiality declaration

19. What else is done with SAFE?
Further information
Applications external financing
Terms & conditions loan financing
Expectations
Multi-dimensional analysis
Economic activity, firm size and age
Time series and cross country analysis
Composite indicator
Composite indicator SME financing sources