1. Nonresponse Bias in the 2003 Survey of Small Business Finances Y. Michael Yang, NORC at the University of Chicago Traci Mach, Federal Reserve Board Lieu Hazelwood, Federal Reserve Board

2. Disclaimer The views expressed herein are those of the authors. They do not necessarily reflect the opinions of the Federal Reserve Board or its staff.

3. Overview Bias due to Nonresponse Data Source: 2003 Survey of Small Business Finances Analytical Methods for Assessing Bias Results Discussions

4. Bias Due to Nonresponse (1) Deterministic model Probabilistic model

5. Bias Due to Nonresponse (2) Implications –Nonresponse bias is variable-specific; –Nonresponse does not necessarily lead to bias; –Nonresponse bias increases as response rate decreases; –Nonresponse bias depends on the response mechanism; –Key is to examine the correlation between response propensity and the survey variable;

6. 2003 SSBF (1) Target population –U.S. for-profit small businesses Frame –Dun’s Market Identifier (DMI) Sample design –Stratified systematic sampling –Screener nonresponse subsampling –Main interview nonresponse subsampling Data collection –Telephone Interview

7. 2003 SSBF (2)

8. 2003 SSBF (3) Scope of analysis –Eligible screener respondents (n=6,520) –Focus on bias due to main interview nonresponse –Adjustments for screener nonresponse and subsampling have been incorporated in the base weight

9. Analytical Methods Comparing response rates across subgroups; Comparing early responders and late responders on survey variables; Comparing first phase responders and follow- up responders on survey variables; Comparing unadjusted and adjusted estimates; Estimate nonresponse bias analytically

10. Main Interview Outcome Rates

11. Outcome Rates by Business Size: Unweighted

12. Outcome Rates by Business Size: Weighted

13. Outcome Rates by Urbanicity: Unweighted

14. Outcome Rates by Urbanicity: Weighted

15. Early ( =10 calls) Completes: Unweighted (1)

16. Early ( =10 calls) Completes: Unweighted (2)

17. Early ( =10 calls) Completes: Unweighted (3)

18. Early ( =10 calls) Completes: Weighted (1)

19. Early ( =10 calls) Completes: Weighted (2)

20. Early ( =10 calls) Completes: Weighted (3)

21. Early ( =19 calls) Completes: Unweighted (1)

22. Early ( =19 calls) Completes: Unweighted (2)

23. Early ( =19 calls) Completes: Unweighted (3)

24. Early ( =19 calls) Completes: Weighted (1)

25. Early ( =19 calls) Completes: Weighted (2)

26. Early ( =19 calls) Completes: Weighted (3)

27. Main Sample Vs. Follow-up Subsample Completes: Unweighted (1)

28. Main Sample Vs. Follow-up Subsample Completes: Unweighted (2)

29. Main Sample Vs. Follow-up Subsample Completes: Unweighted (3)

30. Main Sample Vs. Follow-up Subsample Completes: Weighted (1)

31. Main Sample Vs. Follow-up Subsample Completes: Weighted (2)

32. Main Sample Vs. Follow-up Subsample Completes: Weighted (3)

33. Indirect Estimate of Bias (unadjusted estimate)-(adjusted estimate) Measures the amount of nonresponse adjustment; Measures nonresponse bias if the adjusted estimate is unbiased;

34. Indirect Estimates of Bias CharacteristicsEstimated BiasCharacteristicsEstimated Bias Native American0.14Equipment loans-1.27 White0.07Capital leases-0.18 White non-Hispanic0.24Other loans-0.38 Asian-0.13Line of credit-0.65 Asian or Pacific Islander-0.1Transaction service-0.57 Black0.04Cash management service-0.51 Pacific Islander0.03Credit related service-0.53 Hispanic-0.12Trust service-0.71 Owner managed0.77Brokerage service-0.16 Average owners education0.02Card processing service-0.33 Firm age0.07Total sales-158570.05 Minority or Hispanic-0.1Profit-20781.18 Total employment-1.01Total Assets-77824.28 Mortgage loans0.07D&B credit percentile-0.96 Motor vehicle loans-1.02Total employees-1.31

35. Direct Estimate of Bias Estimate response propensity Compute the covariance between response propensity and survey variable; Estimate bias analytically

36. Direct Estimates of Bias (1) Survey VariableCovarianceEstimated BiasWeighted EstimateBias Ratio Native American0.010.021.411.4% White0.080.1690.820.2% White non-Hispanic0.150.3186.430.4% Asian-0.05-0.114.17-2.6% Asian or Pacific Islander-0.06-0.114.29-2.6% Black-0.04-0.073.63-1.9% Pacific Islander000.120.0% Hispanic-0.09-0.184.28-4.2% Owner managed0.150.2994.160.3% Average owners education0.01 4.610.2% Firm age-0.03-0.0514.3-0.3% Minority or Hispanic-0.17-0.3513.31-2.6% Total employment-1.91-3.858.73-44.1% Mortgage loans-0.11-0.2213-1.7% Motor vehicle loans-0.28-0.5725.33-2.3%

37. Direct Estimates of Bias Bias (2) Survey VariableCovarianceEstimated BiasWeighted EstimateBias Ratio Equipment loans-0.27-0.5410.59-5.1% Capital leases-0.21-0.428.9-4.7% Other loans-0.21-0.4110.08-4.1% Line of credit-0.53-1.0832.45-3.3% Transaction service-0.4-0.838.96-2.1% Cash management service-0.55-1.116.43-17.3% Credit related service-0.25-0.54.82-10.4% Trust service-0.45-0.9217.62-5.2% Brokerage service-0.05-0.115.45-2.0% Card processing service-0.16-0.3136.67-0.8% Total sales-331788.2-669119.94 1,177,572-56.8% Profit-31853.62-64239.45 175,965-36.5% Total Assets-156104.5-314817.2 535,133-58.8%

38. Direct vs. Indirect Bias Estimates

39. Discussion There is significant correlation between nonresponse and key survey variables –Larger and urban businesses have lower response rate; –Larger and urban businesses use more services, have more sales, assets, and profit Unadjusted estimates of the mean/total are biased downward for most variables Response propensity-based weighting appears to be effective, but only a portion of the bias is corrected Weighting method may be improved in future surveys

40. Contact Information Yang-michael@norc.org