Q Rome Italy, July The Mysterious 36% Difference: Studies on a Measurement Error Anette Björnram, Boris Lorenc, Andreas Persson, Klas Wibell Statistics Sweden

Q Rome Italy, July Agenda Context Two measurement methods Results Hypotheses about the difference A methodological study to understand the difference Tentative conclusions and further questions

Q Rome Italy, July Context A customer aimed to measure aspects of late payments of invoices issued by small & middle-sized enterprises (SME’s) in Sweden to other enterprises, e.g. –proportion of late payments –length of delay (in days) –proportion of late payments paid more than 10 days and more than 30 days after the due-in date –etc.

Q Rome Italy, July Methods Study A: posing retrospective questions to person(s) responsible for bookkeeping in the sampled SME’s Study B: sampling invoices issued by the sampled SME’s within the reference period, recording relevant dates and events about these invoices, and calculating the desired information from these data

Q Rome Italy, July Methods (cont’d) Study A – example questions

Q Rome Italy, July Methods (cont’d) Study A – example questions (cont’d)

Q Rome Italy, July Methods (cont’d) Study A – example questions (cont’d)

Q Rome Italy, July Methods (cont’d) Study A – Potential response issues –complex response process: determining the reference period having an account of the number of issued invoices in this period having an account of the number of late paid invoices in this period having an account of the length of delay (in day) of the late paid invoices in this period building the desired averages performing the above by customer type breakdown –instead, the respondents might provide their impressions as answers (satisficing)

Q Rome Italy, July Methods (cont’d) Study B – the questionnaire

Q Rome Italy, July Methods (cont’d) Study B – Potential issues –possibly complex form –a balance between a complex sampling plan and a sub-optimal sampling plan to sample invoices –sampling variation: small number of observations (due to response burden considerations) –insufficient control of the data collection procedure (a self-administered mode) deviations regarding the sampling procedure deviations caused by data unavailability

Q Rome Italy, July Results

Q Rome Italy, July Results (cont’d) No general reason to suspect the difference between the measurement methods A and B –instead of satisficing, the respondents by Method A may have had “running tallies” directly available But, something happened with the item Proportion invoices paid late (a key variable for the customer) Given the issues associated with the two methods, several hypotheses may be considered Thus, a methodological follow-up performed

Q Rome Italy, July Hypotheses

Q Rome Italy, July Hypotheses (cont’d) Probed in a lengthy qualitative interview Those respondents chosen with the largest difference on Item 4 (proportion delayed payments) between Study A and Study B (range of 70-90% difference) 20 respondents chosen initially, data collection suspended after 12 respondents as a clear pattern emerged

Q Rome Italy, July Difference Explained In summary, H8(H1) strongly supported H8: The respondents do not consider short delays as delays H1: The question not comprehended correctly in Study A H2 and H4 got some support H2: Access to data difficult or impossible in Study A H4: There is a time lag in the availability of data in Study A The other hypotheses received weak or no support

Q Rome Italy, July Difference Explained (cont’d) The respondents do not consider short delays as delays (H8/H1) –there is a distinction between the implied definition of ‘delay’ by the respondents and our customer There is a buffer of maybe one week, so up to one week is the same as the due-in date –in some cases related to some act When I feel that I must do something then at least one week must have passed, if it is shorter than that then I don’t care If 4-5 days have passed and the money hasn’t arrived, then I’d call and ask if maybe the invoice had got lost on the way

Q Rome Italy, July Difference Explained (cont’d) –sometimes the question misinterpreted (H1/H3) I thought generally, how many invoices we issue in a year and how many reminders we send out –technical delays (bank, weekends, post delivery) –there is a cost in sending out reminders –even routines do not always alarm promptly (H4) As I don’t get [i.e. see] those [payments] that are between 1 and 4 days late, so of course that I have underestimated the proportion of late payments [one of the two SME’s in the study using an automated routine]  Thus, a “Gray Zone” exists

Q Rome Italy, July Difference Explained (cont’d) But, Item 4 question difficult (H2) –especially customer size Our system does not support provision of such data, so it was mostly feeling, impression. –impression provided rather than data accessed Difficult, as I don’t know the size of the companies I’m selling to.

Q Rome Italy, July Difference Explained (cont’d) A tentative model for gray zone (NB. Very approximate point estimates!)

Q Rome Italy, July Difference Explained (cont’d) Further on ‘Gray Zone’ (based on data from Study B) –Suppose a ‘Gray Zone’ of length 8 days exists –What would the proportion of invoices paid late be if the credit period is extended by the length of Gray Zone (i.e. 8 days) for all the 1550 invoices in Study B?

Q Rome Italy, July Difference Explained (cont’d) NB: the length of delay (given it is a delay) expected to be the same irrespective of whether there is no Gray Zone (as in Study B) or there is one (as in Study A), if the delays follow an exponential distribution memoryless property of the exponential distribution thus, reasonable that there is no difference between Study A and Study B on Item 7 Answer: 15 % close to the value in Study A (there: 14%)

Q Rome Italy, July Tentative conclusions Estimation not necessarily wrong: if information important, an approximate running tally may be kept –further empirical confirmation desirable, including accounting of cognitive mechanisms that provide for this ability

Q Rome Italy, July Tentative conclusions (cont’d) But, the real research question (the length of delays that SME’s are exposed to by their enterprise customers) is difficult to measure by any method –context-specific interpretation of what constitutes a delay –source data changed (e.g. if a customer demands longer credit period) In that sense, it is a sensitive question in an establishment survey

Q Rome Italy, July Tentative conclusions (cont’d) – issues How to study such questions? –quantitatively: the numbers do not reflect the intended phenomenon –quantitatively: the results cannot be expressed in numbers How to recognize such questions?

Q Rome Italy, July Thank You Anette Björnram Boris Lorenc Andreas Persson Klas Wibell