How easily can omission of patients, or selection amongst poorly-reproducible measurements, create artificial correlations? Methods for detection and implications for observational research design in cardiology  Darrel P. Francis  International Journal of Cardiology  Volume 167, Issue 1, Pages 102-113 (July 2013) DOI: 10.1016/j.ijcard.2011.12.018 Copyright © 2011 Elsevier Ireland Ltd Terms and Conditions

Fig. 1 Examples of effect of preferential deletion of “unfavourable” patients on the scatterplot and correlation coefficient. Dark dots are patients retained for publication; faint dots are patients studied but deleted. Published study size increases from the lowest panels to the highest. Extent of deletion increases from none (left panel) to 50% deletion (right). These 12 scatterplots are one experimental run of 12 configurations: 3 published study sizes and 4 fractions of deletion. 999 other such experimental runs were carried out to calculate Table 2. International Journal of Cardiology 2013 167, 102-113DOI: (10.1016/j.ijcard.2011.12.018) Copyright © 2011 Elsevier Ireland Ltd Terms and Conditions

Fig. 2 Effect of preferential selection of most favourable amongst several measurements made in each patient. Dark dots are the values selected for publication. Faint dots are the other values measured but not chosen for publication. (They are numerous because, for example, “best of 3” means 3 possibilities for x and 3 for y, i.e. 9 possible pairings, of which 8 are not chosen). Study size increases from the lowest panels to the highest. Extent of measurement selection increases from none (left panel) to “best of 4” (right). The block of scatterplots in this figure shows one experimental run of 12 configurations. 999 other such experimental runs were carried out. International Journal of Cardiology 2013 167, 102-113DOI: (10.1016/j.ijcard.2011.12.018) Copyright © 2011 Elsevier Ireland Ltd Terms and Conditions

Fig. 3 Shave sign for naïve deletion of patients. LEFT: A correlation arising without manipulation between a pair of normally distributed variables (top left). The histogram of positions of the data points along the short diagonal of the scatterplot (bottom left) is consistent with normal distribution. RIGHT: A correlation of the same magnitude, but arising by deletion of unfavourable patients (top right). The shave sign – unnaturally sharp demarcation of data points – becomes obvious in the histogram of data points along the short diagonal (bottom right). International Journal of Cardiology 2013 167, 102-113DOI: (10.1016/j.ijcard.2011.12.018) Copyright © 2011 Elsevier Ireland Ltd Terms and Conditions

Fig. 4 Distortion of the scatterplot resulting from selection of measurements designed to raise the correlation coefficient, and the bite test to detect this. LEFT: Correlation arising without manipulation between a pair of normally distributed variables. When the data (top left) are folded across the long and short diagonals (dotted lines), the resulting plot (bottom left) shows no “bite” in the highlighted corner. RIGHT: Correlation of the same magnitude arising from selection of most favourable pairings of unrelated irreproducible measurements. The bite sign is visible by eye in the scatterplot (top right), and when the data are folded across their long and short diagonals (bottom right) the bite is even more visible. It is statistically detectable by the bite test shown in Appendix A. For purposes of demonstration, this is the example dataset from Fig. 2 of “best of 3” performed in 120 patients. International Journal of Cardiology 2013 167, 102-113DOI: (10.1016/j.ijcard.2011.12.018) Copyright © 2011 Elsevier Ireland Ltd Terms and Conditions

Fig. 5 Hoist by one's own petard: Enron cross-hairs. Unwary authors may draw cross-hairs highlighting excellent diagnostic accuracy (left panel). To a sensitised reader, however, these cross-hairs may instead be bite-marks (right panel). In real biology, spread along the short diagonal is unlikely to shrivel at a diagnostically interesting point. Genuine high diagnostic accuracy occurs when short-diagonal spread is slender throughout the spectrum. Honest testing for this is less glamorous but more constructive. International Journal of Cardiology 2013 167, 102-113DOI: (10.1016/j.ijcard.2011.12.018) Copyright © 2011 Elsevier Ireland Ltd Terms and Conditions

Fig. 6 Bite test and shave test spreadsheet accompanying this article. The downloadable .xls file operates on Microsoft Excel or the free and open-source OpenOffice.Org. International Journal of Cardiology 2013 167, 102-113DOI: (10.1016/j.ijcard.2011.12.018) Copyright © 2011 Elsevier Ireland Ltd Terms and Conditions