1. Kriegeskorte N, Simmons WK, Bellgowan PSF, Baker CI. Circular analysis in systems neuroscience – the dangers of double dipping slides by Nikolaus Kriegeskorte, based on: Circular analysis in systems neuroscience – the dangers of double dipping Kriegeskorte N, Simmons WK, Bellgowan PSF, Baker CI. (2009) Nature Neuroscience 12(5): 535-40.

2. What is circular inference? Example 1: Pattern-information analysis Example 2: Regional activation analysis Example 3: Data sorting Contrast orthogonality Take-home message Supplementary materials Overview

3. What is circular inference?

5. dataresults

6. analysis dataresults

7. dataresults analysis

8. dataresults analysis assumptions

9. dataresults analysis assumptions

10. Circular inference dataresults analysis assumptions

11. Circular inference dataresults analysis assumptions

12. How do assumptions tinge results? Elimination (binary selection) Weighting (continuous selection) Sorting (multiclass selection) – Through variants of selection!

13. Example 1 Pattern-information analysis

14. Experimental design “Animate?”“Pleasant?” STIMULUS (object category) TASK (property judgment) Simmons et al. 2006

15. define ROI by selecting ventral-temporal voxels for which any pairwise condition contrast is significant at p<.001 (uncorr.) perform nearest-neighbor classification based on activity-pattern correlation use odd runs for training and even runs for testing Pattern-information analysis

16. 0 0.5 1 decoding accuracy task (judged property) stimulus (object category) Results chance level

17. fMRI data using all data to select ROI voxels using only training data to select ROI voxels data from Gaussian random generator 0 0.5 1 0 1 0 1 0 1 decoding accuracy chance level task stimulus...but we used cleanly independent training and test data! ? !

18. Conclusion for pattern-information analysis The test data must not be used in either... training a classifier or defining the ROI continuous weighting binary weighting

19. Data selection is key to many conventional analyses. Can it entail similar biases in other contexts?

20. Example 2 Regional activation analysis

21. ROI definition is affected by noise true region overfitted ROI ROI-average activation overestimated effect independent ROI

22. Regional-average activation analysis Simulation 3D voxel volume (20x11x11 voxels) block-design experiment (4 conditions) spatiotemporal noise: Gaussian, slightly spatially smoothed

23. Regional-average activation analysis true effects overfitted ROI independent-data ROI contrast hypothesis A condition BCD fMRI signal A-D p<0.0001 (uncorr.) central slice p<0.01

24. Regional-average activation analysis truth same-data ROI independent-data ROI blending continuum hypothesis p<0.01

25. Example 3 Data sorting

26. Data sorting dataresults analysis assumptions: sorting criteria

27. Set-average tuning curves stimulus parameter (e.g. orientation) response...for data sorted by tuning noise data

28. ROI-average fMRI response ABCD condition Set-average activation profiles...for data sorted by activation noise data

29. Contrast orthogonality Orthogonal contrast vectors don’t ensure orthogonal contrasts

30. Does selection by an orthogonal contrast vector ensure unbiased analysis? ROI-definition contrast: A+B ROI-average analysis contrast: A-B c selection =[1 1] T c test =[1 -1] T orthogonal contrast vectors 

31. Does selection by an orthogonal contrast vector ensure unbiased analysis? not sufficient contrast vector The design and noise dependencies matter.designnoise dependencies – No, there can still be bias. still not sufficient

32. Take-home message

33. To avoid selection bias......we need to make sure that the results statistics are independent of the selection criteria under the null hypothesis. Selection and results statistics can be either: inherently independent or computed on independent data e.g. independent contrasts

35. Supplementary materials Detailed policy for noncircular analysis Pros and cons of circular analysis Exploration and confirmation Circularity indicators Bias despite orthogonal contrast vectors Severity and prevalence

36. Detailed policy for noncircular analysis

40. Pros and cons of circular analysis

41. Circular analysis Pros highly sensitive widely accepted (examples in all high-impact journals) doesn't require independent data sets grants scientists independence from the data allows smooth blending of blind faith and empiricism Cons

42. Circular analysis Pros highly sensitive widely accepted (examples in all high-impact journals) doesn't require independent data sets grants scientists independence from the data allows smooth blending of blind faith and empiricism Cons

43. Circular analysis Pros highly sensitive widely accepted (examples in all high-impact journals) doesn't require independent data sets grants scientists independence from the data allows smooth blending of blind faith and empiricism Cons [can’t think of any right now] Pros the error that beautifies results confirms even incorrect hypotheses improves chances of high-impact publication

45. Selective analysis as exploration and confirmation

46. Recurrent processing data fitted model hypothesisassumption model fitted model ROI exploratoryconfirmatory fitting testing 12 classifier data ROI activation classifier accuracy results independent ?

48. Circularity indicators

49. C C

51. Biased selection despite orthogonal contrast vectors

52. Assessment by simulation Analytical assessmentUnbalanced design c selection =[1 1]’c test =[1 -1]’ 0.208

53. Orthogonal contrast vectors, balanced design, no temporal autocorrelation no bias

54. Orthogonal contrast vectors, unbalanced design, no temporal autocorrelation bias

55. Orthogonal contrast vectors, balanced design, temporal autocorrelation bias

57. Severity and prevalence of different forms of circular analysis (subjective assessment!)

58. Examples of circular practices same data used for training and testing a classifier non-independent data used for training and testing all data used to define the ROI for a classifier analysis set averages analyzed on the same data used for sorting voxels (or neurons) into the sets example neurons selectively analyzed after statistical selection using the same data ROI-average activation regressed onto some factor that is related to the ROI-definition contrast descriptive or inferential analysis of ROI-average activation not independent of ROI definition severity prevalence subjective