1 Negative Results and Publication Bias in Single- Case Research Applications of the WWC Standards in Literature Reviews

The Legacy of Negative Results and its Relationship to Publication Bias The Importance of Negative Results in Developing Evidence-Based Practices (Kratochwill, Stoiber, & Gutkin, 2000) Negative Results in Single-Case Intervention Research Examples using the WWC Standards 2

The term negative results traditionally has meant that there are either: (a) no statistically significant differences between groups that receive different intervention conditions in randomized controlled trials; or (b) no documented differences (visually and/or statistically) between baseline and intervention conditions in experimental single-case designs. 3

In the domain of SCD research, negative results reflect findings of (a) no difference between baseline (A) and intervention (B) phases (As = Bs), (b) a difference between baseline and intervention phases but in the opposite direction to what was predicted (As > Bs, where B was predicted to be superior to A), (c) no difference between two alternative interventions, B and C (Bs = Cs), or (d) a difference between two alternative interventions, but in the direction opposite to what was predicted (Bs > Cs, where C was predicted to be superior to B). 4

Negative results/findings in SCD intervention research should be distinguished from negative effects in intervention research (i.e., iatrogenic effects). Some interventions may actually produce negative effects on participants (i.e., participants get worse or show negative side effects from an intervention)―see, for example Barlow (2010). 5

Selective results refer to the withholding of any findings in a single study or in a replication series (i.e., a series of single-case studies in which the treatment is replicated several times in independent experiments; see also our discussion below for selective results issues in replication series) and can be considered as a part of the domain of negative results. 6

Erroneous results have been considered in traditional “group” research in situations where various statistical tests are incorrectly conducted or interpreted to yield findings that are reported as statistically significant but are found not to be when the correct test or interpretation is applied (e.g., Levin, 1985). Also included in the erroneous results category are “spurious” findings that are produced in various research contexts. 7

Publication bias results when studies with positive outcomes or more favorable results, are more likely to be published than are those studies with null or negative findings. Negative results are less likely to be published, a tendency often known as publication bias. Publication bias may occur in single-case design research but less is known about this methodology. If literature summaries such as meta-analyses fail to include negative results, they may overestimate the size of an effect.

Sham and Smith (2014) examined publication bias by comparing effect sizes in single-case research in published studies (n=21) and non-published dissertation studies (n=10) in the area of pivotal response treatment (PRS). The effect sizes were assessed with PND. They found that the mean PND for published studies was 22% higher than unpublished studies. Nevertheless, PRS was found to be overall effective in published and unpublished studies.

Shadish, Zelinsky, Vevea, and Kratochwill (2015) surveyed SCD researchers about their publication practices to answer the following questions: (a) Does a bias exist against publishing results from SCD research that shows low effect sizes? (b) Does this bias change based on such characteristics of the data as data overlap and total data variability? (c) Does this bias fluctuate over participant demographics? and (d) Is this bias present in any of three forms: not submitting research to a journal, dropping cases before submission, and not recommending for publication? Results suggest SCD researchers do give preference to large effects in publication decisions.

11