1. DAY 4 Special Topics: Negative Results and Publication Bias Tom Kratochwill

2. Goals of the Presentation
Discuss negative Results and Publication Bias in Single-Case Research
Present the applications of the WWC Pilot Standards in Literature Reviews to Address Negative Results

3. Negative Results in Single-Case Intervention Research
The Legacy of Negative Results and its Relationship to Publication Bias
The Importance of Negative Results in Developing Evidence-Based Practices
Negative Results in Single-Case Intervention Research
Examples using the WWC Pilot Standards
Watch for the upcoming special series edited by Horner, Kratochwill, and Levin on negative results in Remedial and Special Education.

4. Negative Results Definition
The term negative results traditionally has meant that there are either: (a) no statistically significant differences between groups that receive different intervention conditions in randomized controlled trials; or (b) no documented differences (visually and/or statistically) between baseline and intervention conditions in experimental single-case designs.

5. Negative Results in Single-Case Design
In the domain of SCD research, negative results reflect findings of (a) no difference between baseline (A) and intervention (B) phases (As = Bs), (b) a difference between baseline and intervention phases but in the opposite direction to what was predicted (As > Bs, where B was predicted to be superior to A), (c) no difference between two alternative interventions, B and C (Bs = Cs), or (d) a difference between two alternative interventions, but in the direction opposite to what was predicted (Bs > Cs, where C was predicted to be superior to B).

6. Negative Effects
Negative results/findings in SCD intervention research should be distinguished from negative effects in intervention research (i.e., iatrogenic effects). Some interventions may actually produce negative effects on participants (i.e., participants get worse or show negative side effects from an intervention)―see, for example Barlow for a discussion in the psychotherapy literature (2010).

7. Selective Results
Selective results refer to the withholding of any findings in a single study or in a replication series (i.e., a series of single-case studies in which the treatment is replicated several times in independent experiments; see also our discussion below for selective results issues in replication series) and can be considered as a part of the domain of negative results.

8. Erroneous Results
Erroneous results have been considered in traditional “group” research in situations where various statistical tests are incorrectly conducted or interpreted to yield findings that are reported as statistically significant but are found not to be when the correct test or interpretation is applied (e.g., Levin, 1985). Also included in the erroneous results category are “spurious” findings that are produced in various research contexts.

9. Publication Bias: Some Considerations
Publication bias results when studies with positive outcomes or more favorable results, are more likely to be published than are those studies with null or negative findings.
Negative results are less likely to be published, a tendency often known as publication bias.
Publication bias occurs in single-case design research but less is known about it in this methodology.
If literature summaries such as meta-analyses fail to include negative results, they may overestimate the size of an effect.

10. Publication Bias: Some Recent Research
Sham and Smith (2014) examined publication bias by comparing effect sizes in single-case research in published studies (n=21) and non-published dissertation studies (n=10) in the area of pivotal response treatment (PRS). The effect sizes were assessed with PND. They found that the mean PND for published studies was 22% higher than unpublished studies. Nevertheless, PRS was found to be overall effective in published and unpublished studies.
Sham, E., & Smith, T. (2014). Publication bias in studies of an applied behavior-analytic intervention: An initial analysis. Journal of Applied Behavior Analysis, 47,

11. Publication Bias: Some Recent Research (Continued)
Shadish, Zelinsky, Vevea, and Kratochwill (2016) surveyed SCD researchers about their publication practices. Results suggested researchers expressed a preference for submitting manuscripts for review that show large effects. And there was a reported preference for large effects in making publication recommendations on manuscripts that researchers were reviewing. Although data on SCD publication practices is limited, we suggest that there is likely a preference for positive-results studies and very likely a publication bias in the SCD intervention literature.
Shadish, W. R., Zelinsky, N. A. M., Vevea, J. L., & Kratochwill, T. R. (2016). Survey of publication preferences of single-case design researchers when treatments have small or large effects. Journal of Applied Behavior Analysis, DOI: /jaba.308

12. Some Recent Research (Continued)
See Kittelman, Gion, Horner, Levin, & Kratochwill (2017). Establishing Journalistic Standards for the Publication of Negative Results. Manuscript under review.

13. A review of 29 journals from education and psychology was conducted to assess guidance provided to authors for submitting research that report negative results. The most recent issue of each journal was reviewed to determine whether articles reporting negative results were published, and a survey of editors for each of the journals was conducted to assess conditions in which publication of articles that did not demonstrate experimental effects warranted publication. Results indicate that only one of the 29 journals currently provide formal guidance to authors submitting papers about related to negative results. Eleven of the 129 articles published by the 29 journals in their last issue of 2016 included descriptions of negative results, and of the 60% of recruited editors who responded, 96% indicated there were conditions where publication of negative results was appropriate.

14. Example Negative Results Research

15. Weighted Vest Interventions for Individuals with Developmental Disabilities
Sensory-based interventions have been popular and are a common requested intervention for children with developmental disabilities (e.g., autism spectrum disorders);
Sensory interventions are designed to improve sensory processing and increase adaptive functioning;
The interventions are based on sensory integration theory.

16. Cox, Gast, Luscre, and Ayers (2009) studied the effects of a weighted vest on in-seat behaviors of elementary-age students with autism and severe to profound intellectual disabilities
Cox, A.L., Gast, D.L., Luscre, D.,& Ayres, K.M.(2009).The effects of weighted vests on appropriate in seat behaviors of elementary-age students with autism and severe to profound intellectual disabilities. Focus on Autism and Other Developmental Disabilities,24, 17–26.

17. Methods – Study 1 Participants:
May : 5.5 years; probable autism on GARS
Sam: 6.4 years; severe autism on CARS
Ted: 9 years; probable autism on GARS
Conditions:
No vest
Vest without weights
Weighted vest
Sequence was randomly determined
Examined in-seat behaviors:
Child facing forward looking at teacher and body in seat
Design: ATD
In the comparison phase of the study, the three
conditions were rapidly alternated across days. The
sequence of the three conditions was determined before
the study with a semirandom procedure whereby each
student was randomly assigned to begin with a different
condition. Subsequent conditions were drawn randomly
from the remaining two conditions. Once a condition
was scheduled, the next condition had to be either of the
other conditions (selected randomly). Once a condition
was set on the schedule five times, it

18. Results of Study 1

19. A Systematic Review of Sensory-Based Treatments
Barton, E. E., Reichow, B., Schnitz, A., Smith, I. C., & Sherlock, D. (2015). A systematic review of sensory-based treatments for children with disabilities. Research in Developmental Disabilities, 37,

20. A Review Article
Sensory-based therapies are designed to address sensory processing difficulties by helping to organize and control the regulation of environmental sensory inputs. These treatments are increasingly popular, particularly with children with behavioral and developmental disabilities. However, empirical support for sensory-based treatments is limited. The purpose of this review was to conduct a comprehensive and methodologically sound evaluation of the efficacy of sensory-based treatments for children with disabilities. Methods for this review were registered with PROSPERO (CRD ).

21. The Review Methods
Thirty studies involving 856 participants met inclusion criteria and were included in the review. Considerable heterogeneity was noted across studies in implementation, measurement, and study rigor. The research on sensory-based treatments is limited due to insubstantial treatment outcomes, weak experimental designs, or high risk of bias. Although many people use and advocate for the use of sensory-based treatments and there is a substantial empirical literature on sensory-based treatments for children with disabilities, insufficient evidence exists to support their use.

22. Questions and Discussion