1. Overlap Methods derived from Visual Analysis in Single Case Research Methodology In collaboration with Brian Reichow and Mark Wolery

2. Topics Rationale and History of Overlap Methods Overview of Methods and their Problems 1.PND 2.PEM 3.PEM-T (ECL) 4.PAND 5.R-IRD* 6.NAP* 7.TauU*  Recommendations for Overlap Methods

3. Rationale Assumption: Multiple studies are needed to be confident in the evidence supporting a practice –Need to aggregate findings from multiple studies –Meta-analytic methods are well established for group experimental research: Aggregate findings Quantify the magnitude Conduct analyses of moderator variables to account for difference in magnitude across studies (Lipsey & Wilson, 2001) –Consensus does not exist for SCRD How to calculate effect sizes?

4. Issues in Meta-analysis of SCRD Data are not independent –On a single individual –Using the same definitions –Using the same data collection procedures –In the same context –Under the same procedures –Often with short intervals between observations Compromises certain analysis techniques Compromises some quantitative synthesis methods

5. Issues in Meta-analysis of SCRD With single subject research, to identify a functional relation between independent and dependent variables: Threats to internal validity must be controlled The design needs an adequate number of replications of the experimental conditions The data patterns must shift consistently in the predicted (therapeutic) direction with each change in experimental conditions Single subject studies use a replication logic for making judgments about functional relations

6. Overlap Methods: Advantages Distribution free indices –Lack normality or constant variance so not impacted by outliers Small amount of data (short lengths) make it impossible to test parametric assumptions Calculated by hand Directly interpretable –Percent of data showing no overlap Allow the interventionist to remain in control

7. PND: Percent of Non- overlapping Data

8. Overlap Methods: PND Percentage of non-overlapping data (PND) Oldest of the overlap methods (Scruggs, & Mastropieri, 1998; Scruggs, Mastropieri, & Casto, 1987) Used extensively Easily calculated Does not assume data are independent Does not make other assumptions necessary in regression methods Interpreted as: “The percentage of Phase B data exceeding the single highest Phase A datum point.”

9. Overlap Methods: PND Calculated by: 1.Identify the intended change 2.Drawing a straight line from the highest (or lowest) point in Phase A and counting the number of data point in Phase B above the line 3.Quotient= # above the line / total number in Phase B X100 4.> 70% is effective, 50% to 70% is questionable effectiveness, and <50% is no observed effect (Scruggs & Mastropieri, 1998).

10. 1.PND is compromised by a baseline datum point at the floor or ceiling PND = 0 BaselineIntervention Change?

11. 2.PND is compromised by variability in the baseline condition, because it relies on the most extreme datum point in the baseline, perhaps the one that is least representative of the data pattern BaselineIntervention PND = 0 Change?

12. 3.PND is compromised by trends in data within conditions Baseline Intervention PND = 0 Change? BaselineIntervention Change? PND = 100

13. BaselineIntervention PND =66.7 Baseline Intervention PND = 92.3 4.PND is compromised by the number of data points in the intervention condition

14. BaselineIntervention PND =100 BaselineIntervention PND = 100 Change? 5.PND does not measure magnitude of difference

15. BaselineIntervention PND =60 BaselineIntervention PND = 60 Change? Lower graph looks like standard learning curve; doesn’t do well with this pattern

16. BaselineIntervention PND =50 Change? Extinction bursts may be problematic

17. Overlap Methods: PND—Problems 6.PND does not address critical issues of consistent replications Adding the PNDs across replications can lead to inaccurate conclusions

18. BaselineIntervention PND =100 PND =50 PND =100 100 100 +50 250 250/3=83.3 Mean PND=83.3

19. Overlap Methods: PND—Problems 7.Compared with consensus visual analysis, PND resulted in an error in about 1 of 5 condition changes—high rate of errors (Wolery, Busick, Reichow, & Barton, 2010)

20. PEM: Percent Exceeding the Median

21. Overlap Methods: PEM Percentage of Data Exceeding the Median (Ma, 2006) Developed to improve upon PND Rather than the highest datum point, PEM relies on the more stable and representative Phase A median level As with the PND, the PEM is not compromised by serial dependency and other data assumptions Simple calculation Designed to eliminate problem with baseline datum point being at floor or ceiling Designed to not rely on the most extreme datum point Less influenced by variability in baseline

22. Overlap Methods: PEM Calculated by: 1.Drawing a line at the median of Phase A data through Phase B data 2.Count the number of data points in Phase B above (or below) the line and divide by the total number of data points in Phase B

23. BaselineIntervention PEM = 57.1 Baseline Intervention PEM = 100 Change No change 1.PEM is compromised by trends in the data

24. BaselineIntervention PEM =66.7 Baseline Intervention PEM = 92.3 2.PEM is compromised by the number of intervention data points

25. BaselineIntervention PEM =100 BaselineIntervention PEM = 100 3.PEM is not an effect size estimate, because it does not measure magnitude of difference

26. BaselineIntervention PEM = 60 BaselineIntervention PEM = 60 Doesn’t do well with learning curves

27. BaselineIntervention PEM =60 Change? Extinction bursts may be problematic

28. Overlap Methods: PEM: Problems 4.The PEM does not address the critical analysis question (do data patterns consistently shift with each change in the experimental conditions?) Adding the PEMs across replications can lead to the same inaccurate conclusions as with the PND 5.Compared with consensus visual analysis, PEM resulted in an error in about 1 of 6 condition changes—high rate of errors (Wolery et al., 2010) 6.The PEM appears to over-estimate effects and does not discriminate well between graphs (Parker & Hagan-Burke, 2007)

29. PEM-T: Percent Exceeding the Median Trend line

30. Overlap Methods: PEM-T Percentage of Data Exceeding a Median-Based Trend (Wolery et al., 2010) As with the PND and PEM, the PEM-T is not compromised by serial dependency and other data assumptions Simple calculation, but requires graphing on semi-logarithmic paper (and re-graphing data) Designed to eliminate problem  with baseline datum point being at floor or ceiling  relying on the most extreme baseline point  trends in the data

31. Overlap Methods: PEM-T Calculated by: 1.Graph data on semi-logarithmic chart 2.Calculate and draw a split middle line of trend estimation for Phase A data and extend it through Phase B 3.Count # of Phase B data points above/below the split middle line of trend estimation 4.Divide count from Step 4 by # data points in Condition 2 and multiply quotient by 100

32. PEM-T not compromised by trends in the data Change No change Baseline Intervention PEM-T = 100 PEM-T = 0

33. 1.PEM-T is compromised by the number of intervention data points Baseline Intervention PEM-T = 66.7 PEM-T = 92.3

34. 2.PEM-T is not an effect size estimate, because it does not measure magnitude of difference Baseline Intervention PEM-T = 100

35. Baseline Intervention PEM-T = 60 Doesn’t do well with learning curves Extinction bursts may be problematic

36. Overlap Methods: PEM-T: Problems 3.The PEM-T does not address the critical analysis question (do data patterns consistently shift with each change in the experimental conditions?) –Adding the PEM-Ts across replications can lead to the same inaccurate conclusions as with the PND 4.Compared with consensus visual analysis, PEM- T resulted in an error in about 1 of 8 condition changes—high rate of errors (Wolery et al., 2010)

37. PAND: Percent of All Non- overlapping Data

38. Overlap Method: PAND Percentage of All Non-overlapping Data (Parker, Hagan- Burke, & Vannest, 2007) Percentage of data remaining after determining the fewest data points that must be removed to eliminate all between-phase overlap As with the PND, PEM, PEM-T, and IRD is not compromised by serial dependency or other data assumptions Simple calculation, although a bit more complex than PND or PEM Used for the multiple baseline design, but perhaps can be used with long (60-80 data point) A-B-A-B designs

39. Overlap Methods: PAND Calculated by: 1.Count the total number of data points in all tiers 2.Identify how many need data points need to be removed to eliminate overlap 3.Count the number of remaining data points 4.Divide count in step 3 by count in step 1

40. PAND: Issues 1.Variable baseline data likely to compromise PAND 2.Trends in the data will compromise PAND 3.Requires some overlap between phases to have sensitive estimates (Parker et al. 2007) 4.Increasing the length of the intervention condition can produce a higher PAND 5.PAND does not assess magnitude, because different patterns (e.g., shallow and sharp learning curves) can have same values

41. PAND: Issues 7.Combining data across tiers –violates the within-tier analysis logic of the multiple baseline design –Violates the replication logic of the multiple baseline design –assumes each tier is equivalent in terms of changes (or lack thereof) in data patterns 8.Method for calculating confidence intervals give the impression of statistical precision, when indeed it is absent

42. PAND: Issues 9.The PAND does not address the critical analysis question (do data patterns consistently shift with each change in the experimental conditions?)

43. IRD: Improvement Rate Difference

44. Overlap Methods: IRD Improvement Rate Difference (Parker, Vannest, & Brown, 2009) As with the PND and PEM, the PEM-T is not compromised by serial dependency or other data assumptions Simple calculation, although a bit more complex than PND or PEM Allows confidence intervals to be calculated Has history of use in group health care research Note: rate in IRD is not about rate of behavior

45. R-IRD: Calculation IRD (Parker et al., 2009) was developed to improve upon PAND by providing an easily interpretable, reputable effect size index (with a sampling distribution). IRD calculation begins with the same method (fewest data points that must be removed to eliminate all overlap) as PAND, but in a second step converts the results to two improvement rates (IR), for phase A and B respectively. The two IR values are finally subtracted to obtain the “Improvement Rate Difference” (IRD).

46. A1A1 B1B1 A2A2 B2B2 From Parker et al. (2009) (7) (16) (4) (12) A 1 to B 1 : (13/16=81%) – (0/7=0%) = IRD of 81% B 1 to A 2 : (13/16=81%) – (2/4=50%) = IRD of 31% A 2 to B 2 : (11/12=92%) – (0/4=0%) = IRD of 92% IRD =(81+31+92)/3 = 68%

47. R-IRD: Calculation The original IRD article recommended that in the first step, data point ‘removal’ “should be balanced across the contrasted phases” (Parker et al., 2009, p. 141) for more robust results. A better robust IRD solution was later described and formalized as “Robust IRD” (R-IRD). R-IRD requires rebalancing (by hand) of a 2 x 2 matrix IRD is interpreted as the difference in the proportion of high or “improved” scores between phases B and A.

48. R-IRD: Calculation The superior robust version of IRD (R-IRD) requires that quadrants be balanced. Balancing allows for a more conservative effect in instances where a large number of data points may be removed arbitrarily from one side and a few from the other, which can unduly influence the results. R - IRD is unbiased in the sense that it does not allow bias in removal of data points from A versus B, as some datasets provide two or more equally good removal solutions.

49. R-IRD: Calculation 1.Determine the fewest data points that must be removed to eliminate overlap 2.Balance quadrant W and Z 3.Then balance Y = A Phase A: W / (W + Y) 4.Then balance X = B Phase B: X / (X + Z) 5.R - IRD = B – A

50. IRD is not influenced by a data point at floor or ceiling BaselineIntervention A 1 to B 1 : (7/8=88%) – (0/7=0%) = IRD of 88%

51. 1.IRD is compromised by variability in the baseline condition BaselineIntervention A 1 to B 1 : (5/8=63%) – (2/7=29%) = IRD of 34% “An IRD of 50% (.50) indicates that half the scores are overlapping, so did not improve from phase A to B” (Parker et al., 2009, p. 139).

52. 2.IRD is compromised by trends in the conditions Baseline Intervention A 1 to B 1 : (5/8=63%) – (3/7=43%) = IRD of 20%

53. BaselineIntervention IRD =66.7 Baseline Intervention IRD = 92% 3.IRD is compromised by the number of data points in the intervention condition (7/7=100%) (1/3=33%) (7/7=100%) (1/13=8%)

54. BaselineIntervention IRD =100 BaselineIntervention IRD = 100 4.IRD does not measure magnitude of difference (10/10=100%) (0/10=0%) (10/10=100%) (0/10=0%)

55. IRD 5.The IRD does not address replication logic. Although the IRD has some advantages over other overlap methods, it is still flawed.

56. Practice: Calculating R-IRD Phase A: 0 4 3 0 0 Phase B: 5 2 3 5 3 5 6 7

57. 1.Determine the fewest data points that must be removed to eliminate overlap = 2 2.Balance quadrant W and Z = 1, 1 3.Then balance X = B = 7 Phase B: X / (X + Z) = 7/(1+7) =.875 4.Then balance Y = A = 4 Phase A: W / (W + Y) = 1/(1+4)=.2 5.R - IRD = B – A =.875 -.2 =.675 W Not Improved= 2

58. NAP: Non-overlap of All Pairs

59. Overview of NAP The percentage of data that improve from A to B or operationally, the percentage of all pairwise comparisons from Phase A to B which show improvement or growth (Parker & Vannest, 2009) NAP’s limitations include that it is insensitive to trends and outliers

60. Calculating NAP 1.NAP begins with all pairwise comparisons (#Pairs = n A × n B ) between phases. 2.Each paired comparison has one of three outcomes: improvement over time (Pos), deterioration (Neg), or no change over time (Tie). 3.NAP is calculated as (Pos +.5 × Tie) / #Pairs.

61. Practice: Calculating NAP Phase A: 0 4 3 0 0 Phase B: 5 2 3 5 3 5 6 7

62. # of Pairs = 5*8 = 40 #Pos = 34, #Neg = 4, #Tie = 2 NAP = (#Pos +.5*#Ties)/#Pairs NAP = (34 +.5*2)/40 NAP =.875 N=5 N=8

63. TauU (Kendall’s Tau + Mann-Whitney U)

64. Overview of TauU NAP’s major limitation of insensitivity to data trend led to development of a new index that integrates non-overlap and trend: TauU (Parker, Vannest, Davis, & Sauber, 2011). Melding KRC and MW-U are transformations of one another and share the same S sampling distribution The Tau-U score is not affected by the ceiling effect present in other non-overlap methods, and performs well in the presence of autocorrelation.

65. Calculating TauU Simplest TauU (non-overlap only) Conduct the same pairwise comparisons (n A × n B = #Pairs) across phases as is NAP, resulting in a Pos, Neg, or Tie for each pair The TauU simple non-overlap form (not considering trend) is TauU = (Pos - Neg) / Pairs Thus, NAP is percent of non-overlapping data, whereas TauU is percent of non- overlapping minus overlapping data.

66. Practice: Calculating TauU Phase A: 0 4 3 0 0 Phase B: 5 2 3 5 3 5 6 7

67. # of Pairs = 5*8 = 40 #Pos = 34, #Neg = 4, #Tie = 2 TauU = (#Pos - #Neg)/#Pairs TauU = (34 - 4)/40 TauU =.75 N=5 N=8

68. www.singlecaseresearch.org

69. Summary Many different synthesis metrics Each has significant limitations and flaws None are satisfactory PND been quite popular (maybe most flawed), but increasing popularity of PAND is evident Common problems Variability, trends, and “known” data patterns (extinction bursts, learning curves, delayed effects) Failure to measure magnitude Ignoring the replication logic of SSR

70. Summary Complete non-overlap measures offer the most robust option (NAP, TauU) –Complete measures equally emphasize all scores –Incomplete measures emphasize particular scores (e.g., median) Interpretation of ES is a tricky business, with context, social significance, clinical significance, effects of prior studies, and the behaviors under examination all a part of the interpretation  Recommend using IRD, NAP, or TauU

71. For more information: Ma, H. H. (2006).An alternative method for quantitative synthesis of single- subject research: Percentage of datapoints exceeding the median. Behavior Modification, 30, 598–617. Parker, R., & Vannest, K. J. (2008). An improved effect size for single case research: Non-overlap of all pairs (NAP). Behavior Therapy, 40, 357-67. Parker, R. I., Vannest, K. J., & Brown, L. (2009). The improvement rate difference for single case research. Exceptional Children, 75, 135–150. Parker, R. I., Vannest, K. J., & Davis, J. L. (2011). Effect size in single-case research: A review of nine nonoverlap techniques. Behavior Modification, 35, 303-322. Wolery, M., Busick, M., Reichow, B., & Barton, E. (2010). Comparison of overlap methods for quantitatively synthesizing single-subject data. Journal of Special Education, 44, 18-28.