1. Masked Visual Analysis (MVA) A method to ensure control of the Type I error rate when visually analyzing single-case studies

2. Type I Error Control A Type I error is committed when an analyst concludes there was an effect when there really was not. In conventional statistical analyses researchers often set the Type I error rate to.05. With traditional visual analyses it is difficult to know how likely it is that a researcher will incorrectly conclude there was an effect.

4. Type I Error Studies Estimate Study Estimate Study.24 Matyas & Greenwood, 1990.24 Matyas & Greenwood, 1990.25 Stocks & Williams, 1995.28 Fisch, 2001.66 Borckardt, Murphy, Nash, & Shaw, 2004.66 Borckardt, Murphy, Nash, & Shaw, 2004.17/.01 Carter, 2009.17/.01 Carter, 2009

5. MVA Steps for Randomized Designs 1. Plan the study 2. Spit research team into two groups: a) intervention team b) analysis team 3. Intervention team makes random assignment, but does not tell analysis team 4. Research team conducts the study 3. Research team creates a masked graph 4. Analysis team analyzes masked graph

6. Example Application (Thanks to Kendall Jefferies) Study to examine the effect of parent/child interaction therapy (PCIT) on the number of praises given by parent during interaction with child Design Type: Multiple Baseline Across 4 Participants Intervention Schedule: Baseline lengths of 3, 4, 5, and 6 Randomization: Randomize order of participants for intervention

7. Dyadic Parent-Child Interaction Coding System (DPICS): LABELED PRAISES Session

8. Compute the p-value The p-value is computed as: p = # specifications/# possible assignments # possible assignments = 4*3*2*1 = 24 # possible assignments = 4*3*2*1 = 24 p = 1/24 =.0417

9. Type I error control If there were no treatment effects the data would be the same regardless of which random assignments were made. As a consequence, the Analysis Team would make the same decisions and the same specification (e.g., always say the order is 1, 2, 4, 3). Because the assignments are made randomly the probability that the assignment corresponds to the one the Analysis Team would pick is 1 out of the # possible (e.g., the order 1, 2, 4, 3 would be selected randomly 1 out of 24 times).

10. MVA Steps for Response-Guided Randomized Designs 1. Set study parameters Research team agrees upon: Deign type (e.g., MB) Deign type (e.g., MB) Minimums (e.g., minimum of 5 observations per phase) Minimums (e.g., minimum of 5 observations per phase) Randomization (e.g., random order of participants in MB) Randomization (e.g., random order of participants in MB)

11. 2. Split into two teams Analysis Team Visually analyze the data and direct the Intervention Team Intervention Team Conduct the study based on the agreed upon parameters and the direction of the Analysis Team

12. 3. Conduct the study The Intervention Team begins the study and sends the collected outcome data to the Analysis Team The Intervention Team begins the study and sends the collected outcome data to the Analysis Team The Analysis Team analyzes the data and makes decisions about when it would be appropriate to make a random assignment The Analysis Team analyzes the data and makes decisions about when it would be appropriate to make a random assignment The Intervention Team makes random assignments when directed by the Analysis Team and continues to collect and send the outcome measures to the Analysis Team, but they never disclose the results of the random assignments The Intervention Team makes random assignments when directed by the Analysis Team and continues to collect and send the outcome measures to the Analysis Team, but they never disclose the results of the random assignments The Analysis Team indicates when the study should be concluded. The Analysis Team indicates when the study should be concluded.

13. 4. Compute the p-value The Analysis Team specifies what they believe are the results of the random assignments The Analysis Team specifies what they believe are the results of the random assignments The Intervention Team indicates if they are correct The Intervention Team indicates if they are correct If not correct, the Analysis Team continues to make specifications until a correct specification is made. If not correct, the Analysis Team continues to make specifications until a correct specification is made. The p-value is computed as: The p-value is computed as: p = # specifications/# possible assignments

14. Example 1: Multiple Baseline Design – 4 Participants Step 1: Set study parameters Design Type? Multiple Baseline Across Participants Minimums? At least 5 baseline observations Staggers of at least 2 observations Treatment phases with at least 3 observations Randomization? Randomize order of participants for intervention

15. Example 1: Multiple Baseline Design – 4 Participants Step 1: Set study parameters How many possible assignments of participants to treatment order? Who is 1 st, 2 nd, 3 rd, and 4 th ? P = 4! = 24 possible assignments If the treatment has no effect, the probability that a masked visual analyst could identify the correct order p = 1/24 =.0417

16. Example 1: Multiple Baseline Design – 4 Participants Step 2: Split into two teams Step 3: Conduct the study

26. Example 1: Multiple Baseline Design – 4 Participants Step 4: Compute the p-value Analysis Team make a specification Intervention Team, are they correct? If the treatment has no effect, the probability that a masked visual analysts could have identified the correct order p = 1/24 =.0417

27. Example 2: Multiple Baseline Design – 3 Participants Step 1: Set study parameters Design Type? Multiple Baseline Across Participants Minimums? At least 5 baseline observations Staggers of at least 3 observations Treatment phases with at least 5 observations If outlier, at least 3 additional observations in phase

28. Example 2: Multiple Baseline Design – 3 Participants Randomization? How many possible assignments of participants to treatment order? Who is 1 st, 2 nd, and 3 rd ? P = 3! = 6 possible assignments What if we randomly select from Participant 1, Participant 2, Participant 3, and no one? P=4! = 24 possible assignments, if correct p = 1/24 =.0417

29. Example 2: Multiple Baseline Design – 3 Participants Step 2: Split into two teams Step 3: Conduct the study

37. Example 2: Multiple Baseline Design – 3 Participants Step 4: Compute the p-value Analysis Team make a specification Intervention Team, are they correct? If the treatment has no effect, the probability that a masked visual analysts could have identified the assignments p = 1/24 =.0417

38. Example 3: Reversal Design Step 1: Set study parameters Design Type? Reversal Minimums? At least 5 observations per phase At least 3 phase changes (at least ABAB) Randomization? Random assignment of treatment to blocks of observations Because each assignment has 2 possibilities, need 5 assignments to obtain over 20 possible assignments and a p-value <.05. 2 5 =32, so if correct p = 1/32 =.03125

39. Example 3: Reversal Design Step 2: Split into two teams Step 3: Conduct the study

52. Example 3: Reversal Design Step 4: Compute the p-value Analysis Team make a specification Intervention Team, are they correct? If the treatment has no effect, the probability that a masked visual analysts could have identified the assignments p = 1/32 =.03125

53. Example 4: Alternating Treatments Design Step 1: Set study parameters Design Type? Alternating Treatments (2 treatments) Minimums? At least 5 alternating pairs Randomization? Random assignment of one observation in the pair to A and one to B Because each assignment has 2 possibilities, need 5 assignments to obtain over 20 possible assignments and a p-value <.05. 2 5 =32, so if correct with 5 pairs, p = 1/32 =.03125

54. Example 4: Alternating Treatments Design Step 2: Split into two teams Step 3: Conduct the study

61. Example 4: Alternating Treatments Design Step 4: Compute the p-value Analysis Team make a specification Intervention Team, are they correct? Yes? p = 1/64 =.015625 No? Make a second specification If correct this time, p = 2/64 =.03125

62. Example 5: Multiple Probe Design Step 1: Set study parameters Design Type? Multiple Probe with 5 Participants Minimums? At least 5 observations in each phase At least 3 consecutive observations prior to intervention At least 3 consecutive observations after an intervention Temporal staggers of at least 2 observations Randomization? Random assignment of treatment to blocks of observations, where there is one mystery block for each participant at the point the participant becomes eligible for intervention 2 5 =32, so if correct with 5 blocks, p = 1/32 =.03125

63. Example 5: Multiple Probe Design Step 2: Split into two teams Step 3: Conduct the study

64. A B ? ? ? ? ? Dave John Bob Dan Theresa

65. Example 5: Multiple Probe Design Step 4: Compute the p-value Analysis Team make a specification Intervention Team, are they correct? Yes? p = 1/32 =.03125