1. Masked Visual Analysis (MVA)
A method that allows visual analysts to guard against falsely concluding an intervention has an effect

2. What should I do?

3. MVA Steps for Response-Guided Randomized Designs
1. Set study parameters
Research team agrees upon:
Deign type (e.g., MB)
Minimums (e.g., minimum of 5 observations per phase)
Randomization (e.g., random order of participants in MB)

4. 2. Split into two teams Analysis Team
Visually analyze the data and direct the Intervention Team
Intervention Team
Conduct the study based on the agreed upon parameters and the direction of the Analysis Team

5. 3. Conduct the study
The Intervention Team begins the study and sends the collected outcome data to the Analysis Team
The Analysis Team analyzes the data and makes decisions about when it would be appropriate to make a random assignment
The Intervention Team makes random assignments when directed by the Analysis Team and continues to collect and send the outcome measures to the Analysis Team, but they never disclose the results of the random assignments
The Analysis Team indicates when the study should be concluded

6. 4. Compute the p-value
The Analysis Team specifies what they believe are the results of the random assignments
The Intervention Team indicates if they are correct
If not correct, the Analysis Team continues to make specifications until a correct specification is made
The p-value is computed as:
p = # specifications/# possible assignments

7. Example 1: Multiple Baseline Design – 4 Participants
Step 1: Set study parameters
Dependent Variable? % of time on task
Design Type? Multiple Baseline Across Participants
Minimums? At least 5 baseline observations
Staggers of at least 2 observations
Treatment phases with at least 3 observations

8. Example 1: Multiple Baseline Design – 4 Participants
Step 1: Set study parameters
Randomization? Randomize order of participants for intervention
How many possible assignments of participants to treatment order? Who is 1st, 2nd, 3rd, and 4th?
P = 4! = 24 possible assignments
If the treatment has no effect, the probability that a masked visual analyst could identify the correct order
p = 1/24 = .0417

9. Example 1: Multiple Baseline Design – 4 Participants
Step 2: Split into two teams
Step 3: Conduct the study

10. Wendy
% Time on Task
Session
Tom

11. Wendy
Rob
Tom
Joel

12. Wendy
Rob
% Time on Task
Tom
Joel
Session

13. Wendy
Rob
% Time on Task
Tom
Joel
Session

14. Wendy
Rob
% Time on Task
Tom
Joel
Session

15. Wendy
Rob
Tom
Joel

16. Wendy
Rob
Tom
Joel

17. Wendy
Rob
Tom
Joel

18. Wendy
Rob
Tom
Joel

19. Example 1: Multiple Baseline Design – 4 Participants
Step 4: Compute the p-value
Analysis Team make a specification
Intervention Team, are they correct?
If the treatment has no effect, the probability that a masked visual analysts could have identified the correct order
p = 1/24 = .0417

20. Example 2: Multiple Baseline Design – 3 Participants
Step 1: Set study parameters
Design Type? Multiple Baseline Across Participants
Minimums? At least 5 baseline observations
Staggers of at least 3 observations
Treatment phases with at least 5 observations
If outlier, at least 3 additional observations in phase

21. Example 2: Multiple Baseline Design – 3 Participants
Dependent Variable? % intervals with prosocial behavior
Randomization?
- How many possible assignments of participants to treatment order? Who is 1st, 2nd, and 3rd?
P = 3! = 6 possible assignments
- What if we randomly select from Participant 1, Participant 2, Participant 3, and no one?
P=4! = 24 possible assignments, if correct p = 1/24 = .0417

22. Example 2: Multiple Baseline Design – 3 Participants
Step 2: Split into two teams
Step 3: Conduct the study

23. Ann
James
John

24. Ann
James
John

25. Ann
James
John

26. Ann
James
John

27. Ann
James
John

28. Ann
James
John

29. Ann
James
John

30. Example 2: Multiple Baseline Design – 3 Participants
Step 4: Compute the p-value
Analysis Team make a specification
Intervention Team, are they correct?
If the treatment has no effect, the probability that a masked visual analysts could have identified the assignments
p = 1/24 = .0417

31. Example 3: Changing Criterion Design
Step 1: Set study parameters
Design Type? Changing Criterion
Minimums? At least 3 observations per phase
Randomization? Random select without replacement from {M,M,M,M,M,M,S,S} for each phase to determine whether to Move to the next level of the criterion (M) or stay at the current level (S)
Number possible assignment is 𝐶= 8! 6!2! = 28
So if correct on first specification, p = 1/28 = .0357

32. Example 3: Changing Criterion Design
Step 2: Split into two teams
Step 3: Conduct the study

33. Example 3: Changing Criterion Design
Baseline ? ? ? ? ? ?

34. Example 3: Changing Criterion Design
Step 4: Compute the p-value
Analysis Team make a specification
Intervention Team, are they correct?
Yes? p = 1/28 = .0357

35. Example 4: Alternating Treatments Design
Step 1: Set study parameters
Design Type? Alternating Treatments (2 treatments)
Minimums? At least 5 alternating pairs
Randomization? Random assignment of one observation in the
pair to A and one to B
Because each assignment has 2 possibilities, need 5 assignments to obtain over 20 possible assignments and a p-value < .05.
25=32, so if correct with 5 pairs, p = 1/32 =

36. Example 4: Alternating Treatments Design
Step 2: Split into two teams
Step 3: Conduct the study

43. Example 4: Alternating Treatments Design
Step 4: Compute the p-value
Analysis Team make a specification
Intervention Team, are they correct?
Yes? p = 1/64 =
No? Make a second specification
If correct this time, p = 2/64 =

44. Example 5: Reversal Design
Step 1: Set study parameters
Dependent Variable? Number of Disruptive Behaviors
Design Type? Reversal
Minimums? At least 5 observations per phase
At least 3 phase changes (at least ABAB)
Randomization? Random assignment of treatment to blocks of
observations
Because each assignment has 2 possibilities, need 5 assignments to obtain over 20 possible assignments and a p-value < .05.
25=32, so if correct p = 1/32 =

45. Example 5: Reversal Design
Step 2: Split into two teams
Step 3: Conduct the study

59. Example 5: Reversal Design
Step 4: Compute the p-value
Analysis Team make a specification
Intervention Team, are they correct?
If the treatment has no effect, the probability that a masked visual analysts could have identified the assignments
p = 1/32 =

60. Example 6: Multiple Probe Design
Step 1: Set study parameters
Design Type? Multiple Probe with 5 Participants
Minimums? At least 5 observations in each phase
At least 3 consecutive observations prior to intervention
At least 3 consecutive observations after an intervention
Temporal staggers of at least 2 observations
Randomization? Random assignment of treatment to blocks of
observations, where there is one mystery block for
each participant at the point the participant becomes
eligible for intervention
25=32, so if correct with 5 blocks, p = 1/32 =

61. Example 6: Multiple Probe Design
Step 2: Split into two teams
Step 3: Conduct the study

62. A B ?
Dave ?
John ?
Bob ?
Dan ?
Theresa

63. Example 6: Multiple Probe Design
Step 4: Compute the p-value
Analysis Team make a specification
Intervention Team, are they correct?
Yes? p = 1/32 =

64. Applications and Illustrations
Byun, T. M., Hitchcock, E., & Ferron, J. M. (2017). Masked visual analysis: Minimizing type I error in response-guided single-case design for communication disorders. Journal of Speech, Language, and Hearing Research, 60,
DeLoatche, K. J. (2015). Parent-child interaction therapy as a treatment for ADHD in early childhood: A multiple baseline single-case design (Unpublished doctoral dissertation). University of South Florida, Tampa.
Dickerson, E. (2016). Computerized cognitive remediation therapy (CCRT): Investigating change in the psychological and cognitive function of adolescent psychiatric patients. (Unpublished doctoral dissertation). Northeastern University, Boston.
Ferron, J. M., & Levin, J. R. (2014). Single-case permutation and randomization statistical tests: Present status, promising new developments. In T. R. Kratochwill & J. R. Levin (Eds.), Single-case intervention research: Statistical and methodological advances (pp ). Washington, DC: American Psychological Association.
Ferron, J., & Jones, P. K. (2006). Tests for the visual analysis of response-guided multiple-baseline data. Journal of Experimental Education, 75,
Ferron, J. M., Joo, S.-H., & Levin, J. R. (2017). A Monte-Carlo evaluation of masked-visual analysis in response-guided versus fixed-criteria multiple-baseline designs. Journal of Applied Behavior Analysis, 50,
Hinojosa, S. M. (2016). Teacher child interaction therapy: An ecological approach to intervening with young children who display disruptive behaviors. (Unpublished doctoral dissertation). University of South Florida, Tampa.
Hua, Y., Yuan, C., Monroe, K., Hinzman, M. L., Alqahtani, S., Abdulmohsen, A., & Kern, A. M. (2016). Effects of the reread- adapt and answer-comprehend and goal setting intervention on decoding and reading comprehension skills of young adults with intellectual disabilities. Developmental Neurorehabilitation.
Ottley, J. R., Coogle, C. G., Rahn, N. L., & Spear, C. (2017). Impact of bug-in-ear professional development on early childhood co-teachers’ use of communication strategies. Topics in Early Childhood Special Education, 36,
McKenney, E. L., Mann, K. A., Brown, D. L., & Jewell, J. D. (2017). Addressing cultural responsiveness in consultation: An empirical demonstration. Journal of Educational and Psychological Consultation.
McKeown, D., Kimball, K., & Ledford, J. (2015). Effects of asynchronous audio feedback on the story revision practices of students with emotional/behavioral disorders. Education and Treatment of Children, 38,