Masked Visual Analysis (MVA) A method that allows visual analysts to guard against falsely concluding an intervention has an effect

What should I do?

MVA Steps for Response-Guided Randomized Designs 1. Set study parameters Research team agrees upon: Deign type (e.g., MB) Minimums (e.g., minimum of 5 observations per phase) Randomization (e.g., random order of participants in MB)

2. Split into two teams Analysis Team Visually analyze the data and direct the Intervention Team Intervention Team Conduct the study based on the agreed upon parameters and the direction of the Analysis Team

3. Conduct the study The Intervention Team begins the study and sends the collected outcome data to the Analysis Team The Analysis Team analyzes the data and makes decisions about when it would be appropriate to make a random assignment The Intervention Team makes random assignments when directed by the Analysis Team and continues to collect and send the outcome measures to the Analysis Team, but they never disclose the results of the random assignments The Analysis Team indicates when the study should be concluded

4. Compute the p-value The Analysis Team specifies what they believe are the results of the random assignments The Intervention Team indicates if they are correct If not correct, the Analysis Team continues to make specifications until a correct specification is made The p-value is computed as: p = # specifications/# possible assignments

Example 1: Multiple Baseline Design – 4 Participants Step 1: Set study parameters Dependent Variable? % of time on task Design Type? Multiple Baseline Across Participants Minimums? At least 5 baseline observations Staggers of at least 2 observations Treatment phases with at least 3 observations

Example 1: Multiple Baseline Design – 4 Participants Step 1: Set study parameters Randomization? Randomize order of participants for intervention How many possible assignments of participants to treatment order? Who is 1st, 2nd, 3rd, and 4th? P = 4! = 24 possible assignments If the treatment has no effect, the probability that a masked visual analyst could identify the correct order p = 1/24 = .0417

Example 1: Multiple Baseline Design – 4 Participants Step 2: Split into two teams Step 3: Conduct the study

Wendy % Time on Task Session Tom

Wendy Rob Tom Joel

Wendy Rob % Time on Task Tom Joel Session

Wendy Rob % Time on Task Tom Joel Session

Wendy Rob % Time on Task Tom Joel Session

Wendy Rob Tom Joel

Wendy Rob Tom Joel

Wendy Rob Tom Joel

Wendy Rob Tom Joel

Example 1: Multiple Baseline Design – 4 Participants Step 4: Compute the p-value Analysis Team make a specification Intervention Team, are they correct? If the treatment has no effect, the probability that a masked visual analysts could have identified the correct order p = 1/24 = .0417

Example 2: Multiple Baseline Design – 3 Participants Step 1: Set study parameters Design Type? Multiple Baseline Across Participants Minimums? At least 5 baseline observations Staggers of at least 3 observations Treatment phases with at least 5 observations If outlier, at least 3 additional observations in phase

Example 2: Multiple Baseline Design – 3 Participants Dependent Variable? % intervals with prosocial behavior Randomization? - How many possible assignments of participants to treatment order? Who is 1st, 2nd, and 3rd? P = 3! = 6 possible assignments - What if we randomly select from Participant 1, Participant 2, Participant 3, and no one? P=4! = 24 possible assignments, if correct p = 1/24 = .0417

Example 2: Multiple Baseline Design – 3 Participants Step 2: Split into two teams Step 3: Conduct the study

Ann James John

Ann James John

Ann James John

Ann James John

Ann James John

Ann James John

Ann James John

Example 2: Multiple Baseline Design – 3 Participants Step 4: Compute the p-value Analysis Team make a specification Intervention Team, are they correct? If the treatment has no effect, the probability that a masked visual analysts could have identified the assignments p = 1/24 = .0417

Example 3: Changing Criterion Design Step 1: Set study parameters Design Type? Changing Criterion Minimums? At least 3 observations per phase Randomization? Random select without replacement from {M,M,M,M,M,M,S,S} for each phase to determine whether to Move to the next level of the criterion (M) or stay at the current level (S) Number possible assignment is 𝐶= 8! 6!2! = 28 So if correct on first specification, p = 1/28 = .0357

Example 3: Changing Criterion Design Step 2: Split into two teams Step 3: Conduct the study

Example 3: Changing Criterion Design Baseline ? ? ? ? ? ?

Example 3: Changing Criterion Design Step 4: Compute the p-value Analysis Team make a specification Intervention Team, are they correct? Yes? p = 1/28 = .0357

Example 4: Alternating Treatments Design Step 1: Set study parameters Design Type? Alternating Treatments (2 treatments) Minimums? At least 5 alternating pairs Randomization? Random assignment of one observation in the pair to A and one to B Because each assignment has 2 possibilities, need 5 assignments to obtain over 20 possible assignments and a p-value < .05. 25=32, so if correct with 5 pairs, p = 1/32 = .03125

Example 4: Alternating Treatments Design Step 2: Split into two teams Step 3: Conduct the study

Example 4: Alternating Treatments Design Step 4: Compute the p-value Analysis Team make a specification Intervention Team, are they correct? Yes? p = 1/64 = .015625 No? Make a second specification If correct this time, p = 2/64 = .03125

Example 5: Reversal Design Step 1: Set study parameters Dependent Variable? Number of Disruptive Behaviors Design Type? Reversal Minimums? At least 5 observations per phase At least 3 phase changes (at least ABAB) Randomization? Random assignment of treatment to blocks of observations Because each assignment has 2 possibilities, need 5 assignments to obtain over 20 possible assignments and a p-value < .05. 25=32, so if correct p = 1/32 = .03125

Example 5: Reversal Design Step 2: Split into two teams Step 3: Conduct the study

Example 5: Reversal Design Step 4: Compute the p-value Analysis Team make a specification Intervention Team, are they correct? If the treatment has no effect, the probability that a masked visual analysts could have identified the assignments p = 1/32 = .03125

Example 6: Multiple Probe Design Step 1: Set study parameters Design Type? Multiple Probe with 5 Participants Minimums? At least 5 observations in each phase At least 3 consecutive observations prior to intervention At least 3 consecutive observations after an intervention Temporal staggers of at least 2 observations Randomization? Random assignment of treatment to blocks of observations, where there is one mystery block for each participant at the point the participant becomes eligible for intervention 25=32, so if correct with 5 blocks, p = 1/32 = .03125

Example 6: Multiple Probe Design Step 2: Split into two teams Step 3: Conduct the study

A B ? Dave ? John ? Bob ? Dan ? Theresa

Example 6: Multiple Probe Design Step 4: Compute the p-value Analysis Team make a specification Intervention Team, are they correct? Yes? p = 1/32 = .03125

Applications and Illustrations Byun, T. M., Hitchcock, E., & Ferron, J. M. (2017). Masked visual analysis: Minimizing type I error in response-guided single-case design for communication disorders. Journal of Speech, Language, and Hearing Research, 60, 1455-1466. DeLoatche, K. J. (2015). Parent-child interaction therapy as a treatment for ADHD in early childhood: A multiple baseline single-case design (Unpublished doctoral dissertation). University of South Florida, Tampa. Dickerson, E. (2016). Computerized cognitive remediation therapy (CCRT): Investigating change in the psychological and cognitive function of adolescent psychiatric patients. (Unpublished doctoral dissertation). Northeastern University, Boston. Ferron, J. M., & Levin, J. R. (2014). Single-case permutation and randomization statistical tests: Present status, promising new developments. In T. R. Kratochwill & J. R. Levin (Eds.), Single-case intervention research: Statistical and methodological advances (pp. 153-183). Washington, DC: American Psychological Association. Ferron, J., & Jones, P. K. (2006). Tests for the visual analysis of response-guided multiple-baseline data. Journal of Experimental Education, 75, 66-81. Ferron, J. M., Joo, S.-H., & Levin, J. R. (2017). A Monte-Carlo evaluation of masked-visual analysis in response-guided versus fixed-criteria multiple-baseline designs. Journal of Applied Behavior Analysis, 50, 701-716. Hinojosa, S. M. (2016). Teacher child interaction therapy: An ecological approach to intervening with young children who display disruptive behaviors. (Unpublished doctoral dissertation). University of South Florida, Tampa. Hua, Y., Yuan, C., Monroe, K., Hinzman, M. L., Alqahtani, S., Abdulmohsen, A., & Kern, A. M. (2016). Effects of the reread- adapt and answer-comprehend and goal setting intervention on decoding and reading comprehension skills of young adults with intellectual disabilities. Developmental Neurorehabilitation. Ottley, J. R., Coogle, C. G., Rahn, N. L., & Spear, C. (2017). Impact of bug-in-ear professional development on early childhood co-teachers’ use of communication strategies. Topics in Early Childhood Special Education, 36, 218-229. McKenney, E. L., Mann, K. A., Brown, D. L., & Jewell, J. D. (2017). Addressing cultural responsiveness in consultation: An empirical demonstration. Journal of Educational and Psychological Consultation. McKeown, D., Kimball, K., & Ledford, J. (2015). Effects of asynchronous audio feedback on the story revision practices of students with emotional/behavioral disorders. Education and Treatment of Children, 38, 541-564.