Single-Case Research Designs: Training Protocols in Visual Analysis Wendy Machalicek University of Oregon Acknowledgement: Rob Horner Tom Kratochwill Joel Levin Chris Swoboda George Sugai Hariharan Swaminathan Dan Maggin

Specific Goals Review training protocols for competency and fluency in visual analysis

Single-Case Research Designs: Training Protocols in Visual Analysis Wendy Machalicek University of Oregon Acknowledgement: Rob Horner Tom Kratochwill Joel Levin Chris Swoboda George Sugai Hariharan Swaminathan Dan Maggin

Specific Goals Review training protocols for competency and fluency in visual analysis

Context: Is visual analysis reliable across analysts? Some have found visual analysis to be unreliable, but these studies have been conducted with novice analysts e.g. Liberman, Yoder, Reichow, & Wolery (2010); Ottenbacher (1993) When analysts are trained and experienced, we see high agreement Kahng et al. (2010)

WWC White Paper Single-Case Intervention Research Design Standards Thomas R. Kratochwill University of Wisconsin-Madison John H. Hitchcock Ohio University Robert H. Horner University of Oregon Joel R. Levin University of Arizona Samuel M. Odom University of North Carolina at Chapel Hill David M. Rindskopf City University of New York William R. Shadish University of California Merced Available at “Materials”

What is needed Better standards for training in single-case methods More precision in RFA stipulations Establishes expectations for reviewers Better standards for reviewing single-case Journal editors Grant reviewers Development of meta-analysis technology

Evaluate the Design Meets Design StandardsMeets with ReservationsDoes Not Meet Design Standards Evaluate the Evidence Strong EvidenceModerate EvidenceNo Evidence Effect-Size Estimation Social Validity Assessment Does the Design allow Documentation of Experimental Control ? Do the data document Experimental Control ? Is the effect something we should care about? Stop Stop

Visual analysis of single case designs Determine viability of design first ①Assess Baseline ②Assess the within phase pattern of each phase ③Assess the basic effects with each phase comparison ④Assess the extent to which the overall design documents experimental control E.g. Three demonstrations of basic effect, each at a different point in time Assess effect size Assess social validity

Assessing within phase “pattern” and Between phase “basic effect” Level Trend Variability + Overlap Immediacy of Effect Consistency across similar phases _________________________________________ Other: vertical analysis; intercept gap Within Phase Between Phases

Effective training protocols Fisher, Kelley, & Lomas (2003) Train practitioners to use dual-criterion method with AB graphs Baseline split-middle and mean level lines projected onto treatment phase Specified proportion of data must fall above/below both lines Visual analysis improved when graphs included criterion lines, have written rules, and brief verbal instruction/modeling training provided

Effective training protocols Stewart, Carr, Brandt, & McHenry (2007) Train practitioners to use (1) 8-min recorded lecture, demos, conservative dual-criterion method with AB graphs & (2) 12-min recorded lecture Lecture alone didn’t improve accuracy Visual analysis improved to 80% accuracy when graphs included criterion lines, have written rules, and brief verbal instruction/modeling training provided

Effective training protocols Wolfe & Slocum (2015) Random assignment of participants to (a) computer-based instruction, (b) 36 min recorded lecture or (c) control condition AB graphs varied on trend and level

Training Protocol for Visual Analysis of Single-case data Development of visual analysis graphs Rob Horner, Bruce Wampold, Richard Freund, Rick Albin, Tom Kratochwill, Chris Swoboda Level, trend, variability, overlap Purpose is to emphasize interaction Examine and score each graph (ABAB, MBD, ATD graph sets) Use ALL data in the graph Combine assessment of: Level, trend, variability, overlap, immediacy of effect, consistency of data patterns in similar phases. Use scoring metric: 1 = no functional relation 2,3,4 = considerable weaknesses 5 = moderate functional relation (publishable) 7 = strong functional relation Compare scores with Excel file Median scores from five “expert” Single-case Researchers

Using Anyone, any time may use the website to assess visual analysis competency For Instructors Contact Rob with course code, and dates needed Set up “coordinator” Students enter code and name You can download their scores, number of sessions, time For Research Inter-rater agreement with visual analysis Training protocols for teaching visual analysis.

Visual analysis of single case designs Determine viability of design first Assess Baseline Assess the within phase pattern of each phase Assess the basic effects with each phase comparison Assess the extent to which the overall design documents experimental control E.g. Three demonstrations of basic effect, each at a different point in time Assess effect size Assess social validity

Screen shot of results

Summary Visual Analysis of Single Case Designs Determine viability of design first Assess Baseline Assess the within phase pattern of each phase Assess the basic effects with each phase comparison Assess the extent to which the overall design documents experimental control E.g. Three demonstrations of basic effect, each at a different point in time Assess effect size Assess social validity YOUR TURN: The full set of 140 graphs

ACTIVITY

Design activity Design a single-case research or Group experimental design evaluating use of or another training protocol to improve participant acquisition, fluency of visual analysis in single-case research