overview of visual analysis of single-case intervention data Research on visual analysis, Advantages and limitations Wendy Machalicek

Agenda Origins of visual analysis in applied behavior analysis Strengths and limitations of visual analysis Suggestions for use

Why not pie charts, bar charts, a table? Machalicek et al. in preparation

Historical use of graphical displays Statistical thinking rise in 18th and 19th century associated with rise in use of visuals Early research on how humans perceive statistical graphics (Spence & Lewandowsky, 1990) Preference for line graphs emerged as easy to read quickly and accurately

It began with skinner… “Where a reasonable degree of smoothness and reproducibility can be obtained with a few cases….there is little reason…to consider larger numbers” (p 442, Skinner, 1938) “The statistical approach is characterized by relatively unrefined methods of measurement and a general neglect of the problem of direct description; [whereas] the non-statistical approach confines itself to specific instances of behavior and to the development of methods of direct measurement and analysis (p. 443, Skinner, 1938)

Sidman, 1960

Defining Elements of Applied Behavior Analysis Baer, Wolf, & Risley, 1968 Applied (important dependent variable) Behavioral (valid & reliable measurement) Analytic (experimental design) Technological (replicable procedures) Conceptual (principles of behavior) Effective (socially important effect sizes) General (generalization and maintenance) Social Validity (continued use) If the application of behavioral techniques does not produce large enough effects for practical value, then application has failed.

Advantages to use of visual analysis Easy to create with current technology Data intimacy. Transformed as little as possible. Ideal for visually assessing interaction effects between two or more dimensions of behavior Quick visualization of results over time when paired with x and y axis When published, accessible Parsonson, B. S., Baer, D. M., Kratochwill, T. R., & Levin, J. R. (1992). The visual analysis of data, and current research into the stimuli controlling it. Single-case research design and analysis: New directions for psychology and education, 15-4

Issues with visual analysis Use of humans, rather than cumulative recorders to take data Observer drift, complexity of data collection (Kazdin, 1977) Percentage agreement insensitive to some target behavior (Kratochwill & Wetzel, 1977) Inductive process can be obscured w/statistical analysis Autocorrelation of data-inflated Type 1 errors (Busk & Marascuilo, 1988) Type II errors-especially when degree of change is small

Contextual variables affecting visual analysis Treatment acceptance (Spirrison & Maundy, 1994) Participant characteristics (Grigg, Snell, & Loyd, 1989) Social significance of effects (DeProspero & Cohen, 1979) Charting and scaling of y-axis (Furlong & Wampold, 1982)    

McDuffie, A. , Bullard, L. , Nelson, S. , Machalicek, W McDuffie, A., Bullard, L., Nelson, S., Machalicek, W., & Abbeduto, L. (2016). A spoken language intervention for school-aged boys with fragile X syndrome. American Journal on Intellectual and Developmental Disabilities, 121(3), 236-265.

Hansen, S. , Raulston, T. , Machalicek, W. , & Frantz, R. (In press) Hansen, S., Raulston, T., Machalicek, W., & Frantz, R. (In press). Caregiver-mediated joint attention intervention. Behavioral Interventions.

Machalicek et al. in preparation

How are we really doing? IRA = .58 Ottenbacher et al. (1993) Overall weighted mean proportion IRA = .76 for all 32 effects (Ninci et al, 2015) Ninci only included peer reviewed studies, scales reduced for comparison, proportional effect sizes only. Potential moderators evaluated included (a) design families, (b) rater expertise, (c) the provision of contextual information for graphs, (d) the use of visual aids, (e) the provision of an operational definition of the construct being rated, and (f) rating scale ranges. Potential moderators evaluated included (a) design families, (b) rater expertise, (c) the provision of contextual information for graphs, (d) the use of visual aids, (e) the provision of an operational definition of the construct being rated, and (f) rating scale ranges.

What impacts interrater reliability of visual analysis?

What do we know about accuracy & interrater agreement (IRA)? Novice raters may have difficulty with variable data, trend or intervention effect is small and with extreme values (Nelson et al., 2017) Raters may be more accurate with positive versus negative trends (Christ et al., 2014) The use of visual aids may be the most concrete beneficial strategy we have thus far for improving IRA (e.g., Roane et al., 2013) (Ninci et al., 2015) Ninci and colleagues suggested that experienced individuals may have variety of backgrounds and training in SCR with different areas of emphasis. It will be interesting to see the impact of additional training (e.g. WWC models) on IRA.

Risley, 2001

References Baer, D. M., Wolf, M. M., & Risley, T. R. (1968). Some current dimensions of applied behavior analysis. Journal of Applied Behavior Analysis, 1(1), 91-97. Busk,PL.,&Marascuilo,L.A.(1988).Autocorrelation in single-subject research: A counterargument to the myth of no autocorrelation. Behavioral Assessment, 10, 229-242. Fisch, G.S. (1998). Visual inspection of data revisited: Do the eyes still have it? The Behavior Analyst, 21, 111- 123. Hollands, J.G., & Spence, I. (1992). Judgments of change and proportion in graphical perception. Human Factors: The Journal of the Human Factors and Ergonomics Society, 34, 313. Kratochwill, T. R., & Wetzel, R. J. (1977). Observer agreement, credibility, and judgement: Some considerations in presenting observer agreement data. Journal of Applied Behavior Analysis, 10, 133-139. Parsonson, B. S., Baer, D. M., Kratochwill, T. R., & Levin, J. R. (1992). The visual analysis of data, and current research into the stimuli controlling it. Single-case research design and analysis: New directions for psychology and education, 15-4 Spence, I., and Lewandowsky, S. (1991). Displaying proportions and percentages. Applied Cognitive Psychology, 5,61-77.