Evaluating Intervention Effects Chapter Six Evaluating Intervention Effects

OBJECTIVES Summarize and graph or chart data using techniques appropriate for the data. Visually analyze graphed data and write data decision rules. Identify the major types of single subject research designs and give the uses and limitations of each.

Graphing and Charting Data Why? Continuous monitoring of performance, facilitate decision making Formative evaluation of the effectiveness of instruction

Types Graph One or two dependent variables Bar, cumulative, frequency polygon, equal interval, equal ratio Progress Chart Several variables monitored Performance

Components of Graphs The effects of IV on DV for student Ordinate Descriptive Title Ordinate Baseline Intervention 1 Intervention 2 Condition label Data path Data Points dependent variable Condition Line time Abscissa Legend Billy = Sally =

Graphing Conventions Do not connect non-consecutive points Baseline Reinforcement Extinction Label all phases and axes dependent variable Connect consecutive data points Use different symbols and plot lines for different data 0- Do not connect across phase lines time Separate 0 from abscissa Billy = Sally =

Lines of Progress Trend lines Aim lines Visual estimates of future performance Aim lines Visual representation of performance Based on criteria and allotted time of STO Line starts from last 3 days of baseline data Accelerating and decelerating

Middle School Math: Revisited Mean: The average of a set of numbers Mode: The most frequent of a group of numbers Median: The mid point of a set of sequenced numbers 2 What is the Median of these numbers? 1, 2, 5 7.5 What is the Median of these numbers? 4, 7, 8, 10 1 What is the Median of these numbers? 1, 1, 998

Trend lines represent the trend of data within each condition How to generate trend lines: Count total data points Draw vertical line to divide points in half Mid-date: draw vertical line at mid-date point. (repeat on each side of vertical line) Mid-rate: draw horizontal line at mid-rate Connect intersections of mid-rates/dates in each phase

TREND LINE: STEP 1

TREND LINE: STEP 2

TREND LINE: STEP 3

TREND LINE: STEP 4

TREND LINE EXERCISE A

TREND LINE EXERCISE A - Solution

TREND LINE EXERCISE B

TREND LINE EXERCISE B - Solution

TREND LINE EXERCISE C

TREND LINE EXERCISE C - Solution

TREND LINE EXERCISE D

TREND LINE EXERCISE D - Solution

TREND LINE EXERCISE E

TREND LINE EXERCISE E - Solution

TREND LINE EXERCISE F

TREND LINE EXERCISE F - Solution

How to generate aim lines: Aim lines represent the goal or objective and keep us on track toward that goal How to generate aim lines: Determine the aim date and aim rate based on the criteria expressed in the student’s long-term objective Draw an aim star (an A right side up for accelerating target, an A upside down for decelerating target) at the desired rate and date intersection.

Determining Aim Lines Baseline Number Hand raises A • • • • • • Time

Determining Aim Lines Determine the mid-date and mid-rate of the LAST THREE DAYS OF BASELINE DATA POINTS. Mid-date and mid-rate are the median or middlemost points. Don’t average the points, simply count the # and take the middle. Mid-date: count left to right. Mid-rate: count bottom to top. Baseline # hand raises A • • • • • • Time

Determining Aim Lines Draw an aim line through the mid-date and the mid-rate intersection to the aim star. Baseline # hand raises A • • • • • • Time

AIM LINE EXERCISE A Baseline • • • • Behavior • Time

A AIM LINE EXERCISE A - Solution Baseline • • • • • Behavior • • Time

AIM LINE EXERCISE B Baseline Behavior • • • A Time

AIM LINE EXERCISE B - Solution Baseline A Behavior • • • Time

AIM LINE EXERCISE C Baseline Behavior • • • • • • Time

AIM LINE EXERCISE C - Solution Baseline A Behavior • • • • • • Time

Objectives & Monitoring Given the prompt, “point to the…,” Richie will point to the designated object within 5 secs for 10 of 10 trials over 10 consecutive sessions by the 7th day of intervention. How will we measure progress toward this objective? Event recording--controlled presentations (correct response w/in 5 secs).

Ritchie’s Progress 2 4 8 6 10 A Correct Trials Days

Using Data During Intervention Data patterns (within and across conditions) Variability more stable = more predictive look for cyclical patterns (e.g., only on Monday, when reading, etc.) Level changes indicate possible change in functional relationships or influencing factors Trend changes predict future performance indicate possible change in functional relationship

NO CHANGE

CHANGE IN TREND

CHANGE IN LEVEL

CHANGE IN LEVEL AND TREND

Data-Based Decision Making

Data Decision Rules Define adequate progress and dictate when changes are to be made Determined before you intervene Basic “three-day rule”

Intervention Data Patterns & Decisions Decisions made by comparing data with Aim Line Make no change (data at or better than aim) Change goal or aim date (break ddr) Slice back (slight - misrule during instruction) Step back (large - teach prerequisite) Move to new procedure (add/fade prompts) Move to new skill (next skill in hierarchy) Begin compliance training (R+ and error correction) Move to new phase of learning

Use Data to Analyze Errors Need to know why errors occur to plan for future instruction or intervention Random errors: irregular pattern, large discrepancy Systematic errors: consistent, misrule Compliance errors: variable, sharp turn-down Unlearned prerequisites: no progress

DATA SHOWING NEED FOR COMPLIANCE TRAINING

DATA SHOWING NEED TO MOVE TO NEW SKILL

DATA SHOWING THE NEED TO SLICE BACK - RETEACH

DATA SHOWING THE NEED TO STEP BACK - PRETEACH

Using Data After Intervention (post-maintenance) Data used to: Evaluate intervention effectiveness Identify functional relationships between Independent Variable (IV) and Dependent Variable (DV)

Single Subject Research Designs

Single Subject Research Designs Advantages Student serves as his/her own control High internal validity Clear demonstration of experimental control Control for threats to internal validity Small n Formative evaluation Disadvantage Low external validity-replications required

The Logic of Single Subject Research Designs Baseline data are an assessment of the current level of performance, and provide a basis for predicting future performance Data collected during intervention condition are an assessment of the effect of the intervention and a prediction of future performance However, an A and a B condition do not demonstrate experimental control (a functional relationship)--competing explanations for the observed effects cannot be ruled out Therefore, a control condition (in this case, baseline) is implemented to demonstrate that intervention was responsible for observed effects of IV on DV

Types of Designs Selection Based upon the questions and the conditions of the setting The most valid designs may not lend themselves to specific situations/settings Symbols A = baseline condition B = a first intervention condition: manipulation of IV C - Zx = successive manipulations of IV(s)

Case Study Designs - AB Baseline followed by intervention Example: “everyday teaching with data” Advantages Good for hypothesis/question development Fits teaching model Disadvantages Cannot identify functional relationship Many threats to internal validity Cannot reverse if learning occurred

AB Design Baseline Intervention 180 150 120 Number of Seconds 90 60 30   150         120  Number of Seconds 90 60 30          2 4 6 8 10 12 14 16 18 20 Sessions

Changing Conditions (ABCD) Each phase contains a completely different Independent Variable Advantages Evaluate effects of several different interventions Disadvantages Doesn’t identify functional relationships Ordering effects Requires withdrawal of potentially effective intervention Confounded by learning

Changing Conditions 100 . . . 5 10 15 20 25 30 35 40 45 50 55 60 65 70 75 80 85 90 95 . . . . . . . . . . . . . . Goal . . . . . . . . . . . . % Time on Task . . . Assigned . seats Praise- Rules, chair arrangement, Non-contingent free time Ignoring Office Threat Jim tutors & models praise Contingent Free Time "No tutoring" Day 28: Jim stops tutoring (+ a party contingency on day 21) Day 29: Complete reversal continued Fading or transfer of data collection Baseline to teacher A B C D A D/E 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 Observation Days (12:00-1:00p.m.)

Withdrawal (ABAB) Put IV in and take it away (repeat) -Most frequently used SSD in behavior mod. -Aka “replication” or “reversal” design Advantages Identifies functional relationships Can be a teaching design Disadvantages Requires return to baseline Confounded by learning

Withdrawal Baseline (A) Time-Out (B) Baseline2 (A)2 Time-Out2 (B)2 20 . . . . . 15 . . . . . . . Number Per Day . . . 10 . . . . . . . . . . . May skip first baseline if clear history of behavior or behavior is extremely dangerous 5 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 10 15 20 25 30 35 40 45 50 55 Days

Changing Criterion -An AB design with the B sub-phases -Used to evaluate shaping programs of behavior toward a terminal objective Advantages No withdrawal or reversal required Fits will into instructional program Can establish a functional relationship Disadvantages Not for behaviors requiring immediate change ID of appropriate criterion steps may be difficult

Changing Criterion Criterion Baseline Intervention 10 9 8 7 6 5  9    8     7 6       5 Criterion Total Latency (in minutes)  4     3 Vary length of condition and criterion level to demonstrate functional relationship   2              1 2 4 6 8 10 12 14 16 18 20 22 24 26 28 30 32 34 Sessions

Multiple Baseline Identify 3 distinct students, settings, or behaviors -collect data on each while implementing on only one at a time, using the others as control Advantages Not confounded by learning Return to baseline not required Can also be used as “multiple probe” Disadvantages Prolonged baselines Don’t use if behavior can’t be tolerated

Multiple Baseline (across students) Intervention 100 80 60 40 20    Subject 1                             100 80 60 40 20  Percent Correct Subject 2                      100 80 60 40 20                     Subject 3             2 4 6 8 10 12 14 16 18 20 22 24 26 28 30 32

Multiple Baseline (across behaviors) 100 80 60 40 20 A B  Rate Per Minute Throwing Hitting Spitting Sessions

Multiple Baseline (across settings) Intervention Latency (in minutes) Condition 1 Condition 2 Condition 3 Sessions 16 12 8 4 2 4 6 8 10 12 14 16 18 20 22 24 26 28 30 32 34 36 38 40 

Multiple Probe Across Behaviors 100 80 60 40 20       (5) (4)  Word Group 1 (3)      100 80 60 40 20       (5) (4) PERCENT CORRECT Word Group 2  (1) (3)     100 80 60 40 20      (5) (4) Word Group 3   (1) (2) (3)      1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 DAYS

Alternating Treatments -Alternate number of IV through multiple sessions -Can follow with reversal design for one IV Advantages Demonstrate relative effectiveness of treatments Minimize sequencing effects Functional relationship if reversed with one variable Disadvantages Does not identify functional relationship Treatments must be independent and not interactive

Alternating Treatments 15 20 30 25 35 45 40 50 60 55 2 4 6 8 10 12 14 16 18 20 22 24 26 28 30 5 10    = Intervention 1 = Intervention 2 = Intervention 3 Baseline Intervention Number of Events Sessions