Project VIABLE: Overview of Directions Related to Training to Enhance Adequacy of Data Obtained through Direct Behavior Rating (DBR) Sandra M. Chafouleas 1, T. Chris Riley-Tillman 2, Theodore J. Christ 3, & George Sugai 1 University of Connecticut 1 East Carolina University 2 University of Minnesota 3 PROJECT GOALS For additional information, please visit correspondence regarding the project should be directed to Sandra Chafouleas at Phase I: Foundations of Measurement (Instrumentation and Procedures) How should the DBR scale be comprised? (e.g., Likert-type versus continuous), How should assessment items be worded, and how many items should be included for each target behavior?, How many observations and/or what duration should be used for each session?, How long should the DBR observation rating period be in order to obtain a reliable estimate of the behavior? Phase II: Decision Making and Validity In response to an intervention, does the DBR data correspond to data obtained via systematic direct observations, ODRs, traditional rating scales?, Can DBR data be used to screen a classroom and reliably identify students at-risk (engagement, disruption)?, Can DBR data be used in tri-annual assessments to assess level and rate of student behavior? Phase III: Feasibility (Training and Use, Perceived Usability) What does DBR training require?*, How intrusive are procedures?, How well do procedures and instrumentation generalize to teacher assessments within classroom settings?, How do consumers of DBR perceive its usability?, How acceptable is DBR?, How useful are DBR outcomes for school-based decisions? * Content for this poster. FINDINGS TO DATE Rater training may be needed to enhance outcomes, particularly for certain individuals Training to enhance accurate rating of mid-scale levels of behavior most challenging yet important Training need not necessarily be high intensity (lengthy) Components likely to be beneficial include direct training with: Overview with clear review of definitions and modeling of rating Practice and immediate feedback Behavior examples that utilize the scale range (low, medium, high) Impact of incorporating “frame of reference” and “error training” still to be determined NEXT STEPS Would our current findings related to instrumentation/procedures be improved if these training components were included? All future work will include training beyond familiarization What else might be done to enhance a) outcomes for variable individuals, and b) mid-scale rating accuracy? Following training, what happens to rating accuracy over time? How do we ensure training access to all? Develop an on-line dynamic training module What is the influence of rater on data obtained from DBR? In the “absence of training”, work to date under Phase 1 has suggested: Generally, profiles of aggregated DBR are consistent across raters (when averaged within student across occasion or across students across occasions) Reliable estimates of level can be established with relatively few observations (5-10 for low stakes, for high stakes) completed by the same rater, BUT some individual raters do not fall within these guidelines Individual raters seem to anchor ratings within a range of gradients and then subsequent ratings are made relative to that range When looking across raters, preliminary evidence supports systematic bias in rating. What components to training might be included to enhance outcomes? Direct training – active modeling and rehearsal as opposed to didactic or written Opportunities for practice and feedback – direct practice and immediate corrective feedback Intensity of training – how much training is sufficient? Explanation using “frame of reference” – provide clear rationale for rating, including qualitative descriptions Rater error training – definitions/examples of types of error to avoid when rating, such as leniency/severity, halo SUMMARY POINTS: Direct – Rating occurs in close proximity to the time and place of the observation. Thus, the rater must observe the target for a “sufficient” portion of the observation period. Behavior – The target of rating must be well-defined and accessible for observation. Rating – The rating component quantifies rater perception of the target behavior. Defining Direct Behavior Rating The purpose of the study was to examine whether providing users of DBR with a training session utilizing practice and performance feedback would increase rating accuracy. Method involved instructing 59 undergraduate students to watch video clips of student behavior and then rate target behaviors using DBR in one of two conditions: No training: 23 minute overview of DBR and behavior assessment Training: 23 minute instruction on DBR use, with practice and feedback In summary, results were consistent with initial hypotheses in that ratings conducted by trained participants were more accurate than those conducted by untrained participants. Specifically, using standard difference scores or Cronbach’s differential accuracy, raters in the training condition were significantly more accurate than those in the brief familiarization condition for ratings of academic engagement and disruption. As such, training resulted in higher levels of absolute or rank order accuracy. There was also far less variability among the ratings completed by the trained group than by the brief familiarization (no training) group. Schlientz, M.D., Riley-Tillman, T.C., Briesch, A.M., Walcott, C.M., & Chafouleas, S.M. (in press). The impact of training on the accuracy of Direct Behavior Ratings (DBRs). School Psychology Quarterly. STUDY 2 The purpose of the study was to examine whether DBR rating accuracy is significantly impacted by the type of training package. Research questions include: a) Does the addition of frame of reference (FOR) to standard training (ST) improve rater accuracy over and above ST alone?, b) Does the addition of rater error training (RET) to FOR and ST suggest an improvement in rater accuracy over and above ST alone and ST+FOR, and c) Does the addition of increased exposure, defined as opportunity for practice/feedback (2x more = 6x), improve rater accuracy over the standard 3x? Method involved undergraduates watching and rating video clips of student behavior in one of 6 possible conditions (described above). Clips were purposefully chosen to reflect a range of ratings on the DBR scale, and a composite accuracy score for each behavior will be used in analysis. Two-way ANOVA (2 Levels of Exposure X 3 Types of Training) will be conducted to test for differences among groups with regard to differential accuracy (the outcome). Data analyses in process. Chafouleas, S.M., Riley-Tillman, T.C., Kilgus, S.P., Amon, J., Jaffery, G. & Brooks, S. (in preparation). Critical components of DBR training to enhance rater accuracy: An investigation of training content and exposure. STUDY 1 BACKGROUND STUDY 3 STUDY 4 The purpose of this study was to examine whether direct training procedures resulted in greater DBR accuracy than either indirect training or no training. In addition, teacher acceptability of DBR in behavior assessment was assessed. Method involved having middle/high school teachers from a private school watch video clips of student behavior, and then rate academic engagement and disruption using DBR. Teachers were assigned to one of 3 conditions: No training: brief overview of study procedures Indirect: 20 minute instruction on DBR use, with example ratings Direct: 20 minute instruction on DBR use, with practice and feedback In summary, chi square analyses suggested that direct training did not improve rating accuracy. Moderate acceptability for DBR was found. Questions about selection and ordering of video samples during training were raised as initial exposure to mid-scale behavior ratings may create rater frustration. LeBel, T.J., Kilgus, S.P., Briesch, A.M., & Chafouleas, S.M. (in press). The impact of training on the accuracy of teacher-completed Direct Behavior Ratings (DBRs). Journal of Positive Behavioral Interventions. In this study, the extent to which DBR training with different levels of practice and performance feedback impacts rating accuracy was examined. Method involved undergraduates watching and rating video clips of student behavior in one of 3 conditions: No training: Overview of DBR and behavior assessment Training: Overview of DBR and instruction on DBR use, with practice and feedback over 3 opportunities Extended Training: Same but 6 opportunities In summary, initial analysis indicated no significant difference in accuracy between training conditions or at one week re-test. Further analyses compared training vs. control (no training) across three base-rate levels (low, medium and high) for each behavior. Training did not substantially improve overall ability to rate academic engagement but ratings were more accurate with high academic engagement, as demonstrated by lower systematic accuracy difference scores. Training significantly improved accuracy when rating all levels of disruptive behavior, and across conditions, accuracy was highest when disruptive behavior was low or high. Participants rated compliance with significantly less accuracy when targets displayed medium rates of compliance. Riley-Tillman, T.C. Harrison, S, Amon, J., & Brooks, S. (in preparation). An investigation of rating accuracy on DBR following training involving practice and feedback. Example single-item DBR scale