Monitoring Student Progress at the Secondary Level Jim Wright www Monitoring Student Progress at the Secondary Level Jim Wright www.interventioncentral.org

“Few agree on an appropriate curriculum for secondary students…; thus it is difficult to determine in what areas student [academic] progress should be measured.” -- Espin & Tindal (1998) Source: Espin, C. A., & Tindal, G. (1998). Curriculum-based measurement for secondary students. In M. R. Shinn (Ed.) Advanced applications of curriculum-based measurement. New York: Guilford Press.

RTI Literacy: Assessment & Progress-Monitoring To measure student ‘response to instruction/intervention’ effectively, the RTI model measures students’ academic performance and progress on schedules matched to each student’s risk profile and intervention Tier membership. Benchmarking/Universal Screening. All children in a grade level are assessed at least 3 times per year on a common collection of academic assessments. Strategic Monitoring. Students placed in Tier 2 (supplemental) reading groups are assessed 1-2 times per month to gauge their progress with this intervention. Intensive Monitoring. Students who participate in an intensive, individualized Tier 3 intervention are assessed at least once per week. Source: Burns, M. K., & Gibbons, K. A. (2008). Implementing response-to-intervention in elementary and secondary schools: Procedures to assure scientific-based practices. New York: Routledge.

Measuring General vs. Specific Academic Outcomes General Outcome Measures: Track the student’s increasing proficiency on general curriculum goals such as reading fluency. An example is CBM-Oral Reading Fluency (Hintz et al., 2006). Specific Sub-Skill Mastery Measures: Track short-term student academic progress with clear criteria for mastery. An example is CBA-Math Computation Fluency (Burns & Gibbons, 2008). Sources: Burns, M. K., & Gibbons, K. A. (2008). Implementing response-to-intervention in elementary and secondary schools: Procedures to assure scientific-based practices. New York: Routledge. Hintz, J. M., Christ, T. J., & Methe, S. A. (2006). Curriculum-based assessment. Psychology in the Schools, 43, 45-56.

Local Norms: Screening All Students (Stewart & Silberglit, 2008) Local norm data in basic academic skills are collected at least 3 times per year (fall, winter, spring). Schools should consider using ‘curriculum-linked’ measures such as Curriculum-Based Measurement that will show generalized student growth in response to learning. If possible, schools should consider avoiding ‘curriculum-locked’ measures that are tied to a single commercial instructional program. Source: Stewart, L. H. & Silberglit, B. (2008). Best practices in developing academic local norms. In A. Thomas & J. Grimes (Eds.), Best practices in school psychology V (pp. 225-242). Bethesda, MD: National Association of School Psychologists.

Local Norms: Using a Wide Variety of Data (Stewart & Silberglit, 2008) Local norms can be compiled using: Fluency measures such as Curriculum-Based Measurement. Existing data, such as office disciplinary referrals. Computer-delivered assessments, e.g., Measures of Academic Progress (MAP) from www.nwea.org Source: Stewart, L. H. & Silberglit, B. (2008). Best practices in developing academic local norms. In A. Thomas & J. Grimes (Eds.), Best practices in school psychology V (pp. 225-242). Bethesda, MD: National Association of School Psychologists.

Measures of Academic Progress (MAP) www.nwea.org

Applications of Local Norm Data (Stewart & Silberglit, 2008) Local norm data can be used to: Evaluate and improve the current core instructional program. Allocate resources to classrooms, grades, and buildings where student academic needs are greatest. Guide the creation of targeted Tier 2 (supplemental intervention) groups Set academic goals for improvement for students on Tier 2 and Tier 3 interventions. Move students across levels of intervention, based on performance relative to that of peers (local norms). Source: Stewart, L. H. & Silberglit, B. (2008). Best practices in developing academic local norms. In A. Thomas & J. Grimes (Eds.), Best practices in school psychology V (pp. 225-242). Bethesda, MD: National Association of School Psychologists.

Apply the ‘80-15-5’ Rule to Determine if the Focus of the Intervention Should Be the Core Curriculum, Subgroups of Underperforming Learners, or Individual Struggling Students (T. Christ, 2008) If less than 80% of students are successfully meeting academic or behavioral goals, the intervention focus is on the core curriculum and general student population. If no more than 15% of students are not successful in meeting academic or behavioral goals, the intervention focus is on small-group ‘treatments’ or interventions. If no more than 5% of students are not successful in meeting academic or behavioral goals, the intervention focus is on the individual student. Source: Christ, T. (2008). Best practices in problem analysis. In A. Thomas & J. Grimes (Eds.), Best practices in school psychology V (pp. 159-176).

Local Norms: Supplement With Additional Academic Testing as Needed (Stewart & Silberglit, 2008) “At the individual student level, local norm data are just the first step toward determining why a student may be experiencing academic difficulty. Because local norms are collected on brief indicators of core academic skills, other sources of information and additional testing using the local norm measures or other tests are needed to validate the problem and determine why the student is having difficulty. … Percentage correct and rate information provide clues regarding automaticity and accuracy of skills. Error types, error patterns, and qualitative data provide clues about how a student approached the task. Patterns of strengths and weaknesses on subtests of an assessment can provide information about the concepts in which a student or group of students may need greater instructional support, provided these subtests are equated and reliable for these purposes.” p. 237 Source: Stewart, L. H. & Silberglit, B. (2008). Best practices in developing academic local norms. In A. Thomas & J. Grimes (Eds.), Best practices in school psychology V (pp. 225-242). Bethesda, MD: National Association of School Psychologists.

How Does a Secondary School Determine a Student’s Math Competencies? “Tests [to assess secondary students’ math knowledge] should be used or if necessary developed that measure students’ procedural fluency as well as their conceptual understanding. Items should range in difficulty from simple applications of the algorithm to more complex. A variety of problem types can be used across assessments to tap students’ conceptual knowledge.” p. 469 Source: Ketterlin-Geller, L. R., Baker, S. K., & Chard, D. J. (2008). Best practices in mathematics instruction and assessment in secondary settings. In A. Thomas & J. Grimes (Eds.), Best practices in school psychology V (pp.465-475).

Making Use of Existing (‘Extant’) Data

Extant (Existing) Data (Chafouleas et al., 2007) Definition: Information that is collected by schools as a matter of course. Extant data comes in two forms: Performance summaries (e.g., class grades, teacher summary comments on report cards, state test scores). Student work products (e.g., research papers, math homework, PowerPoint presentation). Source: Chafouleas, S., Riley-Tillman, T.C., & Sugai, G. (2007). School-based behavioral assessment: Informing intervention and instruction. New York: Guilford Press.

Advantages of Using Extant Data (Chafouleas et al., 2007) Information is already existing and easy to access. Students are less likely to show ‘reactive’ effects when data is collected, as the information collected is part of the normal routine of schools. Extant data is ‘relevant’ to school data consumers (such as classroom teachers, administrators, and members of problem-solving teams). Source: Chafouleas, S., Riley-Tillman, T.C., & Sugai, G. (2007). School-based behavioral assessment: Informing intervention and instruction. New York: Guilford Press.

Drawbacks of Using Extant Data (Chafouleas et al., 2007) Time is required to collate and summarize the data (e.g., summarizing a week’s worth of disciplinary office referrals). The data may be limited and not reveal the full dimension of the student’s presenting problem(s). There is no guarantee that school staff are consistent and accurate in how they collect the data (e.g., grading policies can vary across classrooms; instructors may have differing expectations regarding what types of assignments are given a formal grade; standards may fluctuate across teachers for filling out disciplinary referrals). Little research has been done on the ‘psychometric adequacy’ of extant data sources. Source: Chafouleas, S., Riley-Tillman, T.C., & Sugai, G. (2007). School-based behavioral assessment: Informing intervention and instruction. New York: Guilford Press.

Universal Screening at Secondary Schools: Using Existing Data Proactively to Flag ‘Signs of Disengagement’ “Across interventions…, a key component to promoting school completion is the systematic monitoring of all students for signs of disengagement, such as attendance and behavior problems, failing courses, off track in terms of credits earned toward graduation, problematic or few close relationships with peers and/or teachers, and then following up with those who are at risk.” Source: Jimerson, S. R., Reschly, A. L., & Hess, R. S. (2008). Best practices in developing academic local norms. In A. Thomas & J. Grimes (Eds.), Best practices in school psychology V (pp. 1085-1097). Bethesda, MD: National Association of School Psychologists. p.1090

Mining Archival Data: What Are the ‘Early Warning Flags’ of Student Drop-Out? A sample of 13,000 students in Philadelphia were tracked for 8 years. These early warning indicators were found to predict student drop-out in the sixth-grade year: Failure in English Failure in math Missing at least 20% of school days Receiving an ‘unsatisfactory’ behavior rating from at least one teacher Source: Balfanz, R., Herzog, L., MacIver, D. J. (2007). Preventing student disengagement and keeping students on the graduation path in urban middle grades schools: Early identification and effective interventions. Educational Psychologist,42, 223–235. .

What is the Predictive Power of These Early Warning Flags? Number of ‘Early Warning Flags’ in Student Record Probability That Student Would Graduate None 56% 1 36% 2 21% 3 13% 4 7% Source: Balfanz, R., Herzog, L., MacIver, D. J. (2007). Preventing student disengagement and keeping students on the graduation path in urban middle grades schools: Early identification and effective interventions. Educational Psychologist,42, 223–235. .

‘Elbow Group’ Activity: What Extant/Archival Data Should Your RTI Team Review Regularly? Discuss the essential extant/archival data that your RTI Team should review as ‘early warning indicators’ of students who are struggling. What process should your school adopt to ensure that these data are reviewed regularly (e.g., every five weeks) to guarantee timely identification of students who need intervention assistance?

Grades as a Classroom-Based ‘Pulse’ Measure of Academic Performance

Grades & Other Teacher Performance Summary Data (Chafouleas et al Teacher test and quiz grades can be useful as a supplemental method for monitoring the impact of student behavioral interventions. Other data about student academic performance (e.g., homework completion, homework grades, etc.) can also be tracked and graphed to judge intervention effectiveness. Source: Chafouleas, S., Riley-Tillman, T.C., & Sugai, G. (2007). School-based behavioral assessment: Informing intervention and instruction. New York: Guilford Press.

Marc Ripley (From Chafouleas et al., 2007) 2-Wk 4-Wk 6-Wk 8-Wk 10-Wk 12-Wk 9/23/07 10/07/07 10/21/07 11/03/07 11/20/07 12/05/07 Source: Chafouleas, S., Riley-Tillman, T.C., & Sugai, G. (2007). School-based behavioral assessment: Informing intervention and instruction. New York: Guilford Press.

Online Grading Systems

Assessing Basic Academic Skills: Curriculum-Based Measurement

Curriculum-Based Measurement: Advantages as a Set of Tools to Monitor RTI/Academic Cases Aligns with curriculum-goals and materials Is reliable and valid (has ‘technical adequacy’) Is criterion-referenced: sets specific performance levels for specific tasks Uses standard procedures to prepare materials, administer, and score Samples student performance to give objective, observable ‘low-inference’ information about student performance Has decision rules to help educators to interpret student data and make appropriate instructional decisions Is efficient to implement in schools (e.g., training can be done quickly; the measures are brief and feasible for classrooms, etc.) Provides data that can be converted into visual displays for ease of communication Source: Hosp, M.K., Hosp, J. L., & Howell, K. W. (2007). The ABCs of CBM. New York: Guilford.

Assessing Basic Academic Skills: Curriculum-Based Measurement Reading: These 3 measures all proved ‘adequate predictors’ of student performance on reading content tasks: Reading aloud (Oral Reading Fluency): Passages from content-area tests: 1 minute. Maze task (every 7th item replaced with multiple choice/answer plus 2 distracters): Passages from content-area texts: 2 minutes. Vocabulary matching: 10 vocabulary items and 12 definitions (including 2 distracters): 10 minutes. Source: Espin, C. A., & Tindal, G. (1998). Curriculum-based measurement for secondary students. In M. R. Shinn (Ed.) Advanced applications of curriculum-based measurement. New York: Guilford Press.

Oral Reading Fluency Probe Sample http://www. rti2

Maze Probe Sample http://www.rti2.org/rti2/mazes

Curriculum-Based Evaluation: Math Vocabulary Format Option 1 20 vocabulary terms appear alphabetically in the right column. Items are drawn randomly from a ‘vocabulary pool’ Randomly arranged definitions appear in the left column. The student writes the letter of the correct term next to each matching definition. The student receives 1 point for each correct response. Each probe lasts 5 minutes. 2-3 probes are given in a session. Source: Howell, K. W. (2008). Best practices in curriculum-based evaluation and advanced reading. In A. Thomas & J. Grimes (Eds.), Best practices in school psychology V (pp. 397-418).

Curriculum-Based Evaluation: Math Vocabulary Format Option 2 20 randomly arranged vocabulary definitions appear in the right column. Items are drawn randomly from a ‘vocabulary pool’ The student writes the name of the correct term next to each matching definition. The student is given 0.5 point for each correct term and another 0.5 point if the term is spelled correctly. Each probe lasts 5 minutes. 2-3 probes are given in a session. Source: Howell, K. W. (2008). Best practices in curriculum-based evaluation and advanced reading. In A. Thomas & J. Grimes (Eds.), Best practices in school psychology V (pp. 397-418).

Assessing Basic Academic Skills: Curriculum-Based Measurement Mathematics: Single-skill basic arithmetic combinations an ‘adequate measure of performance’ for low-achieving middle school students. Websites to create CBM math computation probes: www.interventioncentral.org www.superkids.com Source: Espin, C. A., & Tindal, G. (1998). Curriculum-based measurement for secondary students. In M. R. Shinn (Ed.) Advanced applications of curriculum-based measurement. New York: Guilford Press.

Assessing Basic Academic Skills: Curriculum-Based Measurement Writing: CBM/ Word Sequence is a ‘valid indicator of general writing proficiency’. It evaluates units of writing and their relation to one another. Successive pairs of ‘writing units’ make up each word sequence. The mechanics and conventions of each word sequence must be correct for the student to receive credit for that sequence. CBM/ Word Sequence is the most comprehensive CBM writing measure. Source: Espin, C. A., & Tindal, G. (1998). Curriculum-based measurement for secondary students. In M. R. Shinn (Ed.) Advanced applications of curriculum-based measurement. New York: Guilford Press.

Breaking Down Complex Academic Goals into Simpler Sub-Tasks: Discrete Categorization

Identifying and Measuring Complex Academic Problems at the Middle and High School Level Students at the secondary level can present with a range of concerns that interfere with academic success. One frequent challenge for these students is the need to reduce complex global academic goals into discrete sub-skills that can be individually measured and tracked over time.

Discrete Categorization: A Strategy for Assessing Complex, Multi-Step Student Academic Tasks Definition of Discrete Categorization: ‘Listing a number of behaviors and checking off whether they were performed.’ (Kazdin, 1989, p. 59). Approach allows educators to define a larger ‘behavioral’ goal for a student and to break that goal down into sub-tasks. (Each sub-task should be defined in such a way that it can be scored as ‘successfully accomplished’ or ‘not accomplished’.) The constituent behaviors that make up the larger behavioral goal need not be directly related to each other. For example, ‘completed homework’ may include as sub-tasks ‘wrote down homework assignment correctly’ and ‘created a work plan before starting homework’ Source: Kazdin, A. E. (1989). Behavior modification in applied settings (4th ed.). Pacific Gove, CA: Brooks/Cole..

Discrete Categorization Example: Math Study Skills General Academic Goal: Improve Tina’s Math Study Skills Tina was struggling in her mathematics course because of poor study skills. The RTI Team and math teacher analyzed Tina’s math study skills and decided that, to study effectively, she needed to: Check her math notes daily for completeness. Review her math notes daily. Start her math homework in a structured school setting. Use a highlighter and ‘margin notes’ to mark questions or areas of confusion in her notes or on the daily assignment. Spend sufficient ‘seat time’ at home each day completing homework. Regularly ask math questions of her teacher.

Discrete Categorization Example: Math Study Skills General Academic Goal: Improve Tina’s Math Study Skills The RTI Team—with teacher and student input—created the following intervention plan. The student Tina will: Approach the teacher at the end of class for a copy of class note. Check her daily math notes for completeness against a set of teacher notes in 5th period study hall. Review her math notes in 5th period study hall. Start her math homework in 5th period study hall. Use a highlighter and ‘margin notes’ to mark questions or areas of confusion in her notes or on the daily assignment. Enter into her ‘homework log’ the amount of time spent that evening doing homework and noted any questions or areas of confusion. Stop by the math teacher’s classroom during help periods (T & Th only) to ask highlighted questions (or to verify that Tina understood that week’s instructional content) and to review the homework log.

Discrete Categorization Example: Math Study Skills Academic Goal: Improve Tina’s Math Study Skills General measures of the success of this intervention include (1) rate of homework completion and (2) quiz & test grades. To measure treatment fidelity (Tina’s follow-through with sub-tasks of the checklist), the following strategies are used : Approached the teacher for copy of class notes. Teacher observation. Checked her daily math notes for completeness; reviewed math notes, started math homework in 5th period study hall. Student work products; random spot check by study hall supervisor. Used a highlighter and ‘margin notes’ to mark questions or areas of confusion in her notes or on the daily assignment. Review of notes by teacher during T/Th drop-in period. Entered into her ‘homework log’ the amount of time spent that evening doing homework and noted any questions or areas of confusion. Log reviewed by teacher during T/Th drop-in period. Stopped by the math teacher’s classroom during help periods (T & Th only) to ask highlighted questions (or to verify that Tina understood that week’s instructional content). Teacher observation; student sign-in.

RTI Teams: Recommendations for Data Collection

RTI Teams: Recommendations for Data Collection Collect a standard set of background information on each student referred to the RTI Team. RTI Teams should develop a standard package of background (archival) information to be collected prior to the initial problem-solving meeting. For each referred student, a Team might elect to gather attendance data, office disciplinary referrals for the current year, and the most recent state assessment results.

RTI Teams: Recommendations for Data Collection For each area of concern, select at least two progress-monitoring measures. RTI Teams can place greater confidence in their progress-monitoring data when they select at least two measures to track any area of student concern (Gresham, 1983)-ideally from at least two different sources (e.g., Campbell & Fiske, 1959). With a minimum of two methods in place to monitor a student concern, each measure serves as a check on the other. If the results are in agreement, the Team has greater assurance that it can trust the data. If the measures do not agree with one another, however, the Team can investigate further to determine the reason(s) for the apparent discrepancy.

RTI Teams: Recommendations for Data Collection Monitor student progress frequently. Progress-monitoring data should reveal in weeks--not months--whether an intervention is working because no teacher wants to waste time implementing an intervention that is not successful. When progress monitoring is done frequently (e.g., weekly), the data can be charted to reveal more quickly whether the student’s current intervention plan is effective. Curriculum-based measurement, Daily Behavior Report Cards, and classroom observations of student behavior are several assessment methods that can be carried out frequently.

Monitoring Student Academic Behaviors: Daily Behavior Report Cards

Daily Behavior Report Cards (DBRCs) Are… brief forms containing student behavior-rating items. The teacher typically rates the student daily (or even more frequently) on the DBRC. The results can be graphed to document student response to an intervention.

Daily Behavior Report Cards Can Monitor… Hyperactivity On-Task Behavior (Attention) Work Completion Organization Skills Compliance With Adult Requests Ability to Interact Appropriately With Peers

Daily Behavior Report Card: Jim Blalock May 5 Mrs. Williams Rm 108 Daily Behavior Report Card: Daily Version

Daily Behavior Report Card: Jim Blalock Mrs. Williams Rm 108 Daily Behavior Report Card: Weekly Version 05 05 07 05 06 07 05 07 07 05 08 07 05 09 07 40 60 60 50

Daily Behavior Report Card: Chart

Student Case Scenario: Jim Jim is a 10th-grade student who is failing his math course and in danger of failing English and science courses. Jim has been identified with ADHD. His instructional team meets with the RTI Team and list the following academic and behavioral concerns for Jim. Does not bring work materials to class Fails to write down homework assignments Sometimes does not turn in homework, even when completed Can be non-compliant with teacher requests at times.

www.interventioncentral.org

Closing Thoughts About Local Norms & RTI Assessment at the Secondary Level…

Local Norms: Set a Realistic Timeline for ‘Phase-In’ (Stewart & Silberglit, 2008) “If local norms are not already being collected, it may be helpful to develop a 3-5 year planned rollout of local norm data collection, reporting, and use in line with other professional development and assessment goals for the school. This phased-in process of developing local norms could start with certain grade levels and expand to others.” p. 229 Source: Stewart, L. H. & Silberglit, B. (2008). Best practices in developing academic local norms. In A. Thomas & J. Grimes (Eds.), Best practices in school psychology V (pp. 225-242). Bethesda, MD: National Association of School Psychologists.

Activity: How ‘RTI Ready’ is Your Middle or High School in Measuring Academic Skills? In your ‘elbow groups’: Discuss the range of assessments that your school can access to measure the current performance and progress of students in basic academic skill areas. How ‘RTI ready’ is your school in measuring these academic skills? Be prepared to share your discussion points with the larger group.

End

RTI Team Initial Meeting Form: Secondary Student Progress-Monitoring Page 15

RTI Team Initial Meeting Form: Secondary Student Progress-Monitoring Page 16

RTI Team Initial Meeting Form: Secondary Student Progress-Monitoring Page 17