Rubric for Evaluation of Reading Language Arts Instructional Materials for Grades K-5

Acknowledgement and disclaimer Information and materials for this presentation are supported by IES/NCEE’s Regional Educational Laboratory Southeast at Florida State University (Contract ED-IES-17-C-0011) as resources and examples for the viewer's convenience. Their inclusion is not intended as an endorsement by the Regional Educational Laboratory Southeast or its funding source, the Institute of Education Sciences. In addition, the instructional practices and assessments discussed or shown in this presentation are not intended to mandate, direct, or control a State’s, local educational agency’s, or school’s specific instructional content, academic achievement system and assessments, curriculum, or program of instruction. State and local programs may use any instructional content, achievement system and assessments, curriculum, or program of instruction they wish.

REL-SE at Florida State University

Ask A REL Free reference desk service Provides references and summaries of research tailored to your specific questions For more information and to submit a question visit http://ies.ed.gov/ncee/edlabs/askarel/index.asp?REL=southeast

https://ies.ed.gov/ncee/edlabs/projects/project.asp?projectID=4506 Rubric for Evaluation of Reading Language Arts Instructional Materials for Grades K-5 https://ies.ed.gov/ncee/edlabs/projects/project.asp?projectID=4506

Summary This rubric is a tool for evaluating reading/language arts instructional materials for grades K–5. Based on rigorous research, the rubric can be used by state-, district-, and school-level practitioners and by university faculty who review instructional materials.

Rubric Overview K-2 and 3-5 content areas (e.g., writing) include a list of criteria for evidence-based instructional practice. Reviewers records findings based on extent to which the criteria were met using a 1-5 scale. Space for comments beside each criterion. Reviewers record overall ratings for K-2 & 3-5. Reviewers meet to discuss overall ratings. Facilitator shares/discusses results with state, district, or school leaders.

Research support for the development of the rubric The rubric is based on what rigorous research indicates is the most effective way to teach reading/language arts. The What Works Clearinghouse identifies research studies that provide credible evidence of the effectiveness of a given practice, program, or policy (collectively referred to as “interventions”) and disseminates summary information and reports on its website (http://ies.ed.gov/ncee/wwc). The content of this rubric is based on six What Works Clearinghouse practice guides that pertain to content areas within reading/ language arts.

WWC Practice Guides Practice guides are: Geared toward helping educators and administrators address challenges in classrooms and schools Focused on a specific topic Guided by an expert panel Based on rigorous evidence Comprised of evidence-based instructional recommendations The WWC has produced 22 practice guides.

Practice Guides on Literacy Instruction The rubric is based on the 6 released WWC practice guides on literacy For younger learners Reading comprehension (RC) Foundational skills (FR) Elementary writing skills (Writing) For older learners Adolescent literacy (Adol.) For special populations Response to intervention (RTI) Instruction for English learners (EL)

Elements of design for reading/language arts instructional materials

Planning a review: Selecting a Facilitator The instructional materials review process works best when the administrator in charge of the process selects a dedicated facilitator to lead the effort and review the rubric in detail before the review process begins. The facilitator should understand: Instructional design and content Pedagogical research Policies, procedures, implementation How to listen carefully How to lead structured decisionmaking discussions

Planning a review: Facilitator Duties The facilitator has several responsibilities during the review process: • Recruiting and convening the review team (3–10 educators, depending on the quantity of materials to be reviewed). • Ensuring that reviewers have enough time for reviews. • Developing and implementing professional learning for the reviewers. • Determining initial inter-rater reliability. • Assigning instructional materials to reviewers. • Developing and providing checklists of requirements for publishers. • Considering pros and cons of reviewing hard copy versus digital instructional materials. • Facilitating consensus meetings.

Planning a review: Training Reviewers The content of the professional learning developed by the facilitator includes: An overview of the rubric. How to conduct a review. Whom to contact with questions. Guidance on the number of hours it will take to review instructional materials and to develop a schedule of interim and final deadlines to complete the reviews (this will depend on the quantity of materials being reviewed). A practice opportunity to review a sample set of instructional materials.

Planning a review: Developing a checklist The facilitator may develop a checklist of required items that the publisher of the instructional materials under consideration needs to submit in order for a comprehensive review to occur. This checklist is provided to both the publisher and the review team. Items for the checklist may include: • An overview (for example, a presentation) of the program, sample materials, and video of demonstration lessons. • A detailed scope and sequence, including an alignment to adopted state standards and to research. • A mapping of the location of content in the materials to criteria on the rubric. • A list of the titles of books and documentation of their range and complexity.

Content areas: K-2 The content areas for grades K–2 are: foundational reading skills (print concepts, phonological awareness, phonemic awareness, phonics, and fluency); reading comprehension for literary and informational texts; writing development and skills; speaking and listening development and skills; and language development and skills (academic language skills and vocabulary).

Content areas: 3-5 The content areas for grades 3–5 are: foundational reading skills (advanced word analysis such as affixes, Greek and Latin roots, and syllable patterns); reading comprehension for literary and informational texts and text complexity; writing development and skills; speaking and listening development and skills; and language development and skills.

Criteria Each content area includes a list of criteria for evidence-based instructional practice. Each reviewer records findings based on the extent to which the criteria were met using a 1–5 rating scale: • 1 indicates the criterion was not met. • 2 indicates the criterion was partially met. • 3 indicates the criterion was adequately met. • 4 indicates the criterion was substantially met. • 5 indicates the criterion was completely met. It can be helpful for a reviewer first to determine whether the criterion was met and then determine how well it was met (3—adequately met, 4—substantially met, or 5—completely met), or not met (1—not met or 2—partially met). After each criterion, there is space for reviewer comments (such as specific examples to support ratings, strengths, or concerns).

Overall rating The overall rating subsection is where a reviewer records an overall holistic rating after considering the entire set of instructional materials. For example, if a 3 is recorded across all the items, a 3 would be expected in the overall rating section. If it is possible for the review team to meet and discuss the ratings, it would be most beneficial for the discussion to focus on the overall ratings, with the review team going back to the content sections to discuss support for ratings. Alternatively, patterns of strengths and weaknesses at the item level for component skills within content areas can be noted (for example, one set of instructional materials may be strong in foundational skills and weak in reading comprehension skills, whereas the opposite may be true in another set).

Examples of ratings and comments in the rubric (K-2)

Examples of ratings and comments in the rubric (3-5)

Using the rubric Use the Phonics Instruction for Grades K-2 activities to complete the Phonics section of the K-2 rubric. Rate the activities on corresponding criteria and provide support for your ratings.

Blending by chunking with a pocket chart and letter tiles

Building words with Elkonin sound boxes

Word-analysis strategy

Sample word list and connected text for a lesson on oi soil join oink voice noise choice coin foil avoid   Connected text passage Sam went out to buy foil from the store. He lost his coins on the way. He looked for his coins, but he could not see them.  Sam asked Luis to join him and help look for the coins. They could not find them.  Then, Sam and Luis heard a voice. It was Mia. She found Sam’s coins! Sam, Luis, and Mia went to the store together to buy the foil.

The “Star Words” activity

Using the rubric Use the Phonics Instruction for Grades K-2 activities to complete the Grades K-2 overall rating. Rate the activities on the foundational reading skills segment of the overall rating form and provide support for your rating.

Using the rubric Whole Group Discussion: Focus on the overall rating and go back to the content section (Phonics) to support your rating. Note any patterns of strengths or weaknesses at the item level for component skills.

Companion Tool to Accompany the Instructional Materials Rubric

K-2 Reviewer Spreadsheet

K-2 Facilitator Spreadsheet

K-2 Facilitator Spreadsheet

Summary Important Considerations: Facilitator selection Reviewer training Process development Inter-rater reliability Consensus meetings Sharing results

https://ies.ed.gov/ncee/edlabs/regions/southeast/ Questions? Laurie Lee llee@fcrr.org Kevin Smith ksmith@fcrr.org Phyllis Underwood punderwood@fcrr.org https://ies.ed.gov/ncee/edlabs/regions/southeast/