1. Rubric for Evaluation of Reading Language Arts Instructional Materials for Grades K-5

2. Acknowledgement and disclaimer
Information and materials for this presentation are supported by IES/NCEE’s Regional Educational Laboratory Southeast at Florida State University (Contract ED-IES-17-C-0011) as resources and examples for the viewer's convenience. Their inclusion is not intended as an endorsement by the Regional Educational Laboratory Southeast or its funding source, the Institute of Education Sciences.
In addition, the instructional practices and assessments discussed or shown in this presentation are not intended to mandate, direct, or control a State’s, local educational agency’s, or school’s specific instructional content, academic achievement system and assessments, curriculum, or program of instruction. State and local programs may use any instructional content, achievement system and assessments, curriculum, or program of instruction they wish.

3. REL-SE at Florida State University

4. Ask A REL Free reference desk service
Provides references and summaries of research tailored to your specific questions
For more information and to submit a question visit

5. https://ies.ed.gov/ncee/edlabs/projects/project.asp?projectID=4506
Rubric for Evaluation of Reading Language Arts Instructional Materials for Grades K-5

6. Summary
This rubric is a tool for evaluating reading/language arts instructional materials for grades K–5. Based on rigorous research, the rubric can be used by state-, district-, and school-level practitioners and by university faculty who review instructional
materials.

7. Rubric Overview
K-2 and 3-5 content areas (e.g., writing) include a list of criteria for evidence-based instructional practice.
Reviewers records findings based on extent to which the criteria were met using a 1-5 scale.
Space for comments beside each criterion.
Reviewers record overall ratings for K-2 & 3-5.
Reviewers meet to discuss overall ratings.
Facilitator shares/discusses results with state, district, or school leaders.

8. Research support for the development of the rubric
The rubric is based on what rigorous research indicates is the most effective way to teach reading/language arts. The What Works Clearinghouse identifies research studies that provide credible evidence of the effectiveness of a given practice, program, or policy (collectively referred to as “interventions”) and disseminates summary information and reports on its website ( The content of this rubric is based on six What Works Clearinghouse practice guides that pertain to content areas within reading/ language arts.

9. WWC Practice Guides Practice guides are:
Geared toward helping educators and administrators address challenges in classrooms and schools
Focused on a specific topic
Guided by an expert panel
Based on rigorous evidence
Comprised of evidence-based instructional recommendations
The WWC has produced 22 practice guides.

10. Practice Guides on Literacy Instruction
The rubric is based on the 6 released WWC practice guides on literacy
For younger learners
Reading comprehension (RC)
Foundational skills (FR)
Elementary writing skills (Writing)
For older learners
Adolescent literacy (Adol.)
For special populations
Response to intervention (RTI)
Instruction for English learners (EL)

11. Elements of design for reading/language arts instructional materials

12. Planning a review: Selecting a Facilitator
The instructional materials review process works best when the administrator in charge of the process selects a dedicated facilitator to lead the effort and review the rubric in detail before the review process begins.
The facilitator should understand:
Instructional design and content
Pedagogical research
Policies, procedures, implementation
How to listen carefully
How to lead structured decisionmaking discussions

13. Planning a review: Facilitator Duties
The facilitator has several responsibilities during the review process: • Recruiting and convening the review team (3–10 educators, depending on the quantity of materials to be reviewed). • Ensuring that reviewers have enough time for reviews. • Developing and implementing professional learning for the reviewers. • Determining initial inter-rater reliability. • Assigning instructional materials to reviewers. • Developing and providing checklists of requirements for publishers. • Considering pros and cons of reviewing hard copy versus digital instructional materials. • Facilitating consensus meetings.

14. Planning a review: Training Reviewers
The content of the professional learning developed by the facilitator includes:
An overview of the rubric.
How to conduct a review.
Whom to contact with questions.
Guidance on the number of hours it will take to review instructional materials and to develop a schedule of interim and final deadlines to complete the reviews (this will depend on the quantity of materials being reviewed).
A practice opportunity to review a sample set of instructional materials.

15. Planning a review: Developing a checklist
The facilitator may develop a checklist of required items that the publisher of the instructional materials under consideration needs to submit in order for a comprehensive review to occur. This checklist is provided to both the publisher and the review team. Items for the checklist may include: • An overview (for example, a presentation) of the program, sample materials, and video of demonstration lessons. • A detailed scope and sequence, including an alignment to adopted state standards and to research. • A mapping of the location of content in the materials to criteria on the rubric. • A list of the titles of books and documentation of their range and complexity.

16. Content areas: K-2 The content areas for grades K–2 are:
foundational reading skills (print concepts, phonological awareness, phonemic awareness, phonics, and fluency);
reading comprehension for literary and informational texts;
writing development and skills;
speaking and listening development and skills;
and language development and skills (academic language skills and vocabulary).

17. Content areas: 3-5 The content areas for grades 3–5 are:
foundational reading skills (advanced word analysis such as affixes, Greek and Latin roots, and syllable patterns);
reading comprehension for literary and informational texts and text complexity;
writing development and skills;
speaking and listening development and skills;
and language development and skills.

18. Criteria
Each content area includes a list of criteria for evidence-based instructional practice. Each reviewer records findings based on the extent to which the criteria were met using a 1–5 rating scale:
• 1 indicates the criterion was not met.
• 2 indicates the criterion was partially met.
• 3 indicates the criterion was adequately met.
• 4 indicates the criterion was substantially met.
• 5 indicates the criterion was completely met.
It can be helpful for a reviewer first to determine whether the criterion was met and then determine how well it was met (3—adequately met, 4—substantially met, or 5—completely met), or not met (1—not met or 2—partially met). After each criterion, there is space for reviewer comments (such as specific examples to support ratings, strengths, or concerns).

19. Overall rating
The overall rating subsection is where a reviewer records an overall holistic rating after considering the entire set of instructional materials. For example, if a 3 is recorded across all the items, a 3 would be expected in the overall rating section. If it is possible for the review team to meet and discuss the ratings, it would be most beneficial for the discussion to focus on the overall ratings, with the review team going back to the content sections to discuss support for ratings. Alternatively, patterns of strengths and weaknesses at the item level for component skills within content areas can be noted (for example, one set of instructional materials may be strong in foundational skills and weak in reading comprehension skills, whereas the opposite may be true in another set).

20. Examples of ratings and comments in the rubric (K-2)

21. Examples of ratings and comments in the rubric (3-5)

22. Using the rubric
Use the Phonics Instruction for Grades K-2 activities to complete the Phonics section of the K-2 rubric. Rate the activities on corresponding criteria and provide support for your ratings.

23. Blending by chunking with a pocket chart and letter tiles

24. Building words with Elkonin sound boxes

25. Word-analysis strategy

26. Sample word list and connected text for a lesson on oi
soil join oink
voice noise choice
coin foil avoid
Connected text passage
Sam went out to buy foil from the store. He lost his coins on the way. He looked for his coins, but he could not see them. 
Sam asked Luis to join him and help look for the coins. They could not find them. 
Then, Sam and Luis heard a voice. It was Mia. She found Sam’s coins! Sam, Luis, and Mia went to the store together to buy the foil.

27. The “Star Words” activity

28. Using the rubric
Use the Phonics Instruction for Grades K-2 activities to complete the Grades K-2 overall rating. Rate the activities on the foundational reading skills segment of the overall rating form and provide support for your rating.

29. Using the rubric Whole Group Discussion:
Focus on the overall rating and go back to the content section (Phonics) to support your rating.
Note any patterns of strengths or weaknesses at the item level for component skills.

30. Companion Tool to Accompany the Instructional Materials Rubric

31. K-2 Reviewer Spreadsheet

32. K-2 Facilitator Spreadsheet

33. K-2 Facilitator Spreadsheet

34. Summary Important Considerations: Facilitator selection
Reviewer training
Process development
Inter-rater reliability
Consensus meetings
Sharing results

35. https://ies.ed.gov/ncee/edlabs/regions/southeast/
Questions?
Laurie Lee
Kevin Smith
Phyllis Underwood