1. Evaluating Self-Created Lesson/Units and Open Educational Resource (OER) Objects An Introduction to the Achieve OER & Quality Review Rubrics CC BYCC BY Achieve 2013 & Christine Corbley, Ph.D 2013

2. 2 Open : Containing an open license - no restrictions on remixing or reusing [Having no enclosing or confining barrier; not restricted to a particular group or category of participants] Open Educational Resources: [With Webster’s Definitions] O E R Educational: Used for teaching and learning [Pertaining to the action or process of educating or being educated] Resource: Object used to support an effort or task [ A source of support or aid, especially one that can be readily drawn upon when needed]

3. 3 What are OER? OER are teaching, learning, and research resources that contain an open license. They provide extraordinary opportunities for educators to freely share knowledge and resources. They offer great potential for instructional innovation and networks for sharing best practices. There are millions of OER objects available online. Open Educational Resources (OER)

4. 4 OER objects may include (but are not limited to): Images Applets Games Worksheets Lesson plans Original-source texts Assessments Units Textbooks Teacher/learner support materials o NOTE: Any smaller component of a more complex object, that can exist as a stand-alone, also qualifies as an “object” (e.g. a unit in a textbook, a lesson in a unit, or an activity in a lesson). Open Educational Resources (OER)

5. Reviewing Material For Quality Equip Rubric Source: 5

6. Purpose of Equip Rubric Designed to evaluate Lessons that that include instructional activities and assessments aligned to the CCSS that may extend over a few class periods or days Units that include integrated and focused lessons aligned to the CCSS that extend over a period of time. Note: The rubric is not designed to evaluate a single task Source: 6

7. Using the Rubric http://schools.fwps.org/staff/?page_id=23946 Source: 7

8. 1. Record the grade and title of the lesson/unit a the top of the Rubric 2. Scan the lesson/unit materials to see what it contains and how it is organized. Skim key material, particularly those related to the dimensions 3. Identify the grade-level standards that the lesson/unit targets and compare its focus and expectations to those of the targeted CCSS 4. Study and measure the text(s) that serves as the centerpiece for the lesson/unit, analyze text complexity, quality, scope and relationship to instruction. 5. Analyze the lesson/unit for evidence of Dimension 1: alignment to the Rigor of the CCSS 6. Check the criteria in under Alignment to The Rigor of the CCSS Closely examine the evidence presented within a lesson/unit packet through the “lens” of each criterion. If clear and substantial evidence is found, check the criterion on the rubric; if not, use the comment column to provide input on specific improvements that can be made to meet the criterion. Source: 8

9. 7. Make observations and recommendations about this lesson/unit and record on the appropriate criteria sheet. Source: 9

10. Dimension Rating It is helpful to provide a criterion ‐ based rating, in addition to feedback: 1. Examine the pattern of checks in the Dimension’s Sections. Determine which of the descriptors on the 3 ‐ 2 ‐ 1 ‐ 0 rating scale (found below the rubric,) best characterizes the lesson/unit. 3: Meets most of criteria in the dimension 2: Meets many of the “criteria in dimensions 1: Meets some of the criteria in the dimension. 0: Does not meet the criteria in the dimension. Source: 10

11. Repeat for other Dimensions’ Key Shifts in the CCSS  Examine the lesson/unit through the “lens” of the criteria  Check the criteria that are met  Provide input on improvements needed to meet criteria not yet met  Use criteria to determine a 3-2-1-0 rating Instructional Support  Examine the lesson/unit through the “lens” of the criteria  Check the criteria that are met  Provide input on improvements needed to meet criteria not yet met  Use criteria to determine a 3-2-1-0 rating Assessment  Examine the lesson/unit through the “lens” of the criteria  Check the criteria that are met  Provide input on improvements needed to meet criteria not yet met  Use criteria to determine a 3-2-1-0 rating Source: 11

12. Use Dimensional Rating to Determine an Overall Rating 1. Review the patterns of the checked criteria and the ratings for each of the four dimensions. 2. Make a summary judgment about the overall quality of the lesson/unit, using the Overall Rating Scale: E: Exemplar- aligned and meets most to all of the criteria in dimension II, III, IV (total 11-12) E/I:Exemplar if Improved - Aligned and needs some improvement in one or more dimensions (total 8-10) R: Revision needed- Aligned partially and needs significant revision in one or more dimension (total 3-7) N: Not Ready to Review – Not aligned and does not meet criteria (total 0-2) Source: 12

13. 13 The Achieve OER Evaluation Rubrics How and why were the Achieve OER Rubrics created? Educators using the vast system of sharing need a method for filtering OER to meet the needs of their students. The purpose of the rubrics is to provide a structure for evaluating an online resource in a systematic, purposeful and comprehensive way. There are two ways of approaching the evaluation of a resource: o Holistically evaluating the object with a single rating, (similar to the star-rating systems used on Netflix, Yelp and Amazon) o Separately evaluating each of the components of quality, as with the Achieve OER Rubrics. The rubric criteria are based on Achieve protocols used to assist states in alignment and quality review studies. The Achieve OER Evaluation Tool, hosting the rubrics on OERCommons.org, allows a user to identify, evaluate and sort objects based on the specified essential elements of quality.

14. 14 How do the Achieve OER Rubrics work? The rubrics represent an evaluation system for objects found within Open Educational Resources. They are hosted as an online evaluation tool on the repository, OERCommons.org, but can also be used independently. OERCommons.org They are applied to any content area. o NOTE: At this stage only Common Core Standards for English Language Arts/Literacy and Mathematics are available in the Achieve OER Evaluation Tool. Content standards for History, Social Studies, Sciences, and technical subjects are not yet available. Each rubric is applied independently to an object. They rate the potential, not the actual, effectiveness of an object. They use a five-point scoring system to describe levels of quality. Applying the Achieve OER Rubrics

15. 15 There are Seven Rubrics on the Achieve OER Evaluation Tool Rubric I. Degree of Alignment to Standards Rubric II. Quality of Explanation of the Subject Matter Rubric III. Utility of Materials Designed to Support Teaching Rubric IV. Quality of Assessment Rubric V. Quality of Technological Interactivity Rubric VI. Quality of Instructional Tasks and Practice Exercises Rubric VII. Opportunities for Deeper Learning o NOTE: Sometimes one or more of the rubrics may not be relevant to a particular object. In those cases, a rating of N/A is appropriate The Rubrics

16. 16 The Detailed Scoring Descriptors are Specific to Each Rubric 3: Superior 2: Strong 1: Limited 0: Very weak / None N/A: Rubric not applicable to the object o NOTE: N/A should be used when a particular rubric does not apply to the object being rated. This is not a pejorative score; it simply means it would be inappropriate to apply this rubric to the particular object. The Rubrics’ Common Scoring Scale

17. 17 Rubric II: Quality of Explanation of Subject Matter Rubric II is used to rate how well the subject matter is explained or otherwise revealed in the resource. Subject explanation could be aimed at either the teacher or a student. This rubric could apply to objects used for a whole class, small groups, or an individual student (e.g. self-tutoring object). Sometimes the subject matter explanation is included in teacher support material and is intended to help a teacher learn or be refreshed on a specific topic (the teacher becomes the learner). Do not use Rubric II to rate the quality of the explanation of how to present the lesson (planning instructions) for an object. Use an N/A-rating only if there is no explanation of subject matter included in the object or for objects not designed to explain subject matter (e.g. a formula sheet or a map), even if it might be possible to use the object to explain a subject.

18. 18 The rubric does not apply to students interact with each other, but rather how the technology responds to an individual student user. For interactive objects designed for student use, the rubric is applied to the entire object. This is not a rating for technology in general, but only for technological interactivity between the user and the object. For objects designed for teacher use, for example lesson plans, which may include interactive components for use by students, the rubric is applied only to those interactive parts. Does not apply to objects where interaction with the object is limited to, for example, opening a PDF attachment. Rubric V: Quality of Technological Interactivity

19. 19 Applied to objects that contain instructional tasks and/or sets of practice exercises designed to strengthen skills and knowledge. The purpose of these exercises is to deepen understanding and to routinize foundational skills and procedures. (When concepts/skills are introduced providing sufficient practice is critical. When integrating skills into complex tasks, the number of exercises is less important than their richness.) Applied to a group of exercises as a single object. Not applicable if the object does not include practice exercises. Even if one can imagine ways to use the object as practice, if that is not the intention of the object, N/A is the correct rating. Rubric VI: Quality of Instructional Tasks and Practice Exercises

20. 20 Rubric VII is applied to objects designed to engage learners in at least one of the following deeper learning skills: Think critically and solve complex problems Work collaboratively Communicate effectively Learn how to learn Reason abstractly Construct viable, logical arguments and critique the reasoning of others Apply discrete knowledge and skills to real-world situations Construct, use, or analyze models o Not applicable when an object does not appear to be designed to provide the opportunity for deeper learning, even though one might imagine how it could be used to do this. Rubric VII: Opportunities for Deeper Learning