1. How to Use These Modules 1.Complete these modules with your grade level and/or content team. 2.Print the note taking sheets. 3.Read the notes as you view the PowerPoint in slideshow mode. 4.Try to keep in the frame of mind of how this information will inform the assessments we create/use.

2. Module 2: Elements of a Trustworthy Assessment Assessment Creation

3. OVERALL OBJECTIVE Collect information on student learning you can trust How to Create an Assessment so that you can…

4. OVERVIEW OF 3 MODULES Module 1: Assessment Planning Purpose and blueprint Module 2: The elements of a trustworthy assessment Improving trust in assessments Module 3: Assessment review and use Judge the value of your assessment results

5. INTRODUCTION AND OBJECTIVES Explain why VALIDITY AND RELIABILITY are important characteristics in an assessment. Define and use the FOUR ELEMENTS OF ASSESSMENT DESIGN to improve trustworthiness of an assessment and items.

6. INTRODUCTION AND OBJECTIVES Use the ASSESSMENT CHECKLIST to help determine whether an assessment meets the four elements of assessment design. Alignment Rigor Bias Precision Assessment Design Alignment Rigor Bias Precision

7. Reliability THE TWO BIG PROBLEMS OF ASSESSMENTS Your assessment measured the wrong thing. Make sure you measure what you think you are measuring! Your measure is inconsistent. (Students who have similar learning levels receive very different scores.) Make sure what you measure is trustworthy each time it is administered! Validity

8. KEY CONCEPTS Validity and Reliability

9. Four Elements of Assessment Design Assessment Checklist Validity and Reliability KEY CONCEPTS Sources: Kansas State Department of Education, Assessment Literacy Project; Ohio Department of Education, “Assessment Literacy: Identifying and Developing Valid and Reliable Assessments” (2013). validity the extent to which an assessment measures what you need it to measure reliability the extent to which a student’s score will be the same no matter when, where, or in what form the student takes the assessment or who scores it

10. Validity and Reliability Four Elements of Assessment Design Assessment Checklist Validity and Reliability IMPROVING VALIDITY AND RELIABILITY Reliability Measured the wrong thing = Measured inconsistently = Validity correct with checks for… correct with checks for … Assessment Design Rigor Bias Alignment Precision

11. Sources: Kansas State Department of Education, Assessment Literacy Project; Ohio Department of Education, “Assessment Literacy: Identifying and Developing Valid and Reliable Assessments” (2013); Relay Graduate School of Education, Designing and Evaluating Assessments (2014); and Rhode Island Department of Education, “Deepening Assessment Literacy.” Four Elements of Assessment Design

12. Validity and Reliability Four Elements of Assessment Design Assessment Checklist Four Elements of Assessment Design Assessment Design IMPROVING VALIDITY AND RELIABILITY Measure the right content Measure the right level of understanding (DOK/Bloom’s) Be clear, be correct, activate the right knowledge Avoid measuring unrelated knowledge and skills. Alignment Rigor Bias Precision

13. Assessment Checklist

14. Checking Your Assessment Goals of checklist 1.Measure what we need to measure (Validity) 2.Measure consistently (Reliability) Assessment Design Alignment Rigor BiasPrecision Alignment Rigor Bias Precision

15. CHECK FOR UNDERSTANDING How can you improve an assessment’s validity and reliability? To improve validity and reliability, use the assessment checklist to analyze individual items on assessments for alignment, rigor, bias, and precision. Check for Understanding Answer

16. KEY CONCEPTS Alignment

17. Rigor Precision Bias Alignment KEY CONCEPTS: Improving Validity An assessment that is ALIGNED WITH STANDARDS measures student performance against those standards Assessment Design Alignment Rigor Bias Precision

18. Alignment Rigor Precision Bias Alignment KEY CONCEPTS: Alignment Checking for Alignment 1.Start with the standards/learning objectives that students need to learn. 2.Determine what the student can do if they meet the standard/objective. The item/task needs to reflect this. 3.Instruction should enable students to meet the standard/objective. Standards Instruction Classroom Assessments

19. Alignment Rigor Precision Bias Alignment Alignment: Example Source: Common Core State Standards, www.corestandards.org. Interpret whole-number quotients of whole numbers, e.g., interpret 56 ÷ 8 as the number of objects in each share when 56 objects are partitioned equally into 8 shares, or as a number of shares when 56 objects are partitioned into equal shares of 8 objects each. For example, describe a context in which a number of shares or a number of groups can be expressed as 56 ÷ 8. What is 12 ÷ 5 = _____? CCSS.Math.Content.3.OA.A.2 Standard Assessment Item

20. Alignment Rigor Precision Bias Alignment KEY CONCEPTS What is 12 ÷ 5?2.4 or 2 with a remainder of 2 →Interpret whole-number quotients of whole numbers. Standards Classroom Assessments Skill(s) Assessment Item Answer

21. Alignment Rigor Precision Bias Alignment KEY CONCEPTS Is each assessment item aligned with the standard you intend to teach and measure? YesNo Refine

22. KEY CONCEPTS Rigor

23. Alignment Rigor Precision Bias Rigor KEY CONCEPTS: Improving Validity An assessment has an APPROPRIATE LEVEL OF RIGOR if the assessment includes items that match the level of rigor of the skill(s) you intend to measure and the assessment measures a range of student thinking and understanding so that it measures what all students know and can do. Assessment Design Bias Precision Alignment Rigor

24. Alignment Rigor Precision Bias Rigor KEY CONCEPTS Skill Assessment Item “Stretch” Items&Lower-level Items Higher-order Thinking Lower-order Thinking

25. Alignment Rigor Precision Bias Rigor KEY CONCEPTS Skill Higher-order Thinking Lower-order Thinking Assessment Item “Stretch” Items&Lower-level Items

26. Alignment Rigor Precision Bias Rigor KEY CONCEPTS CCSS.ELA-Literacy.RL.6.6 Explain how an author develops the point of view of the narrator or speaker in a text. Which point of view is used in the excerpt from The Road? a. first person b. second person c. third person limited d. third person omniscient Source: New York State Department of Education, “New York State P-12 Common Core Learning Standards for English Language Arts & Literacy” (2010). Standard Assessment Item

27. Alignment Rigor Precision Bias Rigor KEY CONCEPTS Does the assessment measure a range of student thinking and understanding so that it measures what all students know and can do? Does the level of rigor of each assessment item match the level of rigor of the skill you intend to measure? YesNo Refine

28. KEY CONCEPTS Bias

29. Alignment Rigor Precision Bias KEY CONCEPTS An UNBIASED assessment measures students’ knowledge and skills, not differences among groups of students because of their personal characteristics, such as race, gender, socioeconomic status, or religion Assessment Design Alignment Rigor Precision Bias

30. Alignment Rigor Precision Bias KEY CONCEPTS bias when an assessment provides a systematic advantage or disadvantage to groups of students because of their personal characteristics, such as race, gender, socioeconomic status or religion

31. Alignment Rigor Precision Bias KEY CONCEPTS We detect bias at the group level not the individual level.

32. Alignment Rigor Precision Bias KEY CONCEPTS Reading Comprehension Familiarity with Sailing Socioeconomic Status Source of Bias

33. Alignment Rigor Precision Bias KEY CONCEPTS Did you ensure that each assessment item does not provide an advantage or disadvantage to any group of students because of their personal characteristics, such as race, gender, socioeconomic status or religion? Yes No Refine

34. KEY CONCEPTS Precision

35. Alignment Rigor Precision Bias Precision Assessment Design KEY CONCEPTS A PRECISE assessment measures students’ knowledge and skills, not their misinterpretations or their background knowledge or lack thereof. Alignment Rigor Bias Precision

36. Alignment Rigor Precision Bias Precision KEY CONCEPTS Which is a type of mammal? a. carrot b. tree c. apple d. whale Selected Response Multiple Choice Assessment Item d. whale Answer

37. Alignment Rigor Precision Bias Precision KEY CONCEPTS Does the assessment and each assessment item… have all of the information that students will need to demonstrate their knowledge and skills? indicate how many points each assessment item is worth (if relevant) Indicate how much time students have to complete their work? Are all items well-formatted and free of typos and factual errors? Are the instructions for the assessment and each assessment item precise so that students understand how to respond to each item? Yes No Refine

38. CHECK FOR UNDERSTANDING What question can you ask to check for bias? If you like, use the assessment checklist to help you answer the question. Did I ensure that each assessment item does not provide an advantage or disadvantage to any group of students because of their personal characteristics, such as race, gender, socioeconomic status or religion? Check for Understanding Answer

39. CHECK FOR UNDERSTANDING Validity and Reliability Four Elements of Assessment Design Assessment Checklist

40. CHECK FOR UNDERSTANDING Use the ASSESSMENT CHECKLIST to help determine whether an assessment meets the four elements of assessment design. Explain why VALIDITY AND RELIABILITY are important characteristics in an assessment. Define and use the FOUR ELEMENTS OF ASSESSMENT DESIGN to improve trustworthiness of an assessment and items.

41. IMMEDIATE NEXT STEPS Review your school’s assessment strategy. Begin to collaboratively create assessment blueprints with your grade-level and/or content teams. Have your SAL fill in the Google Form after your school completes each module. (http://tinyurl.com/1617modulecompletion)fill in the Google Formhttp://tinyurl.com/1617modulecompletion

42. CONCLUSION