1. Writing High Quality Assessment Items Using a Variety of Formats Scott Strother & Duane Benson 11/14/14

2. Carnegie Community College Pathways

4. Both Pathways’ instructional systems include: Ambitious learning goals Lessons and out-of-class materials Formative and summative assessments Productive persistence Language and literacy Dynamic online environment Advancing quality teaching Rapid analytics to support continuous improvement

5. Carnegie Community College Pathways Both Pathways’ instructional systems include: Ambitious learning goals Lessons and out-of-class materials Formative and Summative Assessments Productive persistence Language and literacy Dynamic online environment Advancing quality teaching Rapid analytics to support continuous improvement

6. Summative Assessments: Purpose Summative Assessments 1)Understand Levels of Student Knowledge 2) Understand Variation in Student Knowledge 3) Program Improvement

7. Summative Assessments: Assessment Building Process 7 Item Writing Language & Literacy Field Testing / Comparison Dissemination and Administration Psychometric Analysis Assessment Assembly Measurement Review and Content Check: Faculty; Carnegie Performance Analysis Measurement Review and Content Check: Faculty; Carnegie 123 456 7 89

8. Summative Assessments: Assessment Report 8 Item Writing Language & Literacy Field Testing / Comparison Dissemination and Administration Psychometric Analysis Assessment Assembly Measurement Review and Content Check: Faculty; Carnegie Performance Analysis Measurement Review and Content Check: Faculty; Carnegie 123 456 7 89

9. Summative Assessments: Purpose Statway Mid-Pathway AssessmentsPathway Difference Fall 2011 +4.9 Spring 2012 +7.2 Fall 2012 +3.5 Spring 2013 +4.1 Statway End-of- Pathway Assessments Spring 2012 +20.8 Spring 2013 +17.4 Quantway 1 Assessments Spring 2012 +15.8 Fall 2012 +13.2 Spring 2013 +11.7

10. Summative Assessments: Purpose Statway Mid-Pathway AssessmentsCorrelation to Grade Fall 2011.61** Spring 2012.44 * Fall 2012.57** Spring 2013.64** Statway End-of- Pathway Assessments Spring 2012.60** Spring 2013.49** Quantway 1 Assessments Spring 2012.60** Fall 2012.66** Spring 2013.66**

11. Summative Assessments: Purpose Summative Assessments 1)Understand Levels of Student Knowledge 2) Understand Variation in Student Knowledge 3)Program Improvement What is Next?

12. Summative Assessments: Improvement Item Diversity 12 Going Beyond Multiple Choice

13. Diversity of Item Types Why not multiple choice? MC items tend to be reliable, seem to be easily written and understood… Some would argue MC items: – Do not test critical thinking, creativity – Inauthentic – Too easily answered by guessing – Encourage teaching to test – Do not engage students

14. Summative Assessments: Improvement Capacity for Assessment Writing 14 Assessment Training

15. Summative Assessments: Improvement Implementation 15 Online Assessments

16. Summative Assessments: Improvement Item Diversity Capacity for Assessment Writing Implementation 16 Going Beyond Multiple Choice Online Assessments Assessment Training

17. Assessment Expert Convening Assessment Team / Training 1. Diversity of Item Types 2. High Quality Items 3. High Quality Assessments

18. Assessment Expert Convening Assessment Team / Training 1. Diversity of Item Types 2. High Quality Items 3. High Quality Assessments

19. Assessment Expert Convening Assessment Team / Training 1. Diversity of Item Types 2. High Quality Items 3. High Quality Assessments

20. Diversity of Item Types Item Formats Brainstorm

21. Diversity of Item Types True False Ordering / Ranking Multiple choice Multiple select Fill in blank / completion Matching Drag n’ drop Hotspot Knowledge matrix Cascaded items Calculations Short answer Visual representation (graphing, plotting, sketch diagram, drawing,, create graphics) Labeling Constructed response / free response Project based construction, (performance task, investigation project, portfolio piece, multistep project, performance assessment Apprentice task Essay Computer Simulations

22. Example of New Item Formats Drag N Drop HotSpot

23. Computer simulations Can help address problems with authentic assessments

24. Trade-Offs OptionExampleBenefitsDrawbacks 1. Established measure Multiple-choice, true-false Cheap, easy to obtain and score Less authentic, complex, engaging 2. Novel but somewhat established measure Open-response, portfolio More complex, less rote Challenges in making raters consistent, time 3. Cutting-edge technology-based measure Performance, computer simulation Much more authentic, engaging Resources (including computers), remaining technical challenges

25. Diversity of Item Types Big Considerations Purpose of Assessment Depth of Knowledge Cognitive Load / Level of Knowledge Recognize and recall Procedural knowledge (skills and concepts) Explain and conclude (strategic thinking) Justification(extended thinking) Design Specifications Grading (time, reliability at scale, etc.) Goal of Item (e.g. productive struggle) Context Item Program

26. Diversity of Item Types Purpose of Assessment Depth of Knowledge Cognitive Load Context Design Specifications Grading Goal of Item True False Ordering / Ranking Labeling Multiple Choice Multiple select Fill in blank / completion Matching Drag n’ drop Hotspot Knowledge Matrix Cascaded items Calculations Short answer Visual representations Constructed response / free response Project based construction, Apprentice task Essay Computer Simulation

27. Diversity of Item Types Purpose of Assessment Depth of Knowledge Cognitive Load Context Design Specifications Grading Goal of Item True False Ordering / Ranking Labeling Multiple Choice Multiple select Fill in blank / completion Matching Drag n’ drop Hotspot Knowledge Matrix Cascaded items Calculations Short answer Visual representations Constructed response / free response Project based construction, Apprentice task Essay

28. High Quality Items Handout Revisit Your Item

29. High Quality Assessments Aligning Items with Learning Outcomes or Standards – Categorical Concurrence – Range-of-Knowledge Correspondence – Balance of Representation – Depth-of-Knowledge Consistency

30. High Quality Assessments Aligning Items with Learning Outcomes or Standards Reliability Validity

31. Validity is an argument Validity: evidence supports specific interpretations of scores for specific uses No such thing as a valid test – there are only valid uses and inferences A variety of evidence can be used to support these arguments Test content Response processes Relationships with other measures Generalizability across administrations, forms, etc.

32. High Quality Assessments Aligning Items with Learning Outcomes or Standards Reliability Validity Bias Depth of Knowledge (targeting various levels and skills*)* Difficulty (ensuring the appropriate spread) Discrimination (of various ability levels) Variety of Item Formats Context / Relevance Purpose of Assessments (e.g. formative, summative)

33. Purpose Examples Sum up what a student has learned? Compare across a wide range of students? Chart progress towards mastery? Understand student’s learning process? Drive academic improvement? Technical/measurement quality Reliability Validity Instructional priorities Authenticity Task complexity Engagement

34. High Quality Assessments Critical things often overlooked: -How will you use the feedback? -useful for teaching and learning? -Program goals (e.g. productive struggle) -Administration platform -BALANCE

35. Process of Item Writing Item writing is hard – Become very familiar with content and learning outcomes – Choose specific learning outcomes, difficulty level, discrimination, item format, etc. first – Consider design specifications of each item format – Consider resources: time, administration platform – Balance – Get help! Collective Understanding Group effort Iterative Feedback Examples are great! Good and bad! – PRACTICE Writing Reviewing!

36. 36 Questions

37. Contact Information Scott Strother strother@carnegiefoundation.org 650-566-5157 37