1. Refining Your Assessment Plan: Session 4 Use of the Results Ryan Smith and Derek Herrmann University Assessment Services

2. INTRODUCTIONS Refining Your Assessment Plan: Session 4

3. Introductions Ryan Smith, Director, UAS Derek Herrmann, Coordinator, UAS Assessment Advisory Council (AAC) Assessment Academy Team

4. Introductions Who you are In which Department/School you work Why you are here

5. OVERVIEW OF PRAAP Refining Your Assessment Plan: Session 4

6. Overview of PRAAP P rocess for the R eview of A cademic A ssessment P lans Two years before Program Review and one year before Self-Study AAC members and UAS staff Use an established rubric

7. Overview of PRAAP ElementsUndevelopedDevelopingEstablishedExemplary Goals and Learning Outcomes Direct Evidence of Student Learning Indirect Evidence of Student Learning Use of the Results

8. USE OF THE RESULTS Refining Your Assessment Plan: Session 4

9. Use of the Results Who is involved? – Chairperson/Director (or Assistant/Associate) – Program Coordinator – Faculty Individuals Committees

10. Use of the Results Where is it done? – Department/School meetings – Retreats (general or assessment) – Informal conversations

11. Use of the Results Examining the data – Rubrics A scoring guide; a list or chart that describes the criteria used to evaluate (or grade) completed student assignments Using a rubric to evaluate (or grade) assignments… – Makes your life easier – Improves student learning

12. Use of the Results Holistic rubric: short narrative descriptions of the characteristics of different levels of work Level 4: Description of the performance at this level Level 3: Description of the performance at this level Level 2: Description of the performance at this level Level 1: Description of the performance at this level

13. Use of the Results Checklist rubric: simple list indicating the presence of the things you’re looking for in an assignment CriteriaPresent Criterion 1 Criterion 2 Criterion 3 Criterion 4 Criterion 5

14. Use of the Results Rating scale rubric: checklist with a rating scale added to show the degree to which the things you’re looking for are present in an assignment CriteriaNot PresentDevelopingEstablishedAdvanced Criterion 1 Criterion 2 Criterion 3 Criterion 4 Criterion 5

15. Use of the Results Descriptive rubric: replaces the checkboxes of rating scale rubrics with brief descriptions of the performances that merit each possible rating CriteriaNot PresentDevelopingEstablishedAdvanced Criterion 1 Description of the performance that merits this rating Criterion 2 Description of the performance that merits this rating Criterion 3 Description of the performance that merits this rating Criterion 4 Description of the performance that merits this rating Criterion 5 Description of the performance that merits this rating

16. Use of the Results Examining the data – Content analysis Involves making sense of the material being reviewed Summarize common themes and the extent of consensus concerning those themes Should be approached with an open mind and a willingness to “hear” what respondents are saying

17. Use of the Results Examining the data – Content analysis Should begin with a clear understanding of the goals associated with the review Sometimes coding categories are predetermined; sometimes, they emerge as materials are reviewed

18. Use of the Results Inter-rater reliability – Indicates the extent of agreement among different reviewers – Generally want to verify that scores based on subjective judgments are reliable (consistent)

19. Use of the Results Inter-rater reliability – Two approaches Have at least two people independently review data, compare notes, and come to agreement Have reviewers work together on all decisions, collaborating as they go to resolve discrepancies

20. Use of the Results Establishing standards – Generally set own standards – Should take the campus and program mission into account when establishing standards

21. Use of the Results Establishing standards – Sometimes faculty want to know how their students compare to students on other campuses – Can set expectations based on the relative performance of their students

22. Use of the Results Summarizing the results – Frequency table: a visual depiction of data that shows how often each value occurred

23. Use of the Results ItemResponse optionsFrequencyPercent What is your present attitude toward your degree program? Strongly negative00.0 Negative15.0 Somewhat negative15.0 Somewhat positive315.0 Positive735.0 Strongly positive840.0 TOTAL20100.0

24. Use of the Results Summarizing the results – Frequency table: a visual depiction of data that shows how often each value occurred – Measures of central tendency Mean: arithmetic average Median: 50 th percentile Mode: most common score

25. Use of the Results ItemResponse optionsFrequencyPercent What is your present attitude toward your degree program? Strongly negative (1)00.0 Negative (2)15.0 Somewhat negative (3)15.0 Somewhat positive (4)315.0 Positive (5)735.0 Strongly positive (6)840.0 TOTAL20100.0 Mean = 5.0Median = 5.0Mode = 6.0

26. Use of the Results Summarizing the results – Qualitative summary: “The majority of respondents indicated that they hold positive attitudes toward their degree program.”

27. Use of the Results Sharing the results – Be open, honest, balanced, and fair – Understand your audiences and their needs – Help your audiences see the big picture

28. Use of the Results Sharing the results – Celebrate good results – Address areas needing improvement Goals and/or outcomes? Direct and/or indirect evidence? Curriculum and/or instruction? – Incorporate results into planning and decision-making processes

29. FINAL THOUGHTS Refining Your Assessment Plan: Session 4

30. Final Thoughts Goals and learning outcomes define how students demonstrate their mastery of what faculty want them to learn A cohesive curriculum is aligned with goals and learning outcomes

31. Final Thoughts Assessment involves collecting direct and indirect evidence concerning student development Embedding assessment in the curriculum has many advantages Assessment should not focus on individual students or faculty

32. Final Thoughts A recurring theme in assessment is collaboration Three major criteria apply to assessment; it should be meaningful, manageable, and sustainable

33. QUESTIONS, COMMENTS, AND DISCUSSION Refining Your Assessment Plan: Session 4