1. Outcomes Assessment 101 Assessment Showcase 2009: Best Practices in the Assessment of Learning and Teaching February 11, 2009 Gwendolyn Johnson, Ph.D. Deborah Levine-Donnerstein, Ph.D. Assessment Coordinating Council

2. Agenda for Today Benefits of Assessment Procedure for Assessing Student Learning Learning Outcomes Statements Commonly Used Measures of Student Learning

3. “Assessment is a means for focusing our collective attention, examining our assumptions, and creating a shared academic culture dedicated to continuously improving the quality of higher learning. Assessment requires making expectations and standards for quality explicit and public; systematically gathering evidence on how well performance matches those expectations and standards; analyzing and interpreting the evidence; and using the resulting information to document, explain and improve performance.” Angelo, Thomas, “Reassessing Assessment,” AAHE Bulletin, Volume 47/Number 8, April, 1995. This definition of assessment was adapted by the University of Arizona. Definition of Assessment

4. Effective Assessment Adapted from work of T. A. Angelo – University of Miami – June 1998 1.Assesses what matters most – in what we actually teach and in what we expect our students to learn 2.Provides information for improving learning (and for improving teaching and assessment) 3.Focuses on process as well as on inputs and outcomes 4.Actively involves teachers and students (and appropriate others) 5.Uses multiple and varied measures 6.Is carried out at various key points 7.Provides feedback to those most affected

5. (B) Review statements from professional organization; peer institutions (D) Investigate existing measures not currently in use (G) Identify relevant externally defined criteria (e.g. what do peers do?) (1) Determine desired learning outcomes (B) Prioritize outcomes; decide on a schedule for assessing particular outcomes (2) Select multiple measures for each outcome being measured (3) Carry out assessment; tally and analyze results Set standards; Benchmarks (4b) Use results to fine-tune the assessment process (4a) Use results to improve program (C) Find out what is already being assessed; revise/adapt as needed. (A) Get input from stakeholders: faculty, students, alumni, employers, parents, etc. (E) Develop measures as needed (F) Determine sample size, schedule of administration, etc A Procedure for Assessing Learning Outcomes OIRPS Johnson 9/5/08 Adapted from work by E. Berman 1999

6. Learning Outcome Statements Describe what students are able to demonstrate in terms of knowledge, skills, and attitudes upon completion of the program Rely on concrete verbs such as “define,” “apply,” or “analyze” Map to the curriculum and educational practices that offer multiple and varied opportunities for students to learn Ideally, are collaboratively authored and collectively accepted Incorporate or adapt professional organizations’ outcomes statements if they exist Can be quantitatively and/or qualitatively assessed during students’ undergraduate or graduate studies Collectively created and developed across the program, learning outcome statements:

7. Learning Outcome Statements Students will be able to: Distinguish and articulate major cultural differences between the target culture and their own. Analyze how specific historical and cultural contexts shape particular perceptions, practices, and products of individuals Interpret mathematical models and represent information symbolically, visually, numerically, and verbally. Write persuasive documents that provide appropriate and effective evidence for various audiences, situations, and purposes. Demonstrate knowledge of the formal and informal structures and processes that make social systems, governments, and economies work. Design experiments, generate and analyze actual data, use abstract reasoning to interpret these, and formulate and test hypotheses with scientific rigor.

8. (B) Review statements from professional organization; peer institutions (D) Investigate existing measures not currently in use (H) Identify relevant externally defined criteria (e.g. what do peers do?) (1) Determine desired learning outcomes (B) Prioritize outcomes; decide on a schedule for assessing particular outcomes (2) Select multiple measures for each outcome being measured (3) Carry out assessment; tally and analyze results (G & H) Set standards; Benchmarks (4b) Use results to fine-tune the assessment process (4a) Use results to improve program (C) Find out what is already being assessed; revise/adapt as needed. (A) Get input from stakeholders: faculty, students, alumni, employers, parents, etc. (E) Develop measures as needed (F) Determine sample size, schedule of administration, etc (G) Consider level of incoming students with regard to benchmark A Procedure for Assessing Student Outcomes OIRPS Johnson 9/5/08 Adapted from work by E. Berman 1999

9. Commonly Used Measures of Student Learning Direct Indicators of Learning Pre/Post Tests Course -Embedded Assessment (papers, assignments, tests) Standardized Tests (national, accreditation) Portfolios Capstone Courses Thesis/Dissertation Evaluation Observation of Performance

10. Indirect Indicators of Learning Student surveys -questionnaires -interviews and focus groups External Reviewers Alumni surveys Employer Surveys Curriculum and Syllabus Analysis Commonly Used Measures of Student Learning

11. (B) Review statements from professional organization; peer institutions (D) Investigate existing measures not currently in use (H) Identify relevant externally defined criteria (e.g. what do peers do?) (1) Determine desired learning outcomes (B) Prioritize outcomes; decide on a schedule for assessing particular outcomes (2) Select multiple measures for each outcome being measured (3) Carry out assessment; tally and analyze results (G & H) Set standards; Benchmarks (4b) Use results to fine-tune the assessment process (4a) Use results to improve program (C) Find out what is already being assessed; revise/adapt as needed. (A) Get input from stakeholders: faculty, students, alumni, employers, parents, etc. (E) Develop measures as needed (F) Determine sample size, schedule of administration, etc (G) Consider level of incoming students with regard to benchmark A Procedure for Assessing Student Outcomes OIRPS Johnson 9/5/08 Adapted from work by E. Berman 1999

12. A Program Assessment Example UnitStudent papers/ projects Student portfolios Embedded assignments /testing Capstone Course evaluation Outcomes Externally evaluated HIS 101xxxx HIS 200xxxx HIS 210xx HIS 309xxxx HIS 322xxx HIS 400xxx HIS 410xxx HIS 450xxxx

13. The Important Questions to Consider:  Do the student learning outcomes match current-established goals?  Are the current outcomes what we intend to measure and assess?  Overall, are students acquiring our expected: skills, knowledge, attitudes and motivation to succeed in their academic experiences?  What changes are needed to address gaps in assessing student learning?  How can we proceed to identify shortfalls between curricula and learning outcome goals at each level?

14. (B) Review statements from professional organization; peer institutions (D) Investigate existing measures not currently in use (H) Identify relevant externally defined criteria (e.g. what do peers do?) (1) Determine desired learning outcomes (B) Prioritize outcomes; decide on a schedule for assessing particular outcomes (2) Select multiple measures for each outcome being measured (3) Carry out assessment; tally and analyze results (G & H) Set standards; Benchmarks (4b) Use results to fine-tune the assessment process (4a) Use results to improve processes (C) Find out what is already being assessed; revise/adapt as needed. (A) Get input from stakeholders: faculty, students, alumni, employers, parents, etc. (E) Develop measures as needed (F) Determine sample size, schedule of administration, etc (G) Consider level of incoming students corresponding to benchmarks A Procedure for Assessing Student Outcomes OIRPS Johnson 9/5/08 Adapted from work by E. Berman 1999

15. How can the assessment process be sustained? Can efficiency in collecting, reporting, and using results be increased in all relevant areas of campus? Should we change the standards, criteria, benchmarks, etc., on an ongoing basis to meet internal and external constraints facing universities? In what ways must campus policy support and reflect assessment goals for student learning? General Considerations

16. Sustainability: Assessing Learning Outcomes in a Changing Academic Environment Within the process of restructuring colleges, programs and departments, a campus retains leadership and gains new leaders to:   continue to enhance the goals of assessing student learning,   reshape and modify procedures for assessing student performance throughout this process,   reestablish linkages among new and existing units to better inform: what is important to assess,  how to address changes in programs and curricula,  develop effective measures to assess new goals and  determine how best to use these outcomes in the classroom, department and college for greater student retention and success

17. References Angelo, Thomas. “Reassessing Assessment,” AAHE Bulletin, Volume 47/Number 8, April, 1995. Maki, Peggy L. (2004). “Assessing for Learning: Building a Sustainable Commitment Across the Institution”. Sterling, VA: Stylus Publishing Walvoord, Barbara (2004). “Assessment Clear and Simple”. San Francisco, CA: Jossey-Bass http://www.provost.wisc.edu/assessment/manual/manual2.html

18. Gwendolyn Johnson, Ph.D. Director of Assessment, Office of Institutional Research and Evaluation Chair, Assessment Coordinating Council gwj@u.arizona.edu 520 621-7726 http://assessment.arizona.edu Deborah Levine-Donnerstein Department of Educational Psychology Contact Information