S TUDENT A FFAIRS R ESEARCH AND A SSESSMENT THE ROLE OF ASSESSMENT IN STUDENT AFFAIRS PRACTICE CSA 502: Organization and Administration in Student Affairs February 10, 2011 Building a culture of evidence:

S TUDENT A FFAIRS R ESEARCH AND A SSESSMENT Presentation outline What is assessment & why do we do it? The role of SARA and assessment offices in general Assessment planning & the assessment cycle It’s your turn - practice makes perfect!

S TUDENT A FFAIRS R ESEARCH AND A SSESSMENT Learning objectives: At the end of this session, you should be able to: Identify why assessment is important in all aspects of student affairs. Describe the general role of an assessment office. Describe the assessment cycle. Perform the basic steps to create an assessment plan.

S TUDENT A FFAIRS R ESEARCH AND A SSESSMENT What is assessment? It is “any effort to gather, analyze, and interpret evidence which describes…effectiveness (Upcraft & Schuh, 996) Upcraft & Schuh (2002) argue that research informs theory while assessment informs practice. But what does that really mean in practical terms??

S TUDENT A FFAIRS R ESEARCH AND A SSESSMENT Why is assessment important? Helps determine if you are meeting your educational objectives Helps ensure that you have the resources you need Helps prioritize efforts Can contribute to our understanding of student learning and development

S TUDENT A FFAIRS R ESEARCH AND A SSESSMENT Student Affairs Research & Assessment (SARA) Mission Student Affairs Research and Assessment provides leadership and service to the Division of Student Affairs in the areas of assessment, learning outcomes, educational programming, and strategic planning. SARA collects and disseminates data about Penn State students, their experiences and learning, and their perceptions of the campus environment. SARA enhances students’ educational experiences through the design, implementation, and evaluation of programs that support the University’s Cocurricular Learning Outcomes.

S TUDENT A FFAIRS R ESEARCH AND A SSESSMENT What does an assessment office do? Conducts assessments, duh! Provides training and professional development to others Environmental scanning Consulting Brings together evidence Contributes to strategic planning and accreditation activities

S TUDENT A FFAIRS R ESEARCH AND A SSESSMENT Assessment planning There is a great deal of pressure to DO assessment, but where do you start?

S TUDENT A FFAIRS R ESEARCH AND A SSESSMENT First, review your goals Clues can be found in your: Mission, vision, and objectives Strategic plan Stated learning outcomes Where do you want to be?

S TUDENT A FFAIRS R ESEARCH AND A SSESSMENT It is really helpful to get out of your silo You can only learn so much from documents. Institutional knowledge: ◦ Who? ◦ What? ◦ How?

S TUDENT A FFAIRS R ESEARCH AND A SSESSMENT Next, develop a plan.

S TUDENT A FFAIRS R ESEARCH AND A SSESSMENT Start with a model 1. Janusian Duality For internal improvement For External accountability II. Five Questions 1.Are you meeting your goals? 2.Are you improving? 3.Do you meet prof. standards? 4.How do you compare? 5.Are you efforts cost-effective? III. Select Methods & Measures For Each Level: -Institution -Unit -Individual students For Academic Effectiveness: Student learning, Research, and Scholarship For Administrative Effectiveness: Strategic Planning & Resource Mgmt. IV. Collect & Analyze Evaluation Evidence V. Communicate Results Take Academic & Administrative Action Improve & Strengthen Programs The Volkwein Model (2009)

S TUDENT A FFAIRS R ESEARCH AND A SSESSMENT The assessment cycle Identify outcomes Gather evidence Interpret evidence Refine efforts

S TUDENT A FFAIRS R ESEARCH AND A SSESSMENT Key steps - PDCA Identify outcomes Identify appropriate measures Choose appropriate assessment methods Choose an appropriate research design Collect the data Analyze & interpret the data Disseminate the findings Take action Wash, Rinse, Repeat!

S TUDENT A FFAIRS R ESEARCH AND A SSESSMENT Think long-term Chances are you cannot assess all of your outcomes every semester or even every year. Be realistic about your resources and develop a comprehensive, multi-year plan. Don’t forget to incorporate all steps of the cycle into your plan.

S TUDENT A FFAIRS R ESEARCH AND A SSESSMENT A good plan: Collects multiple types of evidence (triangulation), Takes into consideration your population(s) of interest, Takes into consideration your audience, Has a realistic timeline, Identifies its products, Incorporates dissemination & action, and Is always subject to revision!

S TUDENT A FFAIRS R ESEARCH AND A SSESSMENT A matrix can be a helpful to organize your strategy and look for holes in your plan OutcomeSurvey Reflective writing Expert evaluation Knowledge /skills test Critical thinking FA2011 Leadership SP2012FA2013 Communication FA2013SP2014 Civic engagement SP2014 Healthy lifestyle FA2011SP2012 Social justice perspective SP2015FA2014

S TUDENT A FFAIRS R ESEARCH AND A SSESSMENT Once you have your multi-year plan, you can bite it off in chunks. Identify a subset of outcomes Identify appropriate measures Choose an appropriate assessment method Choose an appropriate research design Collect the data Analyze & interpret the data Disseminate the findings Take action Wash, Rinse, Repeat!

S TUDENT A FFAIRS R ESEARCH AND A SSESSMENT Your turn… Identify outcomes Gather evidence Interpret evidence Refine efforts

S TUDENT A FFAIRS R ESEARCH AND A SSESSMENT Example from the PRCC: By the end of the World Cultural workshop students will be able to describe four unique traditions, values and beliefs of other races and cultures. By the end of the social justice retreat, students will demonstrate the ability to incorporate social justice techniques.

S TUDENT A FFAIRS R ESEARCH AND A SSESSMENT Audience By the end of the World Cultural workshop students will be able to describe four unique traditions, values and beliefs to other races and cultures. By the end of the social justice retreat, students will demonstrate the ability to incorporate social justice techniques.

S TUDENT A FFAIRS R ESEARCH AND A SSESSMENT Behavior By the end of the World Cultural workshop students will be able to describe four unique traditions, values and beliefs of other races and cultures. By the end of the social justice retreat, students will demonstrate the ability to incorporate social justice techniques.

S TUDENT A FFAIRS R ESEARCH AND A SSESSMENT Condition By the end of the World Cultural workshop students will be able to describe four unique traditions, values and beliefs to other races and cultures. By the end of the social justice retreat, students will demonstrate the ability to incorporate social justice techniques.

S TUDENT A FFAIRS R ESEARCH AND A SSESSMENT Next… Identify outcomes Gather evidence Interpret evidence Refine efforts

S TUDENT A FFAIRS R ESEARCH AND A SSESSMENT Determine appropriate measures Attitudes? Experiences? Knowledge? Abilities? Retention? Graduation? Employment?

S TUDENT A FFAIRS R ESEARCH AND A SSESSMENT Some types of assessments Standardized exams Test of abilities or knowledge Simulation or performance appraisals Interviews and focus groups External examiners Surveys Archival records and transcript analysis Portfolios Behavior observations Student self-evaluations Reflective writing Minute papers/muddiest point

S TUDENT A FFAIRS R ESEARCH AND A SSESSMENT Tests ◦ Can be used to measure a specific skill or knowledge base ◦ “Objective” ◦ Feasible? Surveys ◦ Attitudes ◦ Experiences ◦ Self-reported abilities & knowledge ◦ Can be confidential/anonymous ◦ Can reach large numbers of people

S TUDENT A FFAIRS R ESEARCH AND A SSESSMENT Interviews ◦ Rich & detailed information ◦ Body language can be helpful ◦ Respondents may feel inhibited ◦ Repeated interviews can build trust/comfort ◦ Time consuming Focus groups ◦ Many of the same benefits as interviews, but not as in-depth ◦ Participants “feed off of” each other ◦ Can reach more people than interviews

S TUDENT A FFAIRS R ESEARCH AND A SSESSMENT Back to the PRCC: By the end of the World Cultural Workshop students will be able to describe four unique traditions, values and beliefs of other races and cultures. Measure: Knowledge Some possible assessment methods: ◦ Knowledge test (direct evidence) ◦ Survey self assessment of knowledge (indirect) ◦ Portfolio ◦ Reflective writing Research design could be ex-post facto, pre- test/post-test, etc.

S TUDENT A FFAIRS R ESEARCH AND A SSESSMENT Next… Identify outcomes Gather evidence Interpret evidence Refine efforts

S TUDENT A FFAIRS R ESEARCH AND A SSESSMENT What if: World Cultural Workshop participants rated their knowledge of other cultures more highly BEFORE the workshop than after (pre-/post-test design)? In a reflective writing exercise you notice that participants are commonly mismatching the culture/race and its unique characteristics?

S TUDENT A FFAIRS R ESEARCH AND A SSESSMENT Next… Identify outcomes Gather evidence Interpret evidence Refine efforts

S TUDENT A FFAIRS R ESEARCH AND A SSESSMENT Potential refinements to the World Cultural Workshop based on our assessment Have a single assessment in which students focus on what/how much they learned in the workshop, rather than a pre-/post-test comparison. Cover a smaller number of races/cultures and spend more time on each. Consider refining objectives

S TUDENT A FFAIRS R ESEARCH AND A SSESSMENT And again... Identify outcomes Gather evidence Interpret evidence Refine efforts

S TUDENT A FFAIRS R ESEARCH AND A SSESSMENT Questions? If you are driving at the speed of light and you turn on your headlights, do they do anything? How your boss is likely to make assessment plans:

S TUDENT A FFAIRS R ESEARCH AND A SSESSMENT Resources & References Astin, A. W. (1993). Assessment for excellence: The philosophy and practice of assessment and evaluation in higher education. American Council on Education Series on Higher Education. Phoenix, AZ: Oryx Press. Improving Educational Programming online training module: Pope, R. L., Reynolds, A. L., Mueller, J. A., & Cheatham, H.E. (2004). Multicultural competence in student affairs. San Francisco: Jossey-Bass. SARA Assessment Resources website: Volkwein, J. F. (2009). A model for Assessing Institutional Effectiveness. In J. F. Volkwein (ed.), Assessing Student Outcomes: Why, who, what, how? New Directions for Institutional Research Assessment Supplement (pp ). San Francisco: Jossey-Bass.