Exploring High Quality Student Affairs Learning Outcomes Assessment Practices at Three American Research Universities Adam S. Green, Ed.D. West Virginia Higher Education Policy Commission Research & Planning Analyst 8 June 2007 International Assessment & Retention Conference

Problem Shift in the role of student affairs professionals From service providers to educators From satisfaction measures to learning outcomes No research on how those who have responded to calls to assess have successfully and fully implemented their student learning outcomes assessment plans

What is Assessment? Assessment defined “The process of gathering and discovering information from multiple and diverse sources in order to develop a deep understanding of what students know, understand and can do with their knowledge as a result of their educational experiences; the process culminates when assessment results are used to improve subsequent learning” (Huba & Freed, 2000, p. 8). Knowledge, skills, attitudes, and habits of mind that students take with them from a learning experience (Suskie, 2004, p. 75). Explain how this differes from EVAL

Purpose & Significance of Study To explore high quality student affairs learning outcomes assessment practices at three American Research Universities in order to: Produce a rich & descriptive explanation of model student affairs assessment practices. Provide insight into how these student affairs divisions are enhancing student learning. Provide information from which to better understand what these divisions learned from their assessment processes, thus allowing others to develop high quality assessment strategies.

Research Questions What are the student learning outcomes articulated in the student affairs assessment plans? How are these learning outcomes assessed? Who is involved in the assessment planning process, and what are the major responsibilities of these individuals?

Research Questions Continued Are student affairs educators collaborating with others, such as academic affairs, in the creation and administration of assessments? How? How is student affairs assessment data used to enhance student learning experiences? What successes and challenges do student affairs educators face as they implement their assessment plans?

Conceptual Framework: Assessment Process Create & articulate student learning outcomes Use results to enhance student learning experiences Select assessment participants Analyze assessment results Develop or select assessment measures Evaluate assessment process Report assessment results Implement assessment

Review of Literature Calls for reform Articulating student learning outcomes Using results to enhance student learning experiences Sustaining student affairs assessment

Research Design Qualitative research design Case study approach Multiple case design—three Research Universities Case institutions identified as those with high quality student affairs learning outcomes assessment models will guide case selection. Strengths and limitations of research design Generalizability Triangulation Reliability and validity

Pilot Study Pilot Study Data collection On-site interviews with: Student Affairs Vice President Two Assistant Student Affairs Vice Presidents One Director of Multicultural Services and Disability Support Services Modifications made based upon pilot study feedback Additional script—explanation of learning outcomes Move question #1 to question #3 in interview protocol Add a final question: Should I need to contact you… Make sure all participant invitation letters are photocopied in color Follow-up emails to participants

Research Methods Site Selection Selection of Sites –Initially 15 possible sites Three sites participated in this study 2 Doctoral Research Extensive Institutions 1 Doctoral Research Intensive Institution Initiating contact and gaining approval Assessment Directors/Coordinator served as liaisons for study—they identified appropriate participants based upon researcher’s criteria

Research Methods Purposeful Sampling—Researcher’s Selection Criteria One Senior Student Affairs Officer (if not available, then an Assistant Senior Student Affairs Officer) One Director/Coordinator of Student Affairs Assessment Three Student Affairs Assessment Committee/Council Members Three Other Student Affairs Professional Staff Members (with at least one serving in the capacity of unit director and one non-unit director) Purposeful Sampling—Number of participants 8 participants at Western 8 participants at Southern 9 participants at Eastern 25 total

Research Methods Data collection—Interviews and Document Analysis 1 X Points of data aligned with research questions Research Question Interviews Assessment Plans Assessment Reports Other Documents 1 X 2 3 4 5 6

Research Methods Data organization and analysis Concurrent process Analysis within case institution before comparing Coding procedures

Overview of Case Institutions Carnegie Classification (at time of study) Founded Enrollment Accrediting Body Southern University Research I Extensive 1880s Public Land Grant Approx 26,300 Fall 2004 SACS Western 1850s Approx 19,200 NWCCU Eastern Research I Intensive 1970s 30,000 Fall 2005

Overview and Rank of Participants Level Southern University Western Eastern Vice President 1 Asst./Assoc. VP or Dean 3 Director of Unit 5 2 Asst./Assoc. Director of Unit Unit Staff Member

Case Institutions Student Affairs Assessment History Southern University Western University Eastern University Began efforts in 2002-03 Hired full-time director of student affairs assessment in 2003-04 Majority of units submitted assessment plans in 2004-05 Second cycle of assessment planning and reporting began in 2005-06 Assessment committee created broad learning objectives (2005) Began efforts in 1995 (satisfaction & benchmarks) Hired full-time director of student affairs assessment in 2001-02 Few units articulated learning outcomes in assessment plans in 2003-04 Majority of units articulated learning outcomes in assessment plans in 2004-05 Assessment committee created broad learning goals and assessment handbook (2005) Began efforts in 2000 Creation of standardized assessment method—survey in 2001-02 New VP of Student Affairs named in 2003-04 New Coordinator of Assessment identified in 2005 Standardized Survey “phased-out” in 2005 Division partnered with external assessment consulting group in 2005

RQ 1: What are the student learning outcomes articulated in the student affairs assessment plans? Taxonomy Southern University Western University Eastern University % of Outcomes N=71 Cognitive 22 24 N/A 64.8 Affective 10 15 35.2 Total 32 39 100.0

RQ 1: What are the student learning outcomes articulated in the student affairs assessment plans? Cognitive Taxonomy Southern University Western University Eastern University % of Outcomes N=46 Remember 5 7 N/A 26.1 Understand 6 8 30.4 Apply 9 39.1 Analyze 2 4.4 Evaluate 0.0 Create

RQ 1: What are the student learning outcomes articulated in the student affairs assessment plans? Affective Taxonomy Southern University Western University Eastern University % of Outcomes N=25 Receive 3 7 N/A 40.0 Respond 5 Value 2 20.0 Organize 0.0 Internalize values

RQ 2: How are these learning outcomes assessed? Most common methods identified at Southern and Western Method Number of Report Methods % Based Upon Total # of Outcomes (N=71) Survey 41 57.8 Interview 13 18.3 Observation 11 15.5 Document Analysis 7 9.9 Pre- and Post-Test

RQ 2: How are these learning outcomes assessed? Eastern University’s Approach Primarily centralized method—Survey (1=strongly disagree & 5=strongly agree) 2002 Administered at 273 programs 2003 Administered at 575 programs Limited evidence of unit level assessment Student Learning Outcome 2002 Mean 2003 Mean Mean Difference I learned new information 4.23 4.32 +0.09 I can apply what I have learned 4.10 4.25 +0.15 I learned new skills 3.76 4.03 +0.27 I expect to do things differently 3.85 3.97 +0.12 My attitudes have changed 3.62 3.80 +0.18

RQ3: Who is involved in the assessment planning process, and what are the major responsibilities of these individuals? Four Vital Levels of Participation Vice Presidents Leadership & Financial support Assessment Directors/Coordinator Facilitator & Change agent Assessment Committees Create reporting timelines and guidelines Initiate assessment dialogue Review unit level assessment plans and provide feedback Unit Level Worker bees

Very little evidence of collaboration across cases RQ 4: Are Student Affairs Educators Collaborating With Others, Such as Academic Affairs in the Creation and Administration of Assessments? How? Very little evidence of collaboration across cases Few Examples Identified in Study Included: Southern University: Multicultural Services and Greek Life Western University: Assessment Committee’s creation of six broad learning goals Eastern University: Office of Diversity Programs and Provost’s Office

Southern and Western Universities RQ 5: How are Student Affairs Assessment Data Used to Enhance Student Learning Experiences? Southern and Western Universities Type of Decision N=29 % of Decisions Made Modify education program or service 13 44.8 Modify assessment instrument 8 27.6 Continue same practices 7 24.1 Enhance assessment administration training 1 3.5

Recommendations Based Upon Standardized Survey Results RQ 5: How are Student Affairs Assessment Data Used to Enhance Student Learning Experiences? Eastern University Recommendations Based Upon Standardized Survey Results Use data that exists on campus and develop a process for analyzing and using existing data. Ensure that assessment moves from sporadic to ongoing. Having a half or full-time assessment person who has the skills to coordinate the efforts would be helpful. Develop a process for ensuring the flow and use of information at the unit level. Highlight and support exemplary work being done at the unit level (e.g. Orientation’s study of Orientation Leaders).

RQ 6: What Successes and Challenges do Student Affairs Educators Face as They Implement Their Assessment Plans? Southern University Successes Challenges Student affairs educators see value in assessment (75%) Time constraints (75%) Creation of systematic data collection strategies within the division (25%) Limited financial resources (63%) Making decisions based upon assessment data (25%) Limited assessment expertise within individual units (25%) Creation of division-level broad learning objectives (25%) Assessment is not a priority to everyone (25%)

RQ 6: What Successes and Challenges do Student Affairs Educators Face as They Implement Their Assessment Plans? Western University Successes Challenges Student affairs educators enjoy assessment professional development learning opportunities (50%) Time constraints (37.5%) Student affairs educators demonstrate a high level of dedication to the initiative (50%) Additional education is needed (37.5%) Creation of a division-level director of assessment position (25%) There is an absence of an institution-wide assessment culture (25%) Student affairs educators are viewed as leaders of the assessment initiative at their institution (25%) Some units struggle to articulate manageable outcomes (25%)

RQ 6: What Successes and Challenges do Student Affairs Educators Face as They Implement Their Assessment Plans? Eastern University Successes Challenges Student affairs division has made progress since 2000 (33%) Assessment is not a priority to everyone (56%) Student affairs educators are now making decisions based upon data (22%) Limited assessment expertise within individual units (11%) Select student affairs units have excelled (22%) Limited financial resources (11%) The overall image of the division has been enhanced (22%) Standardized survey became cumbersome (11%)

Conclusions Assessment activities across cases depended highly upon four levels of support Vice President, Coordinator/Director of Assessment, Assessment Committees, and Unit Level Staff Decentralized model of assessment, facilitated by a coordinator or director, is most appropriate in student affairs Southern and Western’s articulated learning outcomes were primarily cognitive (65%) and less were affective (35%) Surveys were most common method used to assess student affairs learning outcomes

Conclusions Assessment planning and implementation is primarily a single-unit task with very little evidence of collaboration between student affairs units or outside of the division Less than half (45%) of decisions made at Southern and Western based upon assessment results were to modify educational program or service. Other decisions included: modify assessment method, continue same practices, & enhance assessment administration training Successes and challenges varied across institutions

Recommendations for Practice Engaging in student affairs learning outcomes assessment efforts requires adequate level of support from division’s leaders. Student affairs divisions should identify broad learning goals or objectives based upon the institution’s mission to provide guidance to unit leaders when they are developing their own learning outcomes. Student affairs educators must consider carefully what they want students to learn as a result of their program or service. Continuous professional development opportunities must be provided to help unit level staff become assessment experts.

Recommendations for Future Research This study could be replicated at the unit level (e.g., Compare five career service units’ assessment practices) Comparisons between student affairs assessment efforts that are located within the various accrediting bodies across the United States Examination of assessment recommendations provided from national professional organizations and how units are implementing those recommendations Because evidence of collaboration was limited in this study, it would be useful to examine successful collaborative assessment efforts within the field

Adam S. Green, Ed.D. green@hepc.wvnet.edu 304.558.1112 Questions? Comments? Adam S. Green, Ed.D. green@hepc.wvnet.edu 304.558.1112