1. Exploring High Quality Student Affairs Learning Outcomes Assessment Practices at Three American Research Universities
Adam S. Green, Ed.D.
West Virginia Higher Education Policy Commission
Research & Planning Analyst
8 June 2007
International Assessment & Retention Conference

2. Problem Shift in the role of student affairs professionals
From service providers to educators
From satisfaction measures to learning outcomes
No research on how those who have responded to calls to assess have successfully and fully implemented their student learning outcomes assessment plans

3. What is Assessment? Assessment defined
“The process of gathering and discovering information from multiple and diverse sources in order to develop a deep understanding of what students know, understand and can do with their knowledge as a result of their educational experiences; the process culminates when assessment results are used to improve subsequent learning” (Huba & Freed, 2000, p. 8).
Knowledge, skills, attitudes, and habits of mind that students take with them from a learning experience (Suskie, 2004, p. 75).
Explain how this differes from EVAL

4. Purpose & Significance of Study
To explore high quality student affairs learning outcomes assessment practices at three American Research Universities in order to:
Produce a rich & descriptive explanation of model student affairs assessment practices.
Provide insight into how these student affairs divisions are enhancing student learning.
Provide information from which to better understand what these divisions learned from their assessment processes, thus allowing others to develop high quality assessment strategies.

5. Research Questions
What are the student learning outcomes articulated in the student affairs assessment plans?
How are these learning outcomes assessed?
Who is involved in the assessment planning process, and what are the major responsibilities of these individuals?

6. Research Questions Continued
Are student affairs educators collaborating with others, such as academic affairs, in the creation and administration of assessments? How?
How is student affairs assessment data used to enhance student learning experiences?
What successes and challenges do student affairs educators face as they implement their assessment plans?

7. Conceptual Framework: Assessment Process
Create & articulate student learning outcomes
Use results to enhance student learning experiences
Select assessment participants
Analyze assessment results
Develop or select assessment measures
Evaluate assessment process
Report assessment results
Implement assessment

8. Review of Literature Calls for reform
Articulating student learning outcomes
Using results to enhance student learning experiences
Sustaining student affairs assessment

9. Research Design Qualitative research design Case study approach
Multiple case design—three Research Universities
Case institutions identified as those with high quality student affairs learning outcomes assessment models will guide case selection.
Strengths and limitations of research design
Generalizability
Triangulation
Reliability and validity

10. Pilot Study Pilot Study Data collection On-site interviews with:
Student Affairs Vice President
Two Assistant Student Affairs Vice Presidents
One Director of Multicultural Services and Disability Support Services
Modifications made based upon pilot study feedback
Additional script—explanation of learning outcomes
Move question #1 to question #3 in interview protocol
Add a final question: Should I need to contact you…
Make sure all participant invitation letters are photocopied in color
Follow-up s to participants

11. Research Methods Site Selection
Selection of Sites –Initially 15 possible sites
Three sites participated in this study
2 Doctoral Research Extensive Institutions
1 Doctoral Research Intensive Institution
Initiating contact and gaining approval
Assessment Directors/Coordinator served as liaisons for study—they identified appropriate participants based upon researcher’s criteria

12. Research Methods Purposeful Sampling—Researcher’s Selection Criteria
One Senior Student Affairs Officer (if not available, then an Assistant Senior Student Affairs Officer)
One Director/Coordinator of Student Affairs Assessment
Three Student Affairs Assessment Committee/Council Members
Three Other Student Affairs Professional Staff Members (with at least one serving in the capacity of unit director and one non-unit director)
Purposeful Sampling—Number of participants
8 participants at Western
8 participants at Southern
9 participants at Eastern
25 total

13. Research Methods Data collection—Interviews and Document Analysis 1 X
Points of data aligned with research questions
Research Question
Interviews
Assessment Plans
Assessment Reports
Other Documents 1 X 2 3 4 5 6

14. Research Methods Data organization and analysis Concurrent process
Analysis within case institution before comparing
Coding procedures

15. Overview of Case Institutions
Carnegie Classification
(at time of study)
Founded
Enrollment
Accrediting Body
Southern
University
Research I Extensive
1880s
Public
Land Grant
Approx 26,300
Fall 2004
SACS
Western
1850s
Approx
19,200
NWCCU
Eastern
Research I Intensive
1970s
30,000
Fall 2005

16. Overview and Rank of Participants
Level
Southern
University
Western
Eastern
Vice President 1
Asst./Assoc. VP or Dean 3
Director of Unit 5 2
Asst./Assoc. Director of Unit
Unit Staff Member

17. Case Institutions Student Affairs Assessment History
Southern University
Western University
Eastern University
Began efforts in
Hired full-time director of student affairs assessment in
Majority of units submitted assessment plans in
Second cycle of assessment planning and reporting began in
Assessment committee created broad learning objectives (2005)
Began efforts in 1995 (satisfaction & benchmarks)
Hired full-time director of student affairs assessment in
Few units articulated learning outcomes in assessment plans in
Majority of units articulated learning outcomes in assessment plans in
Assessment committee created broad learning goals and assessment handbook (2005)
Began efforts in 2000
Creation of standardized assessment method—survey in
New VP of Student Affairs named in
New Coordinator of Assessment identified in 2005
Standardized Survey “phased-out” in 2005
Division partnered with external assessment consulting group in 2005

18. RQ 1: What are the student learning outcomes articulated in the student affairs assessment plans?
Taxonomy
Southern University
Western University
Eastern University
% of Outcomes
N=71
Cognitive 22 24
N/A
64.8
Affective 10 15
35.2
Total 32 39
100.0

19. RQ 1: What are the student learning outcomes articulated in the student affairs assessment plans?
Cognitive
Taxonomy
Southern University
Western University
Eastern University
% of Outcomes
N=46
Remember 5 7
N/A
26.1
Understand 6 8
30.4
Apply 9
39.1
Analyze 2
4.4
Evaluate
0.0
Create

20. RQ 1: What are the student learning outcomes articulated in the student affairs assessment plans?
Affective
Taxonomy
Southern University
Western University
Eastern University
% of Outcomes
N=25
Receive 3 7
N/A
40.0
Respond 5
Value 2
20.0
Organize
0.0
Internalize values

21. RQ 2: How are these learning outcomes assessed?
Most common methods identified at Southern and Western
Method
Number of Report Methods
% Based Upon Total # of Outcomes (N=71)
Survey 41
57.8
Interview 13
18.3
Observation 11
15.5
Document Analysis 7
9.9
Pre- and Post-Test

22. RQ 2: How are these learning outcomes assessed?
Eastern University’s Approach
Primarily centralized method—Survey (1=strongly disagree & 5=strongly agree)
2002 Administered at 273 programs
2003 Administered at 575 programs
Limited evidence of unit level assessment
Student Learning Outcome
2002 Mean
2003 Mean
Mean Difference
I learned new information
4.23
4.32
+0.09
I can apply what I have learned
4.10
4.25
+0.15
I learned new skills
3.76
4.03
+0.27
I expect to do things differently
3.85
3.97
+0.12
My attitudes have changed
3.62
3.80
+0.18

23. RQ3: Who is involved in the assessment planning process, and what are the major responsibilities of these individuals?
Four Vital Levels of Participation
Vice Presidents
Leadership & Financial support
Assessment Directors/Coordinator
Facilitator & Change agent
Assessment Committees
Create reporting timelines and guidelines
Initiate assessment dialogue
Review unit level assessment plans and provide feedback
Unit Level
Worker bees

24. Very little evidence of collaboration across cases
RQ 4: Are Student Affairs Educators Collaborating With Others, Such as Academic Affairs in the Creation and Administration of Assessments? How?
Very little evidence of collaboration across cases
Few Examples Identified in Study Included:
Southern University: Multicultural Services and Greek Life
Western University: Assessment Committee’s creation of six broad learning goals
Eastern University: Office of Diversity Programs and Provost’s Office

25. Southern and Western Universities
RQ 5: How are Student Affairs Assessment Data Used to Enhance Student Learning Experiences?
Southern and Western Universities
Type of Decision
N=29
% of Decisions Made
Modify education program or service 13
44.8
Modify assessment instrument 8
27.6
Continue same practices 7
24.1
Enhance assessment administration training 1
3.5

26. Recommendations Based Upon Standardized Survey Results
RQ 5: How are Student Affairs Assessment Data Used to Enhance Student Learning Experiences?
Eastern University
Recommendations Based Upon Standardized Survey Results
Use data that exists on campus and develop a process for analyzing and using existing data.
Ensure that assessment moves from sporadic to ongoing. Having a half or full-time assessment person who has the skills to coordinate the efforts would be helpful.
Develop a process for ensuring the flow and use of information at the unit level.
Highlight and support exemplary work being done at the unit level (e.g. Orientation’s study of Orientation Leaders).

27. RQ 6: What Successes and Challenges do Student Affairs Educators Face as They Implement Their Assessment Plans?
Southern University
Successes
Challenges
Student affairs educators see value in assessment (75%)
Time constraints (75%)
Creation of systematic data collection strategies within the division (25%)
Limited financial resources (63%)
Making decisions based upon assessment data (25%)
Limited assessment expertise within individual units (25%)
Creation of division-level broad learning objectives (25%)
Assessment is not a priority to everyone (25%)

28. RQ 6: What Successes and Challenges do Student Affairs Educators Face as They Implement Their Assessment Plans?
Western University
Successes
Challenges
Student affairs educators enjoy assessment professional development learning opportunities (50%)
Time constraints (37.5%)
Student affairs educators demonstrate a high level of dedication to the initiative (50%)
Additional education is needed (37.5%)
Creation of a division-level director of assessment position (25%)
There is an absence of an institution-wide assessment culture (25%)
Student affairs educators are viewed as leaders of the assessment initiative at their institution (25%)
Some units struggle to articulate manageable outcomes (25%)

29. RQ 6: What Successes and Challenges do Student Affairs Educators Face as They Implement Their Assessment Plans?
Eastern University
Successes
Challenges
Student affairs division has made progress since 2000 (33%)
Assessment is not a priority to everyone (56%)
Student affairs educators are now making decisions based upon data (22%)
Limited assessment expertise within individual units (11%)
Select student affairs units have excelled (22%)
Limited financial resources (11%)
The overall image of the division has been enhanced (22%)
Standardized survey became cumbersome (11%)

30. Conclusions
Assessment activities across cases depended highly upon four levels of support
Vice President, Coordinator/Director of Assessment, Assessment Committees, and Unit Level Staff
Decentralized model of assessment, facilitated by a coordinator or director, is most appropriate in student affairs
Southern and Western’s articulated learning outcomes were primarily cognitive (65%) and less were affective (35%)
Surveys were most common method used to assess student affairs learning outcomes

31. Conclusions
Assessment planning and implementation is primarily a single-unit task with very little evidence of collaboration between student affairs units or outside of the division
Less than half (45%) of decisions made at Southern and Western based upon assessment results were to modify educational program or service. Other decisions included: modify assessment method, continue same practices, & enhance assessment administration training
Successes and challenges varied across institutions

32. Recommendations for Practice
Engaging in student affairs learning outcomes assessment efforts requires adequate level of support from division’s leaders.
Student affairs divisions should identify broad learning goals or objectives based upon the institution’s mission to provide guidance to unit leaders when they are developing their own learning outcomes.
Student affairs educators must consider carefully what they want students to learn as a result of their program or service.
Continuous professional development opportunities must be provided to help unit level staff become assessment experts.

33. Recommendations for Future Research
This study could be replicated at the unit level
(e.g., Compare five career service units’ assessment practices)
Comparisons between student affairs assessment efforts that are located within the various accrediting bodies across the United States
Examination of assessment recommendations provided from national professional organizations and how units are implementing those recommendations
Because evidence of collaboration was limited in this study, it would be useful to examine successful collaborative assessment efforts within the field

34. Adam S. Green, Ed.D. green@hepc.wvnet.edu 304.558.1112
Questions?
Comments?
Adam S. Green, Ed.D