1. Building an Evidence-Based Culture in Student Affairs Presented at the Ivy Tech Community College Student Affairs Leadership Summit July 1, 2009 By Trudy W. Banta Professor of Higher Education and Senior Advisor to the Chancellor for Academic Planning and Evaluation Indiana University-Purdue University Indianapolis 355 N. Lansing St., AO 140 Indianapolis, Indiana 46202-2896 tbanta@ iupui.edu http://www.planning.iupui.edu

2. Outline 1. Characteristics of an evidence-based culture 2. An excursion into national accountability 3. Assessment in student affairs © TWBANTA-IUPUI

3. Why Assess Outcomes in Student Affairs? 1. Improve programs & services 2. Demonstrate accountability 3. Survive 4. Inform decision-making 5. Maintain accreditation © TWBANTA-IUPUI

4. St. Louis Community College (1) Developing a Culture of Inquiry Faculty and staff collaborate to - discover - interpret - act on INFORMATION © TWBANTA-IUPUI

5. St. Louis Community College (2) College-wide assessment used to  Revise new-student orientation  Track student success at UM-SL and improve counseling/advising  Monitor impact of late registration  Monitor effectiveness of new tuition payment plan © TWBANTA-IUPUI

6. St. Louis Community College (3) Impact of Culture of Inquiry  More collaboration in pursuing mission  More thoughtful interpretation and use of data  Better follow-up on identified issues  More use of outside resources like CCSSE and CAS Standards © TWBANTA-IUPUI

7. Council for the Advancement of Standards in Higher Education www.cas.edu © TWBANTA-IUPUI

8. Plan Implement Evaluate Improve Culture of Evidence © TWBANTA-IUPUI

10. PLANNING 1. Campus mission, goals 2. Unit goals aligned 3. Programs based on assessable goals with PIs 4. Annual reports on the Web © TWBANTA-IUPUI

11. Outline for Annual Reports  IUPUI Theme  Unit Goal  Objective Actions Taken Actions Planned  Evidence of Progress © TWBANTA-IUPUI

12. PAII Evaluation Services 1. Assessment of learning 2. Surveys 3. Program reviews 4. Performance indicators 5. Program cost analysis 6. Web-based evaluation tools 7. Program evaluation/action research 8. Accreditation © TWBANTA-IUPUI

13. IMPROVEMENT AGENDA 1. Reporting to internal constituents 2. Demonstrating accountability 3. Proposing improvement -initiatives based on findings 4. Improving assessment methods © TWBANTA-IUPUI

14. Surveys 1. Enrolled Students  Our own  NSSE 2. Graduates 3. Employers 4. Stop outs 5. Faculty 6. Staff © TWBANTA-IUPUI

15. Information Gateway http://reports.iupui.edu/gateway/ Information about  Students  Faculty  Staff  Alumni  Finances © TWBANTA-IUPUI

16. Since 1993 Campus-wide surveys have stimulated changes in  Curricula  Advising  Increased writing practice  Increased attention to first-year experiences  Placement of graduates © TWBANTA-IUPUI

17. What is ABC? ABC is a costing methodology based upon the fact that different activities and products consume different proportions of resources Resources Product A Product B Product C Activities

18. Activities performed to produce a graduate © TWBANTA-IUPUI

19. Some Activities within Orientation 1. Planning the program 2. Inviting students and parents 3. Purchasing/duplicating materials 4. Obtaining spaces and equipment 5. Setting up spaces and equipment 6. Providing the program 7. Evaluating the program 8. Cleaning space and returning equipment © TWBANTA-IUPUI

20. What Is ABC? Traditional vs. ABC Traditional Accounting Perspective n Salary & wages 1,350,000 n Benefits 495,000 n Travel 45,000 n Facilities 220,000 n Supplies 90,000 Total $2,200,000 Activity-Based Perspective n Planning 940,000 n Materials 430,000 n Spaces & equipment 250,000 n Program 350,000 n Evaluation 230,000 ________________________ Total $2,200,000 © TWBANTA-IUPUI

21. Some Applications of Economic Model 1. Estimate costs of administrative services as compared to cost of outsourcing 2. Determine fees for various programs 3. Restructure processes to expedite work flow and minimize costs © TWBANTA-IUPUI

22. Since 1992 Activity-based Costing has stimulated changes in n Planning n Budgeting n Assessment © TWBANTA-IUPUI

23. Elements of Program Review  Self Study  Review by Respected Peers  Recommendations  Follow-up © TWBANTA-IUPUI

24. Goals of Program Review at IUPUI  To improve student learning  To assess and improve program quality  To increase cross-disciplinary collaboration  To enhance community connections  To reinforce importance of aligning unit and campus planning © TWBANTA-IUPUI

25. Following the Review 1. Department reviews the recommendations 2. Department prepares written response 3. Department presents response in administrative hearing 4. Responsible administrators agree upon responsive action(s) 5. Responsive actions are implemented 6. Re-review ensures action © TWBANTA-IUPUI

26. Since 1995 Program Reviews have stimulated changes in  Planning for the future  Research emphases  Faculty hiring priorities  Advisory councils  Cross-disciplinary collaboration © TWBANTA-IUPUI

27. Program Review at IUPUI www.planning.iupui.edu/assessment/ © TWBANTA-IUPUI

28. Goal and Objectives for Student Learning  Enhance undergraduate student learning and success 1. Strengthen generic skills 2. Provide honors programming 3. Offer learning communities 4. Strengthen advising 5. Provide tutoring and mentoring © TWBANTA-IUPUI

29. Employ Multiple Methods 1) Direct  Projects, papers, tests, observations 2) Indirect  Questionnaires, interviews, focus groups  Unobtrusive measures Syllabi, transcripts © TWBANTA-IUPUI

30. In a Comprehensive Assessment Program... INVOLVE t Students t Faculty t Student Affairs Staff t Administrators t Graduates t Employers © TWBANTA-IUPUI

31. Since 1994 Assessment of Learning has stimulated changes in  Student support programs  Curriculum  Methods of instruction  Internships  Methods of assessment © TWBANTA-IUPUI

32. Establishing a Culture of Evidence takes  Strong leadership  Support  Time  Evidence that findings are used to guide improvements © TWBANTA-IUPUI

33. Open sharing of information and evidence-based decision-making  Financial and satisfaction data for units  Annual planning/budgeting hearings  Performance indicators derived from unit reports over time  Campus performance report for community © TWBANTA-IUPUI

34. Program Review & Assessment Committee Ÿ 2 reps from each school Ÿ 2 librarians Ÿ Other units  Student Life  Faculty Development  Internship coordinator © TWBANTA-IUPUI

35. Characterizing the Culture New initiatives require assessment  University College student support programs  Distance learning  New academic programs © TWBANTA-IUPUI

36. Characterizing the Culture  Appointment of Assessment Specialists  University College  Student Life  Service Learning  Library  Enrollment Services  Appointment of Associate Deans for Assessment © TWBANTA-IUPUI

37. Group Assessment Has Failed to Demonstrate Institutional Accountability Focus on improvement at unit level Rare aggregation of data centrally Too few faculty involved HE scholars focused on K-12 assessment

38. © TWBANTA-IUPUI Now We Have the Press to Assess with a Test

39. © TWBANTA-IUPUI 2006 Commission on the Future of Higher Education  We need a simple way to compare institutions  The results of student learning assessment, including value added measurements (showing skill improvement over time) should be... reported in the aggregate publicly.

40. © TWBANTA-IUPUI Voluntary System of Accountability Report Scores in critical thinking, written communication, analytic reasoning using CAAP MAPP CLA

41. © TWBANTA-IUPUI At the University of Tennessee CAAP Academic Profile (now MAPP) COMP (like CLA and withdrawn by 1990) College BASE

42. © TWBANTA-IUPUI In TN We Learned 1) No test measured 30% of gen ed skills 2) Tests of generic skills measure primarily prior learning 3) Reliability of value added =.1 4) Test scores give few clues to guide improvement actions

43. © TWBANTA-IUPUI An Inconvenient Truth.9 = the correlation between SAT and CLA scores of institutions thus 81% of the variance in institutions’ scores is due to prior learning

44. © TWBANTA-IUPUI How Much of the Variance in Senior Scores is Due to College Impact? Student motivation to attend that institution (mission differences) Student mix based on age, gender socioeconomic status race/ethnicity transfer status college major

45. © TWBANTA-IUPUI How Much of the Variance in Senior Scores is Due to College Impact? (continued) Student motivation to do well Sampling error Measurement error Test anxiety  College effects ______ 19 %

46. Better Ways to Demonstrate Accountability 1. Performance Indicators  Access, social mobility  Diversity  Workforce development  Economic development  Engaging student experience © TWBANTA-IUPUI

47. Better Ways to Demonstrate Accountability 2. Measures of Learning  Standardized tests in major fields  Internship performance  Senior projects  Electronic portfolios  External examiners © TWBANTA-IUPUI

48. Student Electronic Portfolio Students take responsibility for demonstrating core skills Unique individual skills and achievements can be emphasized Multi-media opportunities extend possibilities Metacognitive thinking is enhanced through reflection on contents - Sharon J. Hamilton IUPUI

49. Student Development  Involve Student Affairs staff in planning for assessment  Set expectations for student development s Recreation s Leadership development s Mental and physical health s Spiritual development  Monitor student progress  Take warranted improvement actions © TWBANTA-IUPUI

50. Questions Guiding Assessment of Student Services 1. How many students know about the program? 2. How many students say they need the program? 3. How many students use the program? 4. Do students persist in the program? 5. What knowledge, skills, attitudes, values do students develop in the program? 6. How satisfied are students with the program? 7. Do program outcomes justify its cost? 8. Can the program be improved? 9. Should the program be terminated? © TWBANTA-IUPUI

51. Taxonomy of Educational Objectives (Bloom and Others, 1956) Cognitive domain categories Knowledge Comprehension Application Analysis Synthesis Evaluation Sample verbs for outcomes Identifies, defines, describes Explains, summarizes, classifies Demonstrates, computes, solves Differentiates, diagrams, estimates Creates, formulates, revises Criticizes, compares, concludes

52. © TWBANTA-IUPUI Planning for Learning and Assessment

53. Student Services Activity Records # participating in activities over the years # changing majors # meals served in Union Average age of students in residence halls # voting in campus elections # employers coming to campus to interview # visits to health clinic © TWBANTA-IUPUI

54. Student Affairs Learning Outcomes Leadership skills Self confidence Communication skills Time management skills Ability to manage stress Teamwork skills Overall health © TWBANTA-IUPUI

55. Measures of Student Learning  Questionnaires  Interviews  Focus groups  Observations  Transcript analysis © TWBANTA-IUPUI

56. Primary Trait Scoring Assigns scores to attributes (traits) of a task STEPS  Identify traits necessary for success in assignment  Compose scale or rubric giving clear definition to each point  Grade using the rubric © TWBANTA-IUPUI

57. Primary Trait Scoring ~ E xample ~ Trait Desired:Self expression of a feeling evoked by an event Scale: 1 - No real expression presented 2 - Feeling expressed but in- adequately described 3 - Expression generally competent 4 - Feeling expressed in great detail 5 - Detail plus intelligent response to feeling evoked © TWBANTA-IUPUI

58. Observation of Student Behavior (including participant observers like RAs) Check-list of behaviors for Actual setting Videotape of advising sessions Role-playing Simulation of incidents © TWBANTA-IUPUI

59. Faculty and Staff Development  Focus faculty and student affairs professionals on improving learning in and outside class  Attend conferences together  Study literature on student learning  Provide workshops on teaching and learning  Provide resources (e.g., grants, summer salary, release time) © TWBANTA-IUPUI

60. Virginia Commonwealth University 1. First-year English students wrote 1 response / week for 15 weeks 2. Faculty-student affairs teams read essays 3. Sample of writers interviewed 18 months later Responses to findings: Central advising center, new advising handbook, multicultural workshops © TWBANTA-IUPUI

61. Chicago State University Purpose: Determine reasons students leave TELEPHONE INTERVIEW Finding: No personal contact for 78% Response: Freshman seminar revised, early warning system, referral to mandatory learning strategies course © TWBANTA-IUPUI

62. Santa Barbara City College  Problem: Need to improve retention and satisfaction with goal achievement  COMMUNITY COLLEGE STUDENT EXPERIENCES QUESTIONNAIRE  Findings: Strong relationships between involvement and progress toward goals and satisfaction with instruction  Response: More extra-curricular activities, departmental clubs, space for group study © TWBANTA-IUPUI

63. Designing Effective Assessment: Principles & Profiles of Good Practice Trudy W. Banta Elizabeth A. Jones Karen E. Black Jossey-Bass (Wiley) 2009 © TWBANTA-IUPUI

64. Profiles  Invited over 1000  Received 146  Selected 49 for use in full  Categorized all 146 and published Web sites © TWBANTA-IUPUI

65. Outline for Profiles  Background and Purpose  Methods over ? Years  Resources Required  Findings  Use of Findings  Impact of Using Findings  Success Factors  Web sites © TWBANTA-IUPUI

66. Plan Implement Evaluate Improve Culture of Evidence

67. ~ Organization ~ of Principles & Profiles  Planning  Implementing  Improving & Sustaining - Building a Scholarship of Assessment Banta & Associates Jossey-Bass 2002 © TWBANTA-IUPUI

68. Planning Principles 1. Engaging stakeholders 2. Connecting assessment to valued goals & processes 3. Creating a written plan 4. Timing assessment 5. Building a culture based on evidence © TWBANTA-IUPUI

69. Implementation Principles 1. Providing leadership 2. Creating faculty/staff development 3. Assessing processes as well as outcomes 4. Communicating and using findings © TWBANTA-IUPUI

70. Improving/Sustaining Principles 1. Providing credible evidence of learning to multiple stakeholders 2. Reviewing assessment reports 3. Ensuring use of results 4. Evaluating the assessment process © TWBANTA-IUPUI

71. Cal State-Sacramento (1) Sources of Motivation for Assessment 1. New VP for Student Affairs 2. Reaccreditation looming 3. Enrollment & budget challenges 4. Pledge to become more data-driven and focused on student learning © TWBANTA-IUPUI

72. Cal State-Sacramento (2) 1. Align department & division missions 2. Develop SMART goals, 1 for student learning  Specific  Measurable  Aggressive, yet attainable  Results-oriented  Timely © TWBANTA-IUPUI

73. Cal State-Sacramento (3) Measures  Pre-post MC tests on policies, resources  Essays with rubrics (reinstatement)  Portfolios  Observation of skills (Leadership, RA reports on scenarios, role-playing) © TWBANTA-IUPUI

74. Cal State-Sacramento (4) Findings 1. Some SLOs met 2. Some SLOs not met 3. Some measures not effective 4. Too few participants to assess 5. Too many participants to assess effectively © TWBANTA-IUPUI

75. Cal State-Sacramento (5) Use of Findings 1. Better training for RAs in reporting 2. Better training for peer mentors in orientation (emphasizing policies) 3. More time to discuss films 4. Better Powerpoint presentations 5. Increase participation in counseling 6. Redesign vague test items © TWBANTA-IUPUI

76. IUPUI (1) Themed Learning Communities (TLCs)  3 first-year courses for a cohort  Themes such as Crime in America, Health & Wellness, African-American perspectives  Planning group involving faculty, student affairs, advisor, peer mentor © TWBANTA-IUPUI

77. IUPUI (2) TLCs & Control Group ~ MEASURES ~ 1. GPA 2. Persistence for 1 year 3. Common assignments 4. Student survey responses © TWBANTA-IUPUI

78. IUPUI (3) TLC Benefits 1. Higher GPAs 2. Higher persistence rates 3. See more connections across disciplines 4. Encounter more diverse perspectives 5. Experience more teamwork © TWBANTA-IUPUI

79. IUPUI TLCs (4) Use of Assessment Findings  More detailed collaborative planning  Improved common assignments  More cocurricular activities  More faculty from more disciplines  More TLCs! © TWBANTA-IUPUI

80. An Excellent Resource – Assessment Practice in Student Affairs by John H. Schuh & M. Lee Upcraft Jossey-Bass 2001 © TWBANTA-IUPUI

81. Assessment Institute in Indianapolis October 25-27, 2009 Westin Indianapolis www.planning.iupui.edu © TWBANTA-IUPUI