Building an Evidence-Based Culture in Student Affairs Presented at the Ivy Tech Community College Student Affairs Leadership Summit July 1, 2009 By Trudy W. Banta Professor of Higher Education and Senior Advisor to the Chancellor for Academic Planning and Evaluation Indiana University-Purdue University Indianapolis 355 N. Lansing St., AO 140 Indianapolis, Indiana iupui.edu

Outline 1. Characteristics of an evidence-based culture 2. An excursion into national accountability 3. Assessment in student affairs © TWBANTA-IUPUI

Why Assess Outcomes in Student Affairs? 1. Improve programs & services 2. Demonstrate accountability 3. Survive 4. Inform decision-making 5. Maintain accreditation © TWBANTA-IUPUI

St. Louis Community College (1) Developing a Culture of Inquiry Faculty and staff collaborate to - discover - interpret - act on INFORMATION © TWBANTA-IUPUI

St. Louis Community College (2) College-wide assessment used to  Revise new-student orientation  Track student success at UM-SL and improve counseling/advising  Monitor impact of late registration  Monitor effectiveness of new tuition payment plan © TWBANTA-IUPUI

St. Louis Community College (3) Impact of Culture of Inquiry  More collaboration in pursuing mission  More thoughtful interpretation and use of data  Better follow-up on identified issues  More use of outside resources like CCSSE and CAS Standards © TWBANTA-IUPUI

Council for the Advancement of Standards in Higher Education © TWBANTA-IUPUI

Plan Implement Evaluate Improve Culture of Evidence © TWBANTA-IUPUI

PLANNING 1. Campus mission, goals 2. Unit goals aligned 3. Programs based on assessable goals with PIs 4. Annual reports on the Web © TWBANTA-IUPUI

Outline for Annual Reports  IUPUI Theme  Unit Goal  Objective Actions Taken Actions Planned  Evidence of Progress © TWBANTA-IUPUI

PAII Evaluation Services 1. Assessment of learning 2. Surveys 3. Program reviews 4. Performance indicators 5. Program cost analysis 6. Web-based evaluation tools 7. Program evaluation/action research 8. Accreditation © TWBANTA-IUPUI

IMPROVEMENT AGENDA 1. Reporting to internal constituents 2. Demonstrating accountability 3. Proposing improvement -initiatives based on findings 4. Improving assessment methods © TWBANTA-IUPUI

Surveys 1. Enrolled Students  Our own  NSSE 2. Graduates 3. Employers 4. Stop outs 5. Faculty 6. Staff © TWBANTA-IUPUI

Information Gateway Information about  Students  Faculty  Staff  Alumni  Finances © TWBANTA-IUPUI

Since 1993 Campus-wide surveys have stimulated changes in  Curricula  Advising  Increased writing practice  Increased attention to first-year experiences  Placement of graduates © TWBANTA-IUPUI

What is ABC? ABC is a costing methodology based upon the fact that different activities and products consume different proportions of resources Resources Product A Product B Product C Activities

Activities performed to produce a graduate © TWBANTA-IUPUI

Some Activities within Orientation 1. Planning the program 2. Inviting students and parents 3. Purchasing/duplicating materials 4. Obtaining spaces and equipment 5. Setting up spaces and equipment 6. Providing the program 7. Evaluating the program 8. Cleaning space and returning equipment © TWBANTA-IUPUI

What Is ABC? Traditional vs. ABC Traditional Accounting Perspective n Salary & wages 1,350,000 n Benefits 495,000 n Travel 45,000 n Facilities 220,000 n Supplies 90,000 Total $2,200,000 Activity-Based Perspective n Planning 940,000 n Materials 430,000 n Spaces & equipment 250,000 n Program 350,000 n Evaluation 230,000 ________________________ Total $2,200,000 © TWBANTA-IUPUI

Some Applications of Economic Model 1. Estimate costs of administrative services as compared to cost of outsourcing 2. Determine fees for various programs 3. Restructure processes to expedite work flow and minimize costs © TWBANTA-IUPUI

Since 1992 Activity-based Costing has stimulated changes in n Planning n Budgeting n Assessment © TWBANTA-IUPUI

Elements of Program Review  Self Study  Review by Respected Peers  Recommendations  Follow-up © TWBANTA-IUPUI

Goals of Program Review at IUPUI  To improve student learning  To assess and improve program quality  To increase cross-disciplinary collaboration  To enhance community connections  To reinforce importance of aligning unit and campus planning © TWBANTA-IUPUI

Following the Review 1. Department reviews the recommendations 2. Department prepares written response 3. Department presents response in administrative hearing 4. Responsible administrators agree upon responsive action(s) 5. Responsive actions are implemented 6. Re-review ensures action © TWBANTA-IUPUI

Since 1995 Program Reviews have stimulated changes in  Planning for the future  Research emphases  Faculty hiring priorities  Advisory councils  Cross-disciplinary collaboration © TWBANTA-IUPUI

Program Review at IUPUI © TWBANTA-IUPUI

Goal and Objectives for Student Learning  Enhance undergraduate student learning and success 1. Strengthen generic skills 2. Provide honors programming 3. Offer learning communities 4. Strengthen advising 5. Provide tutoring and mentoring © TWBANTA-IUPUI

Employ Multiple Methods 1) Direct  Projects, papers, tests, observations 2) Indirect  Questionnaires, interviews, focus groups  Unobtrusive measures Syllabi, transcripts © TWBANTA-IUPUI

In a Comprehensive Assessment Program... INVOLVE t Students t Faculty t Student Affairs Staff t Administrators t Graduates t Employers © TWBANTA-IUPUI

Since 1994 Assessment of Learning has stimulated changes in  Student support programs  Curriculum  Methods of instruction  Internships  Methods of assessment © TWBANTA-IUPUI

Establishing a Culture of Evidence takes  Strong leadership  Support  Time  Evidence that findings are used to guide improvements © TWBANTA-IUPUI

Open sharing of information and evidence-based decision-making  Financial and satisfaction data for units  Annual planning/budgeting hearings  Performance indicators derived from unit reports over time  Campus performance report for community © TWBANTA-IUPUI

Program Review & Assessment Committee Ÿ 2 reps from each school Ÿ 2 librarians Ÿ Other units  Student Life  Faculty Development  Internship coordinator © TWBANTA-IUPUI

Characterizing the Culture New initiatives require assessment  University College student support programs  Distance learning  New academic programs © TWBANTA-IUPUI

Characterizing the Culture  Appointment of Assessment Specialists  University College  Student Life  Service Learning  Library  Enrollment Services  Appointment of Associate Deans for Assessment © TWBANTA-IUPUI

Group Assessment Has Failed to Demonstrate Institutional Accountability Focus on improvement at unit level Rare aggregation of data centrally Too few faculty involved HE scholars focused on K-12 assessment

© TWBANTA-IUPUI Now We Have the Press to Assess with a Test

© TWBANTA-IUPUI 2006 Commission on the Future of Higher Education  We need a simple way to compare institutions  The results of student learning assessment, including value added measurements (showing skill improvement over time) should be... reported in the aggregate publicly.

© TWBANTA-IUPUI Voluntary System of Accountability Report Scores in critical thinking, written communication, analytic reasoning using CAAP MAPP CLA

© TWBANTA-IUPUI At the University of Tennessee CAAP Academic Profile (now MAPP) COMP (like CLA and withdrawn by 1990) College BASE

© TWBANTA-IUPUI In TN We Learned 1) No test measured 30% of gen ed skills 2) Tests of generic skills measure primarily prior learning 3) Reliability of value added =.1 4) Test scores give few clues to guide improvement actions

© TWBANTA-IUPUI An Inconvenient Truth.9 = the correlation between SAT and CLA scores of institutions thus 81% of the variance in institutions’ scores is due to prior learning

© TWBANTA-IUPUI How Much of the Variance in Senior Scores is Due to College Impact? Student motivation to attend that institution (mission differences) Student mix based on age, gender socioeconomic status race/ethnicity transfer status college major

© TWBANTA-IUPUI How Much of the Variance in Senior Scores is Due to College Impact? (continued) Student motivation to do well Sampling error Measurement error Test anxiety  College effects ______ 19 %

Better Ways to Demonstrate Accountability 1. Performance Indicators  Access, social mobility  Diversity  Workforce development  Economic development  Engaging student experience © TWBANTA-IUPUI

Better Ways to Demonstrate Accountability 2. Measures of Learning  Standardized tests in major fields  Internship performance  Senior projects  Electronic portfolios  External examiners © TWBANTA-IUPUI

Student Electronic Portfolio Students take responsibility for demonstrating core skills Unique individual skills and achievements can be emphasized Multi-media opportunities extend possibilities Metacognitive thinking is enhanced through reflection on contents - Sharon J. Hamilton IUPUI

Student Development  Involve Student Affairs staff in planning for assessment  Set expectations for student development s Recreation s Leadership development s Mental and physical health s Spiritual development  Monitor student progress  Take warranted improvement actions © TWBANTA-IUPUI

Questions Guiding Assessment of Student Services 1. How many students know about the program? 2. How many students say they need the program? 3. How many students use the program? 4. Do students persist in the program? 5. What knowledge, skills, attitudes, values do students develop in the program? 6. How satisfied are students with the program? 7. Do program outcomes justify its cost? 8. Can the program be improved? 9. Should the program be terminated? © TWBANTA-IUPUI

Taxonomy of Educational Objectives (Bloom and Others, 1956) Cognitive domain categories Knowledge Comprehension Application Analysis Synthesis Evaluation Sample verbs for outcomes Identifies, defines, describes Explains, summarizes, classifies Demonstrates, computes, solves Differentiates, diagrams, estimates Creates, formulates, revises Criticizes, compares, concludes

© TWBANTA-IUPUI Planning for Learning and Assessment

Student Services Activity Records # participating in activities over the years # changing majors # meals served in Union Average age of students in residence halls # voting in campus elections # employers coming to campus to interview # visits to health clinic © TWBANTA-IUPUI

Student Affairs Learning Outcomes Leadership skills Self confidence Communication skills Time management skills Ability to manage stress Teamwork skills Overall health © TWBANTA-IUPUI

Measures of Student Learning  Questionnaires  Interviews  Focus groups  Observations  Transcript analysis © TWBANTA-IUPUI

Primary Trait Scoring Assigns scores to attributes (traits) of a task STEPS  Identify traits necessary for success in assignment  Compose scale or rubric giving clear definition to each point  Grade using the rubric © TWBANTA-IUPUI

Primary Trait Scoring ~ E xample ~ Trait Desired:Self expression of a feeling evoked by an event Scale: 1 - No real expression presented 2 - Feeling expressed but in- adequately described 3 - Expression generally competent 4 - Feeling expressed in great detail 5 - Detail plus intelligent response to feeling evoked © TWBANTA-IUPUI

Observation of Student Behavior (including participant observers like RAs) Check-list of behaviors for Actual setting Videotape of advising sessions Role-playing Simulation of incidents © TWBANTA-IUPUI

Faculty and Staff Development  Focus faculty and student affairs professionals on improving learning in and outside class  Attend conferences together  Study literature on student learning  Provide workshops on teaching and learning  Provide resources (e.g., grants, summer salary, release time) © TWBANTA-IUPUI

Virginia Commonwealth University 1. First-year English students wrote 1 response / week for 15 weeks 2. Faculty-student affairs teams read essays 3. Sample of writers interviewed 18 months later Responses to findings: Central advising center, new advising handbook, multicultural workshops © TWBANTA-IUPUI

Chicago State University Purpose: Determine reasons students leave TELEPHONE INTERVIEW Finding: No personal contact for 78% Response: Freshman seminar revised, early warning system, referral to mandatory learning strategies course © TWBANTA-IUPUI

Santa Barbara City College  Problem: Need to improve retention and satisfaction with goal achievement  COMMUNITY COLLEGE STUDENT EXPERIENCES QUESTIONNAIRE  Findings: Strong relationships between involvement and progress toward goals and satisfaction with instruction  Response: More extra-curricular activities, departmental clubs, space for group study © TWBANTA-IUPUI

Designing Effective Assessment: Principles & Profiles of Good Practice Trudy W. Banta Elizabeth A. Jones Karen E. Black Jossey-Bass (Wiley) 2009 © TWBANTA-IUPUI

Profiles  Invited over 1000  Received 146  Selected 49 for use in full  Categorized all 146 and published Web sites © TWBANTA-IUPUI

Outline for Profiles  Background and Purpose  Methods over ? Years  Resources Required  Findings  Use of Findings  Impact of Using Findings  Success Factors  Web sites © TWBANTA-IUPUI

Plan Implement Evaluate Improve Culture of Evidence

~ Organization ~ of Principles & Profiles  Planning  Implementing  Improving & Sustaining - Building a Scholarship of Assessment Banta & Associates Jossey-Bass 2002 © TWBANTA-IUPUI

Planning Principles 1. Engaging stakeholders 2. Connecting assessment to valued goals & processes 3. Creating a written plan 4. Timing assessment 5. Building a culture based on evidence © TWBANTA-IUPUI

Implementation Principles 1. Providing leadership 2. Creating faculty/staff development 3. Assessing processes as well as outcomes 4. Communicating and using findings © TWBANTA-IUPUI

Improving/Sustaining Principles 1. Providing credible evidence of learning to multiple stakeholders 2. Reviewing assessment reports 3. Ensuring use of results 4. Evaluating the assessment process © TWBANTA-IUPUI

Cal State-Sacramento (1) Sources of Motivation for Assessment 1. New VP for Student Affairs 2. Reaccreditation looming 3. Enrollment & budget challenges 4. Pledge to become more data-driven and focused on student learning © TWBANTA-IUPUI

Cal State-Sacramento (2) 1. Align department & division missions 2. Develop SMART goals, 1 for student learning  Specific  Measurable  Aggressive, yet attainable  Results-oriented  Timely © TWBANTA-IUPUI

Cal State-Sacramento (3) Measures  Pre-post MC tests on policies, resources  Essays with rubrics (reinstatement)  Portfolios  Observation of skills (Leadership, RA reports on scenarios, role-playing) © TWBANTA-IUPUI

Cal State-Sacramento (4) Findings 1. Some SLOs met 2. Some SLOs not met 3. Some measures not effective 4. Too few participants to assess 5. Too many participants to assess effectively © TWBANTA-IUPUI

Cal State-Sacramento (5) Use of Findings 1. Better training for RAs in reporting 2. Better training for peer mentors in orientation (emphasizing policies) 3. More time to discuss films 4. Better Powerpoint presentations 5. Increase participation in counseling 6. Redesign vague test items © TWBANTA-IUPUI

IUPUI (1) Themed Learning Communities (TLCs)  3 first-year courses for a cohort  Themes such as Crime in America, Health & Wellness, African-American perspectives  Planning group involving faculty, student affairs, advisor, peer mentor © TWBANTA-IUPUI

IUPUI (2) TLCs & Control Group ~ MEASURES ~ 1. GPA 2. Persistence for 1 year 3. Common assignments 4. Student survey responses © TWBANTA-IUPUI

IUPUI (3) TLC Benefits 1. Higher GPAs 2. Higher persistence rates 3. See more connections across disciplines 4. Encounter more diverse perspectives 5. Experience more teamwork © TWBANTA-IUPUI

IUPUI TLCs (4) Use of Assessment Findings  More detailed collaborative planning  Improved common assignments  More cocurricular activities  More faculty from more disciplines  More TLCs! © TWBANTA-IUPUI

An Excellent Resource – Assessment Practice in Student Affairs by John H. Schuh & M. Lee Upcraft Jossey-Bass 2001 © TWBANTA-IUPUI

Assessment Institute in Indianapolis October 25-27, 2009 Westin Indianapolis © TWBANTA-IUPUI