Look! Our ‘Stuff’ is Making an Impact: Using Meaningful Assessment to Demonstrate the Effectiveness of Academic Support Services Michael Anne Greer Director, Academic Success Center Amanda Novak Director, Advising and New Student Services Texas Wesleyan University

The Same Old Question: Why Assess? We have to We have to Accreditation Accreditation Reaffirm our roles Reaffirm our roles Demonstrate department’s contributions to institutional goals Demonstrate department’s contributions to institutional goals SEE IF OUR STUFF IS WORKING SEE IF OUR STUFF IS WORKING

Who Should Be Assessing? Academic Departments Academic Departments Experience creating, writing, and evaluating learning outcomesExperience creating, writing, and evaluating learning outcomes Administrative Departments Administrative Departments Why it can be dauntingWhy it can be daunting Which areas assess?Which areas assess? Internal, within the department Internal, within the department Centralized Institutional Research Centralized Institutional Research

Let’s Look at Our ‘Stuff’ Texas Wesleyan- brief statistics Texas Wesleyan- brief statistics 3,204 students3,204 students 1,794 undergraduates1,794 undergraduates PrivatePrivate Academic Success Center Academic Success Center HistoryHistory ServicesServices RolesRoles Advising & New Student Services Advising & New Student Services HistoryHistory ServicesServices RolesRoles

Common Pitfalls Avoiding assessment Avoiding assessment It’s not that bad…after 1 or 2 cycles, much less daunting!It’s not that bad…after 1 or 2 cycles, much less daunting! Don’t be afraid of the dataDon’t be afraid of the data Much easier and more useful if kept up regularlyMuch easier and more useful if kept up regularly Staying on track Staying on track Evaluation GridEvaluation GridEvaluation GridEvaluation Grid Advising Assessment MatricesAdvising Assessment MatricesAdvising Assessment MatricesAdvising Assessment Matrices (Redmond & Babenchuk, 2011) (Robbins, 2011)

Common Pitfalls Alignment with University Alignment with University Crucial to demonstrate how your department is contributing to institutional goals and missionCrucial to demonstrate how your department is contributing to institutional goals and mission Demonstrate needs (funding, staff, resources)Demonstrate needs (funding, staff, resources) Informed decisions for improvementInformed decisions for improvement Alignment with department mission Alignment with department mission Make sure goals support current missionMake sure goals support current mission (Bresciani, 2011)

Common Pitfalls Alignment with University Alignment with University “Our mission at Texas Wesleyan University is to develop students to their full potential as individuals and as members of the world community.”“Our mission at Texas Wesleyan University is to develop students to their full potential as individuals and as members of the world community.” “The Academic Success Center’s mission is to serve as a critical link between students and a fully successful academic experience by providing personal support services that remove barriers to academic excellence and assist in the development of critical thinking, analytical reasoning, and problem solving skills.”“The Academic Success Center’s mission is to serve as a critical link between students and a fully successful academic experience by providing personal support services that remove barriers to academic excellence and assist in the development of critical thinking, analytical reasoning, and problem solving skills.” “Academic advising at Texas Wesleyan University is a teaching and learning process dedicated to student success. Consistent with the university mission, academic advising engages students in developing a plan to realize their educational, career and life goals.”“Academic advising at Texas Wesleyan University is a teaching and learning process dedicated to student success. Consistent with the university mission, academic advising engages students in developing a plan to realize their educational, career and life goals.” (Bresciani, 2011)

Common Pitfalls Unreasonable goals and objectives Unreasonable goals and objectives Keep goals manageable and attainableKeep goals manageable and attainable Be sure goals are relevant and meaningful to stakeholdersBe sure goals are relevant and meaningful to stakeholders Goals should be broad and reflect what your program would like to be, look like, doGoals should be broad and reflect what your program would like to be, look like, do (Bresciani, 2011)

Common Pitfalls Ineffective measures Ineffective measures Are we measuring actual outcomes?Are we measuring actual outcomes? Tracking visits does not measure outcomes.Tracking visits does not measure outcomes. Lack of documentation/ tracking Lack of documentation/ tracking Are we asking for/ collecting/ pulling the data we need at the right time?Are we asking for/ collecting/ pulling the data we need at the right time? We’ve been doing all of these great things, but have no evidence.We’ve been doing all of these great things, but have no evidence. One survey is not assessment! One survey is not assessment!

Common Pitfalls Invalid data Invalid data Does the data give us a valid measure of our outcomes?Does the data give us a valid measure of our outcomes? Incorrect interpretation of data Incorrect interpretation of data Increased GPA does not imply retentionIncreased GPA does not imply retention Ineffective use of data- not closing the loop! Ineffective use of data- not closing the loop!

‘Backwards Thinking’ Some struggle with how to think about assessment Some struggle with how to think about assessment Let’s think backwards! Let’s think backwards! What are we doing? What are we doing? 85 students come in for writing tutoring85 students come in for writing tutoring Held 12 academic skills workshopsHeld 12 academic skills workshops 150 out of 200 freshmen register on the first day of early registration150 out of 200 freshmen register on the first day of early registration Write 2 or 3 of your servicesWrite 2 or 3 of your services

‘Backwards Thinking’ Why are we doing it? Why are we doing it? “Students need/ request it”- not specific enough“Students need/ request it”- not specific enough Need to demonstrate why spending the time, effort, funds to support servicesNeed to demonstrate why spending the time, effort, funds to support services

Outcomes What are we attempting to achieve? What are we attempting to achieve? Outcomes- what the end user should know, have, be able to do as the result of the serviceOutcomes- what the end user should know, have, be able to do as the result of the service Must be specific Must be specific Must be measurable Must be measurable (Bussell, 2011)

Writing Outcomes “80% of undergraduates will utilize tutoring services”- not an outcome “80% of undergraduates will utilize tutoring services”- not an outcome “Students who utilize tutoring services will succeed”- not specific enough “Students who utilize tutoring services will succeed”- not specific enough “Students will benefit from workshops”- not measurable “Students will benefit from workshops”- not measurable Students will receive adequate advising – not easily measured, too broad Students will receive adequate advising – not easily measured, too broad

Writing Outcomes “Students who receive writing assistance will pass writing intensive courses with a C or better” “Students who receive writing assistance will pass writing intensive courses with a C or better” “Students who attend academic workshops will gain effective note-taking methods” “Students who attend academic workshops will gain effective note-taking methods” “Freshman Students will be able to calculate their GPA” “Freshman Students will be able to calculate their GPA” Write 2-3 service outcomes Write 2-3 service outcomes

Measuring Outcomes Is it working? Is it working? How do we know? How do we know? Measures- should assess actual outcome, not another processMeasures- should assess actual outcome, not another process Direct Measures- assess actual performance or product Direct Measures- assess actual performance or product Indirect Measures- measure perceptions or opinions Indirect Measures- measure perceptions or opinions

Measures Direct or Indirect? Direct or Indirect? Practice test at end of math tutoring sessionPractice test at end of math tutoring session Survey to workshop attendeesSurvey to workshop attendees Focus group opinions of tutorsFocus group opinions of tutors Demonstration exercise during workshopDemonstration exercise during workshop Enrollment data during early registrationEnrollment data during early registration Write 1 direct and 1 indirect for your outcomes Write 1 direct and 1 indirect for your outcomes

Gathering the Data What do the measures tell us? What do the measures tell us? Data Collection ExamplesData Collection Examples Survey responses Survey responses Focus group feedback Focus group feedback Retention reports Retention reports Grade reports Grade reports GPA GPA Self Study Audits Self Study Audits Individual observation of students performing a task (especially useful in advising) Individual observation of students performing a task (especially useful in advising)

Assessing the Data Are we doing well? Are we doing well? Did we meet our goals?Did we meet our goals? Do we need to increase/ decrease goals?Do we need to increase/ decrease goals? What are appropriate benchmarks?What are appropriate benchmarks? CAS Standards CAS Standards Professional association standards Professional association standards Peer Institutions Peer Institutions

Using the Data Now what? Now what? Closing the loopClosing the loop What goals were met/ not met? What goals were met/ not met? Why or why not? Why or why not? Needs- resources, staff, fundingNeeds- resources, staff, funding How to improve How to improve Affirmations Affirmations Recommendations/ plans going forward Recommendations/ plans going forward

References Bresciani, M.J. (2011, June). Enhancing outcomes-based assessment for student affairs. Webinar. Lecture conducted from Innovative Educators, Thornton, CO. Bresciani, M.J. (2011, June). Enhancing outcomes-based assessment for student affairs. Webinar. Lecture conducted from Innovative Educators, Thornton, CO. Bussell, K.H. (2011, May). Strategic planning & outcomes assessment. Academic Student Support Services Council retreat. Lecture conducted from Texas Wesleyan University, Fort Worth, TX. Bussell, K.H. (2011, May). Strategic planning & outcomes assessment. Academic Student Support Services Council retreat. Lecture conducted from Texas Wesleyan University, Fort Worth, TX. Redmond, J. & Babenchuk, I. (2011, September). Data-driven decision-making. Learning centers: At the crossroads of student success. Lecture conducted from National College Learning Center Association 26 th Annual Conference, Indianapolis, IN. Redmond, J. & Babenchuk, I. (2011, September). Data-driven decision-making. Learning centers: At the crossroads of student success. Lecture conducted from National College Learning Center Association 26 th Annual Conference, Indianapolis, IN.