Beyond satisfaction, more than a survey: Assessment to practice Heather Ammons and Danielle Clarke NACADA Annual 2016 – Atlanta, Georgia

What our students are saying (part I) How satisfied are you with the quality of your advisor? Year Strongly Disagree Disagree Neutral Agree Strongly Agree 2014-2015 3% 1% 2% 12% 82% 2015-2016 4% 5% 14% 75% How likely are you to recommend your advisor to another student? Year Very Unlikely Unlikely Undecided Likely Very Likely 2014-2015 4% 1% 12% 79% 2015-2016 5% 3% 13% 75%

What our Students are saying (Part II) During advising appointment, I felt comfortable talking with my advisor. Year Strongly Disagree Disagree Neutral Agree Strongly Agree 2014-2015 2% 15% 79% 2015-2016 5% 3% 14% 76% So we delved deeper and could have just left it here, but…

What our Students are Saying (Part II) What would have made you feel more comfortable talking with this advisor? If the advisor made any attempt to show enthusiasm about her job or say anything other than "yeah", "mhmm", or "nah" then I would have felt a little more at ease knowing an advisor cared about my appointment. It seemed as if he was having a long day, but I do not see what his attitude of the day had to do with my advising. Her actually caring about my session. She barely spoke to me and did not make the meeting personal at all. I did not feel confident about my academic situation going into course selection after this meeting and I had to schedule another with someone else in order to know anything about my class options. It being individualized. she didn't even ask my name or major. She was just checking me off of a list. If he had not been rude, and acted like he genuinely cared. This question was only asked to those students that indicated they felt uncomfortable

What our Students are saying (Part II) Did you tell your advisor about factors (personal, financial, academic, etc.) that influence your success at UA? Year Yes No 2015-2016 64% 34% Again, not too bad. Our students like us, they tell us things.

What our students are saying (Part II) In what ways did your advisor address these factors with you? He gave me a direct vision of where I am and where I need to be. In listening, understanding, and planning accordingly My advisor took my individual career goals and current situation into account and made me an updated, personal plan. My advisor was made aware of my major time commitments outside of the classroom, as well as some personal issues I am dealing with. He then advised me on how to create a schedule that was manageable and beneficial to me with those other commitments in mind. What would have made you more likely to discuss these factors? If he asked. At one point he suggested I take a full course load over the summer, which is not financially feasible. I did not feel comfortable enough to express my concerns with that. I didn't feel like I needed to talk about those factors. If advisor had asked If there was a reason to discuss them. Our conversations did not pertain to ability to attend, it was more focused on ensuring all of my classes were accounted for in the semesters they were offered since I have added a second major. There was no opportunity to bring any of these factors up, as we only discussed class options for next semester. Reading over this it’s easy to maybe toss the comments out as maybe an outlier student that is disappointed or a one off comment. Or maybe just an opportunity to work with advisors on how we engage with students. We took it as a view that shows the differing perspectives of advising and that we were missing out on a couple of our outcomes. First column – for those that answered yes. Second column – for those that answered no

Why satisfaction is not enough

What do you know about your office? Students are happy with our services! They like us! They think we are nice and easy to talk to! BUT….. They can be happy and never know that they aren’t growing (problematic) Advising is teaching… What do you know about your office? This is not to say that satisfaction is not of value, because that is not true, but instead to think through exactly what that is telling you about your office.

IT all goes back to the foundation What is the purpose of your office? Are there clearly established goals and outcomes? What are you current practices? What information do you currently track about your students? Why? Are you actually doing what you say you are supposed to do? When was the last time you asked your students about their advising encounters? Essentially, what is the defined vision, mission, goals, and outcomes for your office? This is where we started – at the foundation. Danielle will talk, briefly, about what we did a few summers ago. But really why does a focus on this matter?

Value of multiple measures How do you know if you are meeting your mission, vision, goals, and outcomes? You assess it, but there are so many ways you can do that. Let’s go through different types and why they are of value.

Types of measures Direct Indirect Method of gathering information that asks student/advisor to reflect on what has been learned rather than to demonstrate it. Survey Focus Groups Interviews Document Analysis Direct Indirect Method of gathering information that allows student/advisor to demonstrate their knowledge of skills. Appt #’s Degree plans Demographics GPA Progression to degree

Why use multiple measures? Cannot access everything, but you can access a lot Build powerful cases by pairing/contrasting Tailor your story to the audience Perception v. reality Options to explore: Focus Groups Interviews Mining what you already have Self-assessment

Approaching assessment

Our Assessment Plan Vision, Mission, Shared Values, Goals, and Outcomes reviewed every 5 years Outcomes divided over five years, so we are not trying to assess everything at once Each year we develop instruments and other measure that specifically address each of our emphasized outcomes Yearly review of assessment outcomes with entire office Yearly annual report that summarizes findings (Task Stream) Self study at end of five years using CAS

What measures do we use? Direct Indirect Appt #s and reason codes Businessatbama emails DegreeWorks plans # Who created Course registration audit Surveys Student and Advisor outcomes Includes missed appointments, change of division Focus Groups

Assessment to Practice Assessment included in annual reviews # of appointments Survey and focus group feedback Opportunities to address needs Annual Report What are we accomplishing The good, the not so good, and the bad Make it the entire office’s prerogative to define and work at areas of focus Altered development and training It’s hard to own up that it’s not the policies, it’s the conversations How have our practices changed?

Making sense of it all

There is a lot to this Assessment is a work in progress Cannot measure everything at once Constantly review! Continuously thinking of why you are doing this It’s about change, growth, development Every item, every step, every word in assessment relates back to something that the office values Unique to every office Never done, never static. Sheet with outcome here. Show them one of our outcomes and how we assess it.

Let’s get to brainstorming! Assessment in progress Just beginning Do you have a vision and mission? Yes: Is it part of your assessment? No: Tease it out. Do you have a vision and mission? Yes: Great! How can you use it in assessment? No: Tease it out. Now that we’ve discussed why move beyond satisfaction and what we do in our office – how about you start thinking about your office. Let’s pass out the handout here.

How to contact us Heather Ammons Danielle Clarke hammons@ culverhouse.ua.edu 205-348-4537 dclarke@ culverhouse.ua.edu 205-348-4537