1. Beginner’s Guide to Assessment
NACADA 2017
St. Louis, MO
Matt Dunbar & Theresa Duggar
Introduce speakers

2. Learning Outcomes Recognize different types of assessment
Identify common techniques for assessing advising programs
Judge the applicability of assessment methods
Develop questions to further knowledge of assessment
Demonstrating effectiveness is crucial for advising offices as we seek to serve our students better and meet the accountability calls from stakeholders at all levels. In this session, participants will learn basic assessment types, lenses, and applications. Participants will also experience parts of the assessment process during the program.

3. Outline Introductions Defining “Assessment” Why Assess?
The Assessment Cycle
Types of Assessment
Lenses of Assessment
Q/A
A summative, value-added assessment
The presentation will focus on different types and methods of assessment and the different lenses assessment personnel utilize. The goal of each section is to provide participants with enough knowledge to ask informed questions and seek out more in-depth learning, and are not intended to be comprehensive.

4. Who’s in the room
Name, institution, comfort/familiarity with assessment?
Ask attendees their familiarity or comfort level with assessment to gauge the direction of the conversation.

5. Defining Assessment
Any effort to gather, analyze, and interpret evidence which describes program effectiveness Upcraft & Schuh (1996)
An ongoing process aimed at understanding and improving _______ Angelo (1995)
To ground our discussion in literature, we will primarily utilize Upcraft and Schuh’s (1996) work from Assessment in Student Affairs. Their work provides approachable, straightforward language for those who have not done assessment previously. Likewise, Upcraft and Schuh’s book provides a springboard for further discussions of assessment. In this presentation, Upcraft and Schuh’s definitions of terms will inform a majority of the discussion and definitions used.
“. . . a lack of assessment data can sometimes lead to policies and practices based on intuition, prejudice, preconceived notions, or personal proclivities - note of them desirable bases for making decisions.” (Upcraft and Schuh, 2002, p. 20)

6. Why Assess? Why should you? Why do you? Why don’t you?
Advisors are a part of the University process! The goal of assessment is for development (improvement, to get better, to better meet students’ needs, etc.) and data provides us the information to do that!
Literature points to the importance of advising on student success, thriving, persistence, graduation, etc.
Reasons for assessment:
To evaluate attainment of goals (effectiveness)
To engage faculty, students, administration and staff in continuous, systematic cycles of question asking, feedback and refinement (improvement)
To generate information for stakeholders (accountability)
To serve the university’s mission (student success: RPG)
To enhance advisement, student learning, and program design

7. The Assessment Cycle

8. Types of Assessment Qualitative Quantitative Direct Indirect Formative
Summative

9. Qualitative Dealing with quality, value, words, ideas
“What did you learn at the advisement session?”
“How do you feel about the advisement session?”
Some methods include:
Interviews
Observations
Focus groups
Open-ended questionnaires
Documents

10. Quantitative Dealing with numbers, statistics, counts
“How strongly do you (dis)agree with the following statement?”
“Rate the following skills from 1-10.”
Some methods include:
Experiments
Closed questions on a questionnaire (rating scale)
National Survey of Student Engagement (NSSE)
CORE Institute Alcohol and Drug Survey

11. Questions to pose to attendees - have them guess qualitative or quantitative:
What kind of emotions and attitudes motivate individuals to participate in advisement? (qualitative)
How often do students participate in advisement? (quantitative)
How regularly do students go home for holiday breaks? (quantitative)
How do juniors living in housing perceive their situation and how are they dealing with it? (qualitative)

12. Direct Measures in specific ways and demonstrates knowledge
“How many guests may you sign into the residence hall at one time?”
“Where is Campus Life located?”
Examples:
Direct observation of advising interactions
Pretest/posttest
Standardized tests of students’ learning
Counts of use of advising appointments
Advisor: student ratios
Portfolios
Performance appraisals
Advantages:
Require students to demonstrate knowledge
Provide data that directly measure achievement of expected outcomes

13. Indirect
Captures reflections on learning, the feelings about the thing
“How do you plan to use today’s advisement in your daily life?”
“What emotions describe how you feel after today’s session?”
Examples:
Focus groups
Interviews
Surveys/questionnaires
Tracking students’ perceptions (ratings of advisors, ratings of service, advisor satisfaction)
Tracking advisors’ perceptions (student preparedness, estimation of student learning)
Archival data
Final grades (because they include corrections not related to learning outcomes like extra credit or penalties for unexcused absences)
Advantages:
Ask students to reflect on their learning
Provide clues about what could be assessed directly
Easy to administer
Particularly useful for ascertaining values and beliefs

14. Questions to pose to attendees - have them guess direct or indirect:
Data was gathered using pretests and posttests at new student orientation. (direct)
Data was gathered from students’ journals of their advisement experiences. (direct)
Data was gathered from an exit survey. (indirect)
Data was gathered from graduation and retention rates. (indirect)

15. Formative Gathering information in order to shape our future work
“How comfortable are you with the student organization registration system?”
“Do you currently work on-campus?”
Formative assessments:
Help students identify their strengths and weaknesses and target areas that need work
Help advisors recognize where students are struggling and address problems immediately
Help advisors assess their own progress
Types of formative assessment:
Observations
Homework
Reflections journals
Advising appointments
Student feedback

16. Summative Following-up, seeing where participants have ended-up
Types of summative assessment:
Examinations
Portfolios
Student evaluation of advisement
Advisor self-evaluation

17. Questions to pose to attendees - have them guess formative or summative:
Advising appointments. (formative)
Final exam in a senior seminar course. (summative)
Asking students to submit one to two sentences identifying the main point of the advisement meeting. (formative)
Advising portfolios. (summative)
Advising portfolio: advising philosophy, advising goals/objectives, advisee demographics, advising responsibilities, evidence of growth, and essay about the artifacts.

18. Assessment Lenses Multiple Lenses of Assessment
Criterion: Comparing to a goal or set of requirements (criteria)
Benchmarks: Comparing to peers, standards, or others.
Longitudinal: Comparing to past selves
Value-added: Comparing to where we were before the experience

19. Assessment Lenses Hypothetical Scenario:
66% of students register for courses on time on their registration day.

20. Criterion Key Concept: How did we do against a predetermined standard?
In Action:
The institution set a goal that every student (100%) will register on time during registration day.

21. Benchmark Key Concept: How do we compare to our peers? In Action:
Nationwide, institutions report that 80% of students register for courses on time.

22. Longitudinal Key Concept: Have we improved or declined? In Action:
On time registration has improved by 15% over the last three years.
Reason for change? Did we actively impact this increase?

23. Value-Added Key Concept:
Think about outcomes you want to achieve. Is the targeted population better off?
In Action:
More students are registering on-time.
→ What benefits does this entail?
Advisors are effective; departments can prepare future faculty and semester schedules; institutions can report higher and more accurate enrollment numbers; etc.

24. Summary Assessment is Important Types of Assessment Lens of Assessment
Why?
Types of Assessment
What are you comfortable with?
What type of data fulfills your needs?
Lens of Assessment
Important to apply context to data

25. Activity Start Your Assessment Process! University Mission
Advising Goals
Center Mission
Center Goals
Student Learning Outcomes
Assessment Plan

26. Questions?