The Global Community for Academic Advising

The Global Community for Academic Advising
NACADA Executive Office Kansas State University 2323 Anderson Ave, Suite 225 Manhattan, KS Phone: (785) Fax: (785) © National Academic Advising Association The contents of all material in this presentation are copyrighted by the National Academic Advising Association, unless otherwise indicated. Copyright is not claimed as to any part of an original work prepared by a U.S. or state government officer or employee as part of that person's official duties. All rights are reserved by NACADA, and content may not be reproduced, downloaded, disseminated, published, or transferred in any form or by any means, except with the prior written permission of NACADA, or as indicated below. Members of NACADA may download pages or other content for their own use, consistent with the mission and purpose of NACADA. However, no part of such content may be otherwise or subsequently be reproduced, downloaded, disseminated, published, or transferred, in any form or by any means, except with the prior written permission of, and with express attribution to NACADA. Copyright infringement is a violation of federal law and is subject to criminal and civil penalties. NACADA and National Academic Advising Association are service marks of the National Academic Advising Association. Identifying and Using Multiple Measures Rich Robbins, Ph.D. Bucknell University Sharon Aiken-Wisniewski, Ph.D. University of Utah

Participant Learning Outcomes As a result of attending Plenary 4, participants will: Cognitive Understand the importance of using multiple measurements for any single desired Student Learning Outcome (SLO) Understand why one single measure for any desired SLO may not result in valid data Understand the importance of using multiple measurements for any single desired Process/Delivery (P/D) outcome Understand why one single measure for any desired P/D outcome may not result in valid data Understand the issues regarding relying on student perception surveys on which to base decisions Behavioral Identify multiple measures for at least one of their identified SLOs Complete the measurement columns of the assessment matrix for at least one of their SLOs Identify multiple measures for at least one of their identified P/D outcomes Complete the assessment matrix for at least one of their identified P/D outcomes Affective Appreciate the need for multiple outcome measures for a desired SLO Appreciate the need for multiple outcome measures for a desired P/D outcome Value the need for valid data to determine whether a desired outcome has been met

Initial Considerations Once your Process/Delivery Outcomes and Student Learning Outcomes have been identified, as well as when and where they will occur (mapping), the next step is to determine who or what will be measured and how the data will be gathered… …using multiple measures of varying types

Initial Considerations continued… True assessment of academic advising should involve multiple measures - beyond a simple satisfaction survey or advisor evaluation - individual measures evaluate (rather than assess) a single dimension of a single phenomenon

Initial Considerations continued… Need multiple measures to measure multiple dimensions of a single phenomenon - multiple measures for each process/delivery outcome and multiple measures for each student learning outcome

Initial Considerations continued… The multiple measures utilized should vary in type Qualitative Quantitative Direct Indirect This will help to insure the validity of the results of any single measure

Types of Measures Qualitative - exploratory - small samples - open-ended - emerging information subjective, inductive interpretation of data examples - focus groups - case studies - naturalistic observation Information/data in form of rich, in-depth responses (words)

Types of Measures continued… Quantitative - descriptive - large samples - structured objective, deductive interpretation of data examples - questionnaires - surveys - experiments Information/data in form of numbers, measures (statistics)

Types of Measures continued… Direct - may be qualitative or quantitative examples - direct observation of advising interaction - pre-test/post-test of variable leading to desired outcome - standardized test or inventory measuring student learning - tracking of student data (enrollment rates, retention rates, GPAs, transcript analysis, etc.) - counts of use of services - advisor:student ratios

Types of Measures continued… Indirect - may be qualitative or quantitative examples - focus groups - surveys, questionnaires - interviews - reports - tracking of student perceptions (satisfaction, ratings of advisors, ratings of service, etc.) - tracking of advisor perceptions (student preparedness, estimation of student learning, etc.)

Examples of Existing Instruments To be used as just one measure among multiple measures ACT Survey of Academic Advising Noel-Levitz Student Satisfaction Inventory (SSI) Winston and Sandor’s Academic Advising Inventory (AAI) NACADA Assessment of Advising Commission

Other Sources To be used as just one measure among multiple measures Data from National Survey of Student Engagement (NSSE), Beginning College Survey of Student Engagement (BCSSE), Faculty Survey of Student Engagement (FSSE) Data from other nationally normed, standardized instruments (e.g., ACT, Noel-Levitz) Buros’ Mental Measurements Yearbook list - Existing institutional data

Dangers of Satisfaction Surveys there is often a difference between an advisee receiving good, effective academic advising and being satisfied with the advising process: if any negative information is exchanged during the advising interaction, the student may respond negatively to the survey items even though the information provided was correct and the process of the interaction was appropriate the student will likely rate the advising provided based on the type of interaction desired (e.g., informational, relational)

Measures may (and should) include existing institutional data Information from Institutional Research, Admissions, Registrar, etc. can provide tracking data, GPAs, retention rates, and other information you can utilize as assessment data this can be a source of some of the multiple measures utilized (in addition to formal instruments, satisfaction surveys/inventories, and others)

Institutional Data Why reinvent the wheel, so to speak, if there are existing institutional data that relate to a question you would like answer regarding your advising program? Become knowledgeable regarding what type of data your institution is already collecting Get to know your institutional data people, and include them as a stakeholder in the assessment of your advising program - Remember that any single type of data should be utilized as just one measure among multiple measures for each phenomena you are evaluating within your overall assessment effort

For both process/delivery and student learning outcomes, you need to identify the minimum criteria for success of the outcome measure, e. g., - number of students exhibiting a specific learning performance percentage of students exhibiting a specific learning performance advisor rating of student performance - student rating of specific aspect of advising process - advisor rating of specific aspect of advising process - etc.

The Assessment Cycle (Maki, 2002, 2004) Gather Evidence Interpret Evidence Identify Outcomes Implement Change Mission/Purposes Educational Objectives

The Assessment Flowchart (adapted from Darling, 2005)
The Global Community for Academic Advising The Assessment Flowchart (adapted from Darling, 2005) Values Vision Mission Goals Programmatic Outcomes Process/Delivery Outcomes Student Learning Outcomes Cognitive, Psychomotor, Affective Mapping the Experience What experiences? When or by when? Gathering Evidence When gathered? Where gathered? How often gathered? From whom gathered? How gathered? Minimum performance criteria for success? Sharing and Acting Upon the Results Interpret how results inform practice How and with whom to share interpretation Follow up on implemented changes Start the process all over again!

The Assessment Matrix/Table Institutional Mission Statement Local Mission Statement Specific Goal or Objective Specific Process/Delivery Outcome or Student Learning Outcome Where Outcome Occurs When or By When Outcome Occurs Outcome Measure Minimum Performance Criteria for Success Data Instruments Action(s) Based on Outcome Data Mapping of Outcomes Adapted from Robbins, R. L. (2009). Evaluation and assessment of career advising. In K. Hughey, D. N. Burton Nelson, J. Damminger, and B. McCalla-Wriggins (Eds.) The Handbook of Career Advising (chapter 12). San Francisco: Jossey-Bass. and Robbins, R. (2011). Assessment and accountability of academic advising. In J. Joslin & N. Markee (eds.), NACADA Monograph #22: Academic Advising Administration: Essential Knowledge and Skills for the 21st Century (chapter 4). Manhattan, Kansas: NACADA.

Examples (included in Guide) - Process/Delivery outcome - Student Learning outcome

Exercise 1: Identifying Multiple Measures Select one of your Process/Delivery outcomes and identify three ways to measure whether or not that specific P/DO has been met (including outcome measures, minimum performance criteria for success, and data instruments)

The Assessment Matrix/Table Institutional Mission Statement Local Mission Statement Specific Goal or Objective Specific Process/Delivery Outcome Where Outcome Occurs When or By When Outcome Occurs Outcome Measure Minimum Performance Criteria for Success Data Instruments Action(s) Based on Outcome Data Mapping of Outcomes Adapted from Robbins, R. L. (2009). Evaluation and assessment of career advising. In K. Hughey, D. N. Burton Nelson, J. Damminger, and B. McCalla-Wriggins (Eds.) The Handbook of Career Advising (chapter 12). San Francisco: Jossey-Bass. and Robbins, R. (2011). Assessment and accountability of academic advising. In J. Joslin & N. Markee (eds.), NACADA Monograph #22: Academic Advising Administration: Essential Knowledge and Skills for the 21st Century (chapter 4). Manhattan, Kansas: NACADA.

Exercise 2: Identifying Multiple Measures Select one of your student learning outcomes and identify three ways to measure whether or not that specific SLO has been met (including outcome measures, minimum performance criteria for success, and data instruments)

The Assessment Matrix/Table Institutional Mission Statement Local Mission Statement Specific Goal or Objective Specific Student Learning Outcome Where Outcome Occurs When or By When Outcome Occurs Outcome Measure Minimum Performance Criteria for Success Data Instruments Action(s) Based on Outcome Data Mapping of Outcomes Adapted from Robbins, R. L. (2009). Evaluation and assessment of career advising. In K. Hughey, D. N. Burton Nelson, J. Damminger, and B. McCalla-Wriggins (Eds.) The Handbook of Career Advising (chapter 12). San Francisco: Jossey-Bass. and Robbins, R. (2011). Assessment and accountability of academic advising. In J. Joslin & N. Markee (eds.), NACADA Monograph #22: Academic Advising Administration: Essential Knowledge and Skills for the 21st Century (chapter 4). Manhattan, Kansas: NACADA.

An Additional And Important Consideration even if your outcome data supports the achievement of the desired SLO – how do you know that the student learning was a result of academic advising??? you have to ask students “where did you learn the information,” “how did you know to do the behavior,” and/or “how did you develop the appreciation” in regard to the SLO

Summary and Conclusions Evaluation of one outcome is not assessment Need multiple measures for multiple outcomes concerning multiple phenomena Measures should be qualitative, quantitative, direct, and indirect Outcomes should include process/delivery outcomes and student learning outcomes Measures should be developed as desired outcomes are identified

Exercise 3: DATA from multiple measures to inform specific student learning outcomes. DUE: Friday at Plenary 5

Student Learning Outcome: Student interacts with campus resources to achieve individual goals.

Scenario Institution of 7,000 students with a blended 2 & 4 yr. mission The advisors are evaluating students interaction with campus resources. DATA Available Survey data on need & satisfaction w/advising for resources Qualitative data from interviews with students Rubric data Advising Office Referrals Retention/Graduation Data

1) Survey Data/Need & Satisfaction
The Global Community for Academic Advising 1) Survey Data/Need & Satisfaction 1905 students responded in 20 days (27%) Class rank of Respondents 20% First Yr. 35% Second Yr. 17% Third Yr. 20% Fourth Yr. 8% Other Gender: 54% female/46% male Transfer Status: 50% transfer

Demographic – Race/Ethnicity Demographic – Age Race/Ethnicity Campus Community Respondents to Survey Asian 20% 22% Black 10% 13% Hawaiian/Pacific Islander 7% Latino/Hispanic 15% 25% Multi-racial/ethnic 9% 8% White/Caucasian 35% Other/Did Not Identify 4% 3% Age Group Campus Community Respondents to Survey 22 or Younger 71% 40% 23 to 25 10% 26 to 29 15% 25% 30 and older 14%

Survey: Where Advising Occurred
The Global Community for Academic Advising Survey: Where Advising Occurred Centralized Advising Dept. Trio Office Honors Other First Yr. 65% 12% 19% 3% 1% Second Yr. 49% 32% 4% 10% 5% Third Yr. 20% 2% 11% Fourth Yr. 15% 72% 0% 8%

Most Recent Meeting Last 2 wks Last month Last 3 month Last 6 month Last Yr. Never Don’t Remember Response % 32% 12% 13% 6% 27% 5% 47% had met w/advisor within last 3 months (895 students) 5% never but completed the survey

2) Student Comments (some out of 75 pages)
The Global Community for Academic Advising 2) Student Comments (some out of 75 pages) “We have good advisors in X dept. They are very good at sending out reminders about class registration, tuition, etc We can drop in and get help That easier than having an appt. “They aren’t helpful in determining what I could handle, maybe taking 3000 level classes this semester is a little much or change it around. And no one suggested I should get tutoring.” “She helps me a lot. I had transfer credit from Germany, and that was difficult because no one seemed to really know what they were doing, but she constantly followed up on things for me and got me the credits I needed. I truly appreciate her help.” “I went to the advisor to ask questions about studying abroad. She suggested that I not do that because I would delay my graduation.” “My advisor is a life-saver – seriously. I was so stressed out that I could not plan my schedule for next term. He recognized the signs and assisted me in connecting to counseling. It really helped. He walked me to the Counseling Office and made sure my situation was understood.” “My academic advisor is fine but I am not sure who to see for career advise. I need a resume and some real world experience.” “The advisors at the X institution are great, but I would really enjoy more help on guiding me to a career after I am done, to help give me more of a focus.” “I have always found the advisors to be knowledgeable. I do wish they would have encouraged and provided more guidance towards extracurricular programs such as tutors.”

Themes from Comments Themes Friendly vs. not friendly/Developing a relationship/Having one advisor Lack of consistency of referral to services & resources Appt process: not easy to establish, times available, wait a long time, rushed Requirements not consistent: DARS to Advisor/Advisor to Advisor Location: Privacy to speak with advisor Assume advisors have tremendous power within department when classes are scheduled/classes not offered changing degree requirements

3) Rubric Data - Ways to use this
The Global Community for Academic Advising 3) Rubric Data - Ways to use this What does this rubric tell you about student’s understanding of campus resources? You can develop a score range to see what the average score is for each criterion point. Excellent = 4 Competent = 3 Needs More Info = 2 Not Aware = 1 You can estimate out of 1000, what % of students fall into each performance area for each criterion. Each criterion row would be equal to 100%. Note: Juniors are defined as third year students at institution

4) Appointment Data–Central Advising Office Recorded No. of Referrals to Specific Services Dept. Advisor Career Services Honors TRIO Other 2002 2537 1975 299 167 2626 2003 3023 1862 381 189 2706 2004 4178 2275 525 527 3296 2005 4203 2401 499 672 3402 2006 4094 2307 489 634 3807

5) Institutional Data – Associate Program
The Global Community for Academic Advising 5) Institutional Data – Associate Program Retention Graduation 2nd yr 3rd yr 2002 65% 12% 2003 61% 14% 2004 70% 22% 2005 71% 23% 2006 69% 24% 2nd yr 3rd yr 4th yr 2002 42% 49% 50% 2003 43% 51% 53% 2004 41% 61% 62% 2005 60% 64% 2006 63% 65%

5) Institutional Data – Bachelor’s Program
The Global Community for Academic Advising 5) Institutional Data – Bachelor’s Program Retention Graduation Class 2nd Yr 5th Yr 7th Yr 2002 91.1% 11% 2% 2003 92.2% 13% 2004 88.2% 8% 3% 2005 89.1% 2006 90.2% 4% Class 4th Yr 5th Yr. 6th Yr. 2002 60% 81% 85% 2003 55% 82% 83% 2004 57% 79.1% 84% 2005 59.5% 80% 2006 60.1%

Based on these data . . . What do you know about students interacting with campus resources to achieve goals? What other data would inform your question? What tools or resources would you develop to collect additional data? What institutional or national tools would you request? Based on these data, are there any activities in advising that you would consider in the short-term? Discussion Friday at Plenary 5

Participant Learning Outcomes As a result of attending Plenary 4, participants will: Cognitive Understand the importance of using multiple measurements for any single desired Student Learning Outcome (SLO) Understand why one single measure for any desired SLO may not result in valid data Understand the importance of using multiple measurements for any single desired Process/Delivery (P/D) outcome Understand why one single measure for any desired P/D outcome may not result in valid data Understand the issues regarding relying on student perception surveys on which to base decisions Behavioral Identify multiple measures for at least one of their identified SLOs Complete the measurement columns of the assessment matrix for at least one of their SLOs Identify multiple measures for at least one of their identified P/D outcomes Complete the assessment matrix for at least one of their identified P/D outcomes Affective Appreciate the need for multiple outcome measures for a desired SLO Appreciate the need for multiple outcome measures for a desired P/D outcome Value the need for valid data to determine whether a desired outcome has been met

NACADA Executive Office Kansas State University 2323 Anderson Ave, Suite 225 Manhattan, KS Phone: (785) Fax: (785) © National Academic Advising Association The contents of all material in this presentation are copyrighted by the National Academic Advising Association, unless otherwise indicated. Copyright is not claimed as to any part of an original work prepared by a U.S. or state government officer or employee as part of that person's official duties. All rights are reserved by NACADA, and content may not be reproduced, downloaded, disseminated, published, or transferred in any form or by any means, except with the prior written permission of NACADA, or as indicated below. Members of NACADA may download pages or other content for their own use, consistent with the mission and purpose of NACADA. However, no part of such content may be otherwise or subsequently be reproduced, downloaded, disseminated, published, or transferred, in any form or by any means, except with the prior written permission of, and with express attribution to NACADA. Copyright infringement is a violation of federal law and is subject to criminal and civil penalties. NACADA and National Academic Advising Association are service marks of the National Academic Advising Association. Identifying and Using Multiple Measures NACADA Assessment Institute

The Global Community for Academic Advising

Similar presentations

Presentation on theme: "The Global Community for Academic Advising"— Presentation transcript:

Similar presentations

About project

Feedback

Log in

Auth with social network:

The Global Community for Academic Advising

Similar presentations

Presentation on theme: "The Global Community for Academic Advising"— Presentation transcript:

Similar presentations

About project

Feedback