The Global Community for Academic Advising NACADA Executive Office Kansas State University 2323 Anderson Ave, Suite 225 Manhattan, KS  66502-2912 Phone: (785) 532-5717    Fax: (785) 532-7732 e-mail: nacada@ksu.edu © 2013 National Academic Advising Association The contents of all material in this presentation are copyrighted by the National Academic Advising Association, unless otherwise indicated. Copyright is not claimed as to any part of an original work prepared by a U.S. or state government officer or employee as part of that person's official duties. All rights are reserved by NACADA, and content may not be reproduced, downloaded, disseminated, published, or transferred in any form or by any means, except with the prior written permission of NACADA, or as indicated below. Members of NACADA may download pages or other content for their own use, consistent with the mission and purpose of NACADA. However, no part of such content may be otherwise or subsequently be reproduced, downloaded, disseminated, published, or transferred, in any form or by any means, except with the prior written permission of, and with express attribution to NACADA. Copyright infringement is a violation of federal law and is subject to criminal and civil penalties. NACADA and National Academic Advising Association are service marks of the National Academic Advising Association. Conducting Assessment of Academic Advising 2013 NACADA Summer Institute Jacksonville, Florida Dr. Karen Boston Dr. Rich Robbins

The Global Community for Academic Advising Overview Review of reasons for performing assessment Review of differences between evaluation and assessment The assessment cycle Identifying stakeholders Advising is teaching Identifying student learning outcomes from various sources Outcome measurement Acting upon the results Assessment as research

The Global Community for Academic Advising Assessment Has Multiple Purposes Program effectiveness Program improvement Program accountability Enhancing student success persistence retention Activities aimed at student success, program improvement and accountability are all important

The Global Community for Academic Advising Assessment is An on-going cycle of activity A gathering of a variety of information and data A team effort with faculty, staff, students, and administrators all actively engaged A complex process of comparison

The Global Community for Academic Advising Remember… “…a lack of assessment data can sometimes lead to policies and practices based on intuition, prejudice, preconceived notions, or personal proclivities – none of them desirable bases for making decisions” Upcraft and Schuh (2002. p. 20)

The Global Community for Academic Advising A Distinction with a Difference Assessment Evaluation Focus Programmatic Individual Process Continuous & Embedded Episodic Measures Outcomes Effectiveness Part of assessment, part of measuring outcomes

The Global Community for Academic Advising The Assessment Cycle (Maki, 2002, 2004) Gather Evidence Interpret Evidence Identify Outcomes Implement Change Mission/Purposes Educational Objectives

The Assessment Flowchart The Global Community for Academic Advising The Assessment Flowchart Values Vision Mission Goals Programmatic Outcomes Process/Delivery Outcomes Student Learning Outcomes Cognitive, Psychomotor, Affective Mapping the Experience What experiences? When or by when? Gathering Evidence When gathered? Where gathered? How often gathered? From whom gathered? How gathered? Minimum performance criteria for success? Sharing and Acting Upon the Results Interpret how results inform practice How and with whom to share interpretation Follow up on implemented changes Start the process all over again! (adapted from Darling, 2005, 2010)

The Global Community for Academic Advising The Assessment Matrix/Table Institutional Mission Statement Local Mission Statement Specific Goal or Objective Specific Process/Delivery Outcome or Student Learning Outcome Where Outcome Occurs By When You Want Outcome To Occur Outcome Measure Data Instruments Minimum Performance Criteria for Success (Threshold) Action(s) Based on Outcome Data Mapping of Outcomes Adapted from Robbins, R. L. (2009). Evaluation and assessment of career advising. In K. Hughey, D. N. Burton Nelson, J. Damminger, and B. McCalla-Wriggins (Eds.) The Handbook of Career Advising (chapter 12). San Francisco: Jossey-Bass. Robbins, R. (2011). Assessment and accountability of academic advising. In J. Joslin & N. Markee (eds.), NACADA Monograph #22: Academic Advising Administration: Essential Knowledge and Skills for the 21st Century (chapter 4). Manhattan, Kansas: NACADA. Robbins, R. & Zarges, K.M. (2011). Assessment of Academic Advising: A Summary of the Process. NACADA Clearinghouse of Academic Advising Resources: http://www.nacada.ksu.edu/Clearinghouse/AdvisingIssues/assessment-Robbins-Zarges.htm

The Global Community for Academic Advising Exercise 1: Identify Your Stakeholders

The Global Community for Academic Advising Key Terms Student Learning Outcomes – Articulate what students are expected to know, do, and appreciate as a result of involvement in the academic advising experience Process/Delivery Outcomes – Articulate the expectations for how academic advising is delivered and what information should be delivered through the academic advising experience Mapping – The process of determining when, where and how the outcomes for advising will be accomplished over the students’ academic career and beyond

The Global Community for Academic Advising Process/Delivery Outcomes Do not assess student learning, but evaluate what processes occurred during the advising interaction Typically what is evaluated via student satisfaction surveys e.g., “the advisor provided accurate information” e.g., “the setting for the advising session was appropriate” etc.

The Global Community for Academic Advising Focus on Student Learning: Advising is Teaching

The Global Community for Academic Advising Advisor as Teacher facilitator of communication coordinator of learning experiences referral agent who connects students with all of the institution’s resources and co-curricular opportunities that can help them be successful

The Global Community for Academic Advising Exercise 2: What skills used in teaching are also used in advising?

The Global Community for Academic Advising Teaching Knowledge of subject matter Preparation - planning and organization of course material Advising Knowledge of institutional policies, procedures, programs, resources Preparing for advising meetings

The Global Community for Academic Advising Teaching Engaging students in participation of their learning Giving students feedback on their progress Helping students learn to analyze and problem solve Advising Guiding students to be self-directed and autonomous Working with student to regularly evaluate the student’s goals and progress Assisting students in decision-making skills

The Global Community for Academic Advising Teaching Clarity of presentation of subject matter Establishment of dialogue with students in the classroom Demonstration of excellent listening skills Advising Sharing information in a clear manner Leading students to question and interact with the advisor Listening to what advisees are saying verbally and non-verbally

The Global Community for Academic Advising Teaching Assigning out-of class work Working from a developmental perspective Providing a learner-centered environment Advising Assigning out-of-session tasks Working from a developmental perspective Providing a learner-centered environment

The Global Community for Academic Advising Teaching Identification and communication of student learning outcomes for the curriculum or course Use of a course syllabus Advising Identification and communication of student learning outcomes for the advising process/interaction Use of an advising syllabus 20

The Global Community for Academic Advising Focus on the Learner/Advisee As a result of academic advising, what do we want students to demonstrate that they… Know (cognitive student learning outcome) Are able to do (behavioral student learning outcome) Value and appreciate (affective student learning outcome)

The Global Community for Academic Advising It’s All About Student Learning

The Global Community for Academic Advising Student Learning Outcomes What do you expect students to be able to Know (cognitive learning) Do (behavioral learning) Value and Appreciate (affective learning) as a result of participating in academic advising?

The Global Community for Academic Advising Cognitive Elements Know general education requirements Know about academic support services Know how to use the student information system to register Know how to use the catalog Etc……. What do we want students to KNOW as a result of participating in academic advising?

The Global Community for Academic Advising Exercise 3: Identify Three Things You Want Students to Know as a Result of Academic Advising

The Global Community for Academic Advising Tips on Developing SLOs avoid compound SLOs, e.g., “students understand and value the purpose of a liberal arts education,” “students understand and utilize their degree progress reports” as you are developing each SLO, think about where and when the opportunities for student learning may occur for that desired SLO as you are developing each SLO, think about how you might measure whether or not the SLO has been achieved

Skills/Behavioral/Psychomotor Elements The Global Community for Academic Advising Skills/Behavioral/Psychomotor Elements Generate their degree audit Make advising appointments Keep advising appointments Ask for help Access course descriptions and degree requirements using the online catalog Etc…. What do we want students to Do as a result of participating in academic advising?

The Global Community for Academic Advising Exercise 4: Identify Three Things You Want Students to Do as a Result of Academic Advising

The Global Community for Academic Advising Affective Elements Value/Appreciate general education Value/Appreciate the advising relationship Value/Appreciate the process of learning Etc…. What do we want students to Value or Appreciate as a result of participating in academic advising?

The Global Community for Academic Advising Exercise 5: Identify Three Things You Want Students to Value or Appreciate as a Result of Academic Advising

The Global Community for Academic Advising Developing SLOs from Programmatic Goals

The Global Community for Academic Advising What do students need to demonstrate they know, can do, or appreciate in order to say that your programmatic goals have been achieved?

The Global Community for Academic Advising Exercise 6: Identify the services your advising program offers Identify 1 student learning outcome for each service Prioritize If you could assess only two to start, which ones would they be?

The Global Community for Academic Advising Deriving SLOs from other sources

The Global Community for Academic Advising Clear and measurable student learning outcomes for academic advising may also be derived from the NACADA Concept of Academic Advising the NACADA Core Values the CAS Standards for Advising

The Global Community for Academic Advising NACADA Concept of Advising http://www.nacada.ksu.edu/Clearinghouse/AdvisingIssues/Concept-advising-introduction.htm multi-dimensional and intentional grounded in teaching and learning includes specific purpose and content includes specific student learning outcomes for academic advising

The Global Community for Academic Advising NACADA Statement of Core Values Six values that NACADA adopted in 2004 as being important regarding what academic advisors are expected to do as part of the advising interaction May be easily translated into Student Learning Outcomes… http://www.nacada.ksu.edu/Clearinghouse/AdvisingIssues/Core-Values.htm

The Global Community for Academic Advising NACADA Core Values: Advisors Teach Students… to value the learning process to apply decision-making strategies to put the college experience into perspective to set priorities and evaluate events to develop thinking and learning skills to make choices

The Global Community for Academic Advising NACADA Core Values as Student Learning Outcomes As a result of academic advising, students will value the learning process apply decision-making strategies put the college experience into perspective set priorities and evaluate events develop thinking and learning skills make choices

The Global Community for Academic Advising Using the CAS Standards in Program Assessment Council for the Advancement of Standards in Higher Education (CAS) Standards and Guidelines for Academic Advising http:// http://www.nacada.ksu.edu/Resources/Clearinghouse/View-Articles/CAS-Advising-Standards.aspx

The Global Community for Academic Advising CAS Standards for Academic Advising 16 “relevant and desirable” student learning outcomes for academic advising (revised in 2006) Developed based on every type of higher education academic setting Each outcome will be achieved based on exposure to student and student maturity May require a systematic approach re: introducing these concepts and expectations in order for students to achieve these student learning outcomes

The Global Community for Academic Advising Advising Programs… …promote learning and development in students by encouraging experiences which lead to: intellectual growth the ability to communicate effectively appropriate career choices leadership development the ability to work independently and collaboratively and so on for all 16 student learning outcomes…

The Global Community for Academic Advising It is not necessary to attempt to use all of the CAS Standards as your outcome statements… …for those you may choose to use, be sure to develop related student learning outcomes as well as process/delivery outcomes

The Global Community for Academic Advising Developmental SLOs The offering of opportunities for students to learn the desired SLOs will vary by SLO The time by which your students need to demonstrate achievement of a specific SLO will vary by SLO Not all SLOs are achieved at same time

The Global Community for Academic Advising Thus, you may also take a developmental approach, and utilize different SLOs as desired outcomes for students at different times during their college careers

The Global Community for Academic Advising Mapping of Outcomes The process of determining when, where and how the SLOs for academic advising will be accomplished over the students’ academic career and beyond

The Global Community for Academic Advising Mapping the Learning Experience What should be learned: e.g., student knows the components of the institution’s General Education requirements Where it should be learned: e.g., orientation workshops, advising sessions, personal reading of catalog or curriculum guide When it should be learned: e.g., prior to first year (orientation); by end of first year (via advising sessions); by end of first year (via personal reading)

The Global Community for Academic Advising The Assessment Matrix/Table Institutional Mission Statement Local Mission Statement Specific Goal or Objective Specific Process/Delivery Outcome or Student Learning Outcome Where Outcome Occurs When or By When Outcome Occurs Outcome Measure Data Instruments Minimum Performance Criteria for Success (Threshold) Action(s) Based on Outcome Data Mapping of Outcomes

The Global Community for Academic Advising Exercise 7: Mapping Your Desired Outcomes: Identifying Where/When Outcomes Occur

The Global Community for Academic Advising Measuring Outcomes Once the desired Process/Delivery outcomes and Student Learning outcomes have been identified, as well as when and where they will occur, the next step is to determine who or what will be measured and how the data will be gathered… …using multiple measures of varying types

Initial Considerations The Global Community for Academic Advising Initial Considerations True assessment of academic advising should involve multiple measures beyond a simple satisfaction survey or advisor evaluation individual measures evaluate (rather than assess) a single dimension of a single phenomenon

The Global Community for Academic Advising Types of Measures Qualitative exploratory small samples open-ended emerging information subjective, inductive interpretation of data examples focus groups case studies naturalistic observation Information/data in form of rich, in-depth responses (words)

Types of Measures continued… The Global Community for Academic Advising Types of Measures continued… Quantitative descriptive large samples structured objective, deductive interpretation of data examples questionnaires surveys experiments Information/data in form of numbers, measures (statistics)

Types of Measures continued… The Global Community for Academic Advising Types of Measures continued… Direct may be qualitative or quantitative examples direct observation of advising interaction pre-test/post-test of variable leading to desired outcome standardized test or inventory measuring student learning tracking of student data (enrollment rates, retention rates, GPAs, transcript analysis, etc.) counts of use of services advisor:student ratios

Types of Measures continued… The Global Community for Academic Advising Types of Measures continued… Indirect may be qualitative or quantitative examples focus groups surveys, questionnaires interviews reports tracking of student perceptions (satisfaction, ratings of advisors, ratings of service, etc.) tracking of advisor perceptions (student preparedness, estimation of student learning, etc.)

Examples of Existing Instruments The Global Community for Academic Advising Examples of Existing Instruments To be used as just one measure among multiple measures ACT Survey of Academic Advising Noel-Levitz Student Satisfaction Inventory (SSI) Winston and Sandor’s Academic Advising Inventory (AAI) NACADA Assessment of Advising Commission www.nacada.ksu.edu/Commissions/C32/index.htm

The Global Community for Academic Advising Other Sources To be used as just one measure among multiple measures Data from National Survey of Student Engagement (NSSE) Data from other nationally normed, standardized instruments Existing institutional data

The Global Community for Academic Advising Institutional Data Why reinvent the wheel, so to speak, if there are existing institutional data that relate to a question you would like answer regarding your advising program? Become knowledgeable regarding what type of data your institution is already collecting Get to know your institutional data people, and include them as a stakeholder in the assessment of your advising program Remember that any single type of data should be utilized as just one measure among multiple measures for each phenomena you are evaluating within your overall assessment effort

The Global Community for Academic Advising For both process/delivery and student learning outcomes, you need to identify the minimum criteria for success of the outcome measure, e. g., number of students exhibiting a specific learning performance percentage of students exhibiting a specific learning performance advisor rating of student performance student rating of specific aspect of advising process advisor rating of specific aspect of advising process etc.

The Global Community for Academic Advising Suggestion In the absence of any existing relevant measures of the identified student learning outcomes for academic advising, the initial cycle of assessment for any given desired student learning outcome should be considered a baseline data gathering effort

The Global Community for Academic Advising The Assessment Matrix/Table Institutional Mission Statement Local Mission Statement Specific Goal or Objective Specific PDO or SLO Where Outcome Occurs When or By When Outcome Occurs Outcome Measures Data Instruments Minimum Performance Criteria for Success (Threshold) Action(s) Based on Outcome Data Mapping of Outcomes Adapted from Robbins, R. L. (2009). Evaluation and assessment of career advising. In K. Hughey, D. N. Burton Nelson, J. Damminger, and B. McCalla-Wriggins (Eds.) The Handbook of Career Advising (chapter 12). San Francisco: Jossey-Bass

The Global Community for Academic Advising Exercise 8: Identifying Multiple Measures Select one student learning outcome, and identify three ways to measure whether or not that specific outcome has been met

The Global Community for Academic Advising An Additional And Important Consideration even if your outcome data supports the achievement of the desired SLO – how do you know that the student learning was a result of academic advising??? you literally have to ask students “where did you learn the information,” “how did you know to do the behavior,” and/or “how did you develop the appreciation” in regard to the SLO

The Global Community for Academic Advising Advising Syllabus as a Tool for Communicating Key Elements If academic advising is teaching… Advisors are teachers Teachers have a discipline The advisor’s discipline is academic advising Academic advising is a discipline Individual academic advisors offer the “course” academic advising Individuals in disciplines author unique courses Each course has a syllabus

The Global Community for Academic Advising What Does an Advising Syllabus Include? Purpose of academic advising Scope of academic advising Tools, texts, resources SLOs for academic advising Advisor responsibilities Advisee responsibilities Criteria for successful academic advising experiences (outcome measurements) Other elements as individually appropriate

The Global Community for Academic Advising Acting Upon the Outcome Data

The Global Community for Academic Advising Determine How and With Whom Results Are Shared Administration: President, Provost, various committees via annual report, strategic plan, white paper, Web sites, etc. Faculty: all faculty, curricular committees, faculty advisors via performance reviews, annual reports, strategic plans, Web sites, etc. Students: all students, student advisees, student senate, student groups via newsletters, annual reports, Web sites, etc. Budgeting entities via annual reports, budget requests, Web sites, etc. Accreditors via self-studies, accreditation reports, Web sites, etc.

The Global Community for Academic Advising Interpret How Results Will Inform Decision Making Revise pedagogy or curriculum Develop/revise advisor training programs Design more effective programming advising, orientation, mentoring, etc. Increase out-of-class learning opportunities Shape institutional decision making planning, resource allocation Other…

The Global Community for Academic Advising Decide How You Will Follow-up on Implemented Changes Timetable to implement changes implement all or specific components on a schedule Assessment of implemented changes repeat assessment cycle again Continuous assessment assessment is on-going

The Global Community for Academic Advising Professional Development Using assessment to inform and support professional development Revise advisor training and development programming accordingly Demonstrate the need for additional training and development Demonstrate the need for additional resources to meet goals

The Global Community for Academic Advising Building a Culture and Capacity for Assessment The Culture Commitment Communication Collaboration The Capacity Support

The Global Community for Academic Advising Exercise 9: Acting Upon the Outcome Data Identify at least one use for the outcome data of your assessment of advising

The Global Community for Academic Advising At the end of the day, assessment of academic advising is all about… developing consensus around collective expectations about student learning that should occur in advising gathering evidence in order to understand student learning resulting from academic advising using this evidence to support improvements in academic advising that will contribute to improvements in student learning

The Global Community for Academic Advising Remember: assessment is much more than just a single evaluation

The Global Community for Academic Advising Assessment as Research

The Global Community for Academic Advising Why Research Academic Advising? Academic advising is a collective experience among most college students Academic advising is an integral part of student development Academic advising is teaching, with a discipline and pedagogy Professional disciplines utilize inquiry and resulting data to inform decision making There is much anecdotal information re: the power of effective advising, but little empirical research

The Global Community for Academic Advising Wide Range of Audiences Peers in academic advising Advising administrators Higher Education Field of Advising Deans, Provosts, and Presidents Individual advisors Students Parents

The Global Community for Academic Advising Scholarly Inquiry May Include… Hypothesis testing Replication of existing knowledge in new setting Discovery of a novel phenomenon Development of a new theory Creation of new knowledge Evaluation of effectiveness of new implementation or approach

The Global Community for Academic Advising Methods of Inquiry Experimental Quasi-experimental Ex-post facto (after-the-fact) Correlational Historical Ethnographic (cultural interpretation) Phenomenological (description of experience) Case study Longitudinal Program Assessment

The Global Community for Academic Advising Continuing Assessment as Research Once you have obtained assessment data and act upon it via implementing a new strategy or model, the subsequent assessment of the new implementation serves as research: experimental assessment current versus historical data ethnographic inquiry qualitative inquiry quantitative inquiry other research/inquiry

The Global Community for Academic Advising Learning Outcomes for Presentation Cognitive Learning Outcomes: Understand the differences between assessment and evaluation Understand the differences between Process/Delivery Outcomes (PDO) and Student Learning Outcomes (SLO) Understand the concept of “mapping” of desired outcomes Understand the importance of using multiple measures for any single desired outcome

The Global Community for Academic Advising Learning Outcomes for Presentation Behavioral Learning Outcomes: Identify your stakeholders in the assessment of advising Identify at least one SLO for your assessment of advising process Map at least one SLO Identify multiple measures for at least one SLO Identify at least one use for the outcome data of your assessment of advising

The Global Community for Academic Advising Learning Outcomes for Presentation Affective Learning Outcomes: Value the cyclical process of assessment Appreciate the importance of assessment (besides accountability) Appreciate the value of student learning in advising Appreciate the importance of assessing student learning in advising Value the process of identifying opportunities for learning (mapping) in the advising process Appreciate the need for multiple measures for any single desired PDO or SLO Value the role of outcome data in informing decisions

The Global Community for Academic Advising Questions/Discussion

The Global Community for Academic Advising THANK YOU!