Assessing the Impact of Faculty Advising: Implications for a Peer Advising Program Anecdotal evidence suggested that a discrepancy existed between what faculty believed students knew as a function of faculty advising and what students actually did know. We specifically questioned whether students had the knowledge necessary to successfully navigate advising issues on campus. We wanted empirical evidence to support the perception that students had the necessary knowledge concerning academic planning, use of academic and career development resources, and the confidence to engage in the tasks necessary to control and master their own academic endeavors. We created an assessment to evaluate five specific goals related to advising, recognizing that if students do not demonstrate proficiency on the assessment, additional programming (such as a peer advising program) would be necessary. Peter Swerdzewski Sara J. Finney Anna Lynn Bell Step 1: Assessment Design and Creation of Objectives Using assessment to determine if students lack the knowledge and confidence that would necessitate a peer advising program Step 2: Instrument Development Step 3: Administration and Samples Step 4: Results Simply surveying students about the university ’ s academic advisors may reveal perceptions about the advising program, but may not necessarily reveal what students know or feel confident in doing with respect to academic planning. Four outcomes of the university ’ s advising program were identified that address what students should know as a result of their advising experiences. One outcome of the university ’ s advising program was identified that addresses how confident students are in their abilities to complete the advising tasks expected of them by the university. All outcomes were crafted by a team consisting of the individual responsible for advising at the institution, a member of the Student Government Association, and liaisons from the university ’ s assessment center. Outcome 1: Increased student knowledge of academic resources Four items that address knowledge of the location of resources, including: Registration dates and deadlines Financial aid The Registrar’s Office The Career Planning Office Assessment Day students answered 66.07% of the items correctly. Make-up students answered 64% of the items correctly. Outcome 2: Increased understanding of the student’s role in academic advising Six items that address students’ understanding of their responsibility in academic planning, including: Course overrides Changing majors Scheduling & preparing for a meeting with advisor Legitimate expectations of advisor Assessment Day students correctly answered 65.98% of the items correctly, indicating that students understand their role in academic advising. Make-up students correctly answered 60.74% of the items correctly. Outcome 3: Greater knowledge of the nuts and bolts of academic advising like how to use e-campus, how to register for classes and how to make a four-year plan Nine items that address students’ understanding of the policies and processes related to academic planning, including: Credit requirements (General Education, graduation, etc.) Grade point requirements Course withdrawal rules Which academic requirements apply to various situations Assessment Day students correctly answered 62.06% of the items correctly, indicating a good understanding of the nuts and bolts of academic advising. Make-up students correctly answered 60.99% of the items correctly. Outcome 4: Increased awareness of special opportunities like study abroad, internships, and competitive scholarships 84.29% of the Assessment Day students correctly answer the single item that served as the indicator for the outcome % of make-up students correctly answered the single item correctly. One should be cautious in making inferences from this single item to the overall outcome because the item clearly does not cover the breadth of the outcome. Outcome 5: Increased student confidence in fulfilling the graduation requirements and utilizing academic tools and resources without the help of their faculty adviser. Five items that address students’ confidence in the following: Completing graduation requirements Ability to interpret degree progress report Process for requesting an override Use of Web site to gather requirement information Navigating eCampus (student registration Web portal) All students indicated a confidence level somewhere between “a fair amount of confidence” and “much confidence” to fulfill the graduation requirements and utilize academic tools and resources without the help of their faculty adviser. Given the empirical evidence from this study, it is apparent that students are knowledgeable about advising-related procedures and are confident in their abilities to carry out advising-related tasks. When synthesizing all the findings, it appears that some students are not satisfied with some specific advisors, and that there are some specific areas of the advising program that could be improved, but that investing in a peer advising program may not be the most judicious use of resources due to the high levels of knowledge and confidence reflected in the assessment results. Based on these assessment results, the university will take/has taken the following actions: 1.Assessment results were and continue to be disseminated to key university stakeholders (university president, deans, faculty senate, student government). 2.A decision was made that it is premature to commit funding to the peer advising initiative until further studies can be conducted. 3.Major Advising Programs, with student input through the Student Government Association, will work to expand the advising website to address some of the students’ advising knowledge deficiencies highlighted in the assessment results. 4.Because students perceive that advisors have a lack of knowledge about the curriculum and related advising issues, Major Advising Programs will initiate conversations with the Center for Faculty Innovation to brainstorm ways of delivering more comprehensive training to faculty advisers. The committee created indicators for each of the five outcomes. These items were constructed to represent the breadth and depth of each of the five outcomes. Rather than asking if students thought they knew specific knowledge related to the institution ’ s advising procedures, we created a multiple- choice test so we would actually have evidence of students ’ knowledge. Items were crafted under the assumption that a student with between 45 and 70 credit hours should correctly respond to them. In addition to the multiple choice items, students also responded to Likert-type items addressing their confidence to complete advising tasks (self-efficacy). Traditional survey-type and open-ended items were included in order to provide a profile of students ’ attitudes toward current advising programming and a new peer advising initiative. This information was ancillary; the primary focus was the assessment of student knowledge and self-efficacy related to academic planning, not their attitude about their advisor or the university ’ s advising program. The Administration The test was administered at the university ’ s annual Assessment Day, a required day in which students with between 45 and 70 credit hours are asked to take a three-hour battery of instruments used to assess the institution ’ s general education and student affairs programs. The Advising Assessment was administered via computer in a proctored session. The Students A random sample of all students at the university with between 45 and 70 credits was required to take the advising assessment ( “ Regular Sample ” ; N = 401). This sampling provided results that are readily generalizable to the population of students at the university with 45 to 70 credit hours. Additionally, a sample of those students who did not attend the required Assessment Day were randomly assigned to take the advising assessment during make-up sessions ( “ Make-Up Sample ” ; N = 65). These students were administered the Advising Assessment because it was thought that if they intentionally skipped the required Assessment Day, they would also be the type of student who may not be competent in navigating the advising-related tasks expected of them by the university. The Samples Regular Sample ( N = 401) Make-Up Sample ( N = 65) Avg. Age19.72 yrs20.26 yrs % Female66.1%60.0% % White80.0%73.8% % Transfers12.2%16.9% % In-State67.6%69.2% Avg. # of Credits 5355 Avg. SAT Score Most common major Interdisciplinary Liberal Studies Marketing Sample Multiple-Choice Item: What is the minimum cumulative grade point average a student must maintain to be in good academic standing?  1.0  1.5  2.0  2.5  3.0 Sample Confidence Item: How confident are you in your ability to interpret your degree progress report?  No confidence at all  A little confidence  A fair amount of confidence  Much confidence  Very much confidence  Complete confidence Sample Attitude Item: To what extent would you trust the information from a student peer adviser? Select the statement that is most characteristic of you.  I would trust information from a peer adviser more than I would a faculty adviser.  I would trust information from a peer adviser as much (equally) as I would a faculty adviser.  I would trust information from a peer adviser somewhat less than I would a faculty adviser.  I would not trust at all information from a peer adviser. Step 5: Use of Results Ancillary Analyses: Although empirical evidence indicates that students’ knowledge and confidence related to advising at JMU is quite high (especially given the low-stakes testing environment in which the data was collected), approximately 30% of the Assessment Day sample (N = 120) and 40% of the make-up sample (N = 26) indicated that they are not satisfied with the advising they have received at JMU. Primary reasons stated for this lack of satisfaction include the belief that advisers lack competence or quality in the information they provide, and scheduling issues detract from the quality of advising. Students suggest that advisers become more knowledgeable in advising, and that the scheduling of meetings improve. Importantly: The negative attitudes expressed toward advising were not due to students’ lack of knowledge of academic planning (students had the knowledge, but there were still complaints about advising). Furthermore: Those students who may be most in need of a peer advising center (e.g., have avoidant or adversarial attitudes toward the university, such as students who skip the required Assessment Day and must attend make-up sessions) tend to indicate they would not feel comfortable approaching a peer advisor and may not trust a peer advisor. JAMES MADISON UNIVERSITY