SURVEY OF RECENT GRADUATES AS AN INSTRUMENT FOR ASSISTING IN THE ASSESSMENT OF INSTITUTIONAL EFFECTIVENESS Dr. Teresa Ward Ms. Beth Katz Office of Institutional Research Georgia State University

Presentation Outline Conceptual Model Technical Logistics Dissemination of Survey Data Utilization of Data

ObjectivesContentQuestions General education learning outcomes critical thinking skills, writing skills, oral presentation skills, research and analytical skills, cooperative learning skills, independent learning skills, technical skills 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 13 Program of study learning outcomes instruction in major program, academic support, scholarship, preparation for Advanced study/career, faculty support 23, 24, 25 Student Engagement interaction with faculty, interaction with other students, social events, campus meetings, athletic events, use Of recreation facilities 6, 17, 27, 28, 29, 30, 31 Student Satisfaction courses, instruction, advisement, Recommendation of program 14, 15, 16, 18, 19, 20, 21, 22, 26 Survey Matrix General Academic Skills – Questions 1-13 Major Area of Study and University Support – Questions Engagement – Questions Comments

Survey Administration Current ProcessOnline Process Time/Expense Level Survey framework (matrix construction) High Material costsHighNone Data integrityMedium to HighHigh Timely dissemination of survey information LowHigh Response RateLow to MediumMedium to High

Survey of Recent Graduates Response Rates

Please rate the degree of competency you had in the following areas when you entered/graduated from Georgia State University Graph 1 (Entering)Graph 2 (Exiting) *Mean range: 1=very weak competency to 5=very strong competency **p<.001 all pairs Learning Outcomes Communication Technology Collaboration Quantitative Skills Critical Thinking Contemporary Issues

Mean English Majors All Others Ability to write clearly and effectively Entering Ability to write clearly and effectively Exiting

English Majors (Seniors) All Others (Seniors) Prepared two or more drafts of a paper or assignment before turning it in Number of written papers or reports of 20 PAGES OR MORE Number of written papers or reports BETWEEN 5 AND 19 PAGES Number of written papers or reports of FEWER THAN 5 PAGES Institutional contribution: Writing clearly and effectively NSSE Writing Items

Ability to analyze quantitative problems Entering Mean Ability to analyze quantitative problems Exiting Math Majors All Others

Mathematics Majors (Seniors) All Others (Seniors) Number of problem sets (problem- based homework assignments) that take you MORE than an hour to complete Number of problem sets (problem- based homework assignments) that take you LESS than an hour to complete Institutional contribution: Analyzing quantitative problems Coursework emphasized: MEMORIZING facts, ideas, or methods from your courses and readings so you can repeat them in pretty much the same form NSSE Quantitative Items

Apply scientific reasoning in problem-solving Entering Mean Physics Majors All Others Apply scientific reasoning in problem-solving Exiting

GPA Mean of communication (exiting) Mean of communication (entering)

GPA Mean of quantitative skills (entering) Mean of quantitative skills (exiting)

GPA Mean of critical thinking (entering) * * *p<.01 Mean of critical thinking (exiting)

GPA Mean of technology (entering) Mean of technology (exiting)

*Mean range: 1=poor to 4=excellent Spring 2005 Summer 2005 Fall 2005 Spring 2006 Summer 2006 Fall 2006 Spring 2007 Summer 2007 Fall 2007 Mean Quality of instruction In your major program Scholarly competency Of faculty in your major program point scale: Poor to Excellent

Utilization of Survey Data Department Level –Identify strengths and weaknesses using comparative data –Identify trends and implications of program changes using longitudinal data –Ability to triangulate survey data with other assessment measures –Starting point for conversations

University Level –Identify trends and implications of University policy changes using longitudinal data –Ability to triangulate survey data with other assessment measures –Contribute to strategic planning and policy discussions

Conclusion High reliability index Focused on indirect measures of learning outcomes Saved OIR a considerable amount of money and time via online administration Increased response rates Facilitated access to the data Contributed to department and University assessment activities

Teresa Ward – Bethann Katz – Office of Institutional Research, Georgia State University –