1. SURVEY OF RECENT GRADUATES AS AN INSTRUMENT FOR ASSISTING IN THE ASSESSMENT OF INSTITUTIONAL EFFECTIVENESS Dr. Teresa Ward Ms. Beth Katz Office of Institutional Research Georgia State University

2. Presentation Outline Conceptual Model Technical Logistics Dissemination of Survey Data Utilization of Data

3. ObjectivesContentQuestions General education learning outcomes critical thinking skills, writing skills, oral presentation skills, research and analytical skills, cooperative learning skills, independent learning skills, technical skills 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 13 Program of study learning outcomes instruction in major program, academic support, scholarship, preparation for Advanced study/career, faculty support 23, 24, 25 Student Engagement interaction with faculty, interaction with other students, social events, campus meetings, athletic events, use Of recreation facilities 6, 17, 27, 28, 29, 30, 31 Student Satisfaction courses, instruction, advisement, Recommendation of program 14, 15, 16, 18, 19, 20, 21, 22, 26 Survey Matrix General Academic Skills – Questions 1-13 Major Area of Study and University Support – Questions 14-26 Engagement – Questions 27-31 Comments

4. Survey Administration Current ProcessOnline Process Time/Expense Level Survey framework (matrix construction) High Material costsHighNone Data integrityMedium to HighHigh Timely dissemination of survey information LowHigh Response RateLow to MediumMedium to High

5. Survey of Recent Graduates Response Rates

6. Please rate the degree of competency you had in the following areas when you entered/graduated from Georgia State University Graph 1 (Entering)Graph 2 (Exiting) *Mean range: 1=very weak competency to 5=very strong competency **p<.001 all pairs Learning Outcomes Communication Technology Collaboration Quantitative Skills Critical Thinking Contemporary Issues

7. Mean 5 4.5 4 3.5 3 English Majors All Others Ability to write clearly and effectively Entering Ability to write clearly and effectively Exiting 3.7 3.9 4.7 4.5

8. English Majors (Seniors) All Others (Seniors) Prepared two or more drafts of a paper or assignment before turning it in 2.632.47 Number of written papers or reports of 20 PAGES OR MORE 1.501.57 Number of written papers or reports BETWEEN 5 AND 19 PAGES 2.932.57 Number of written papers or reports of FEWER THAN 5 PAGES 2.962.87 Institutional contribution: Writing clearly and effectively 3.563.11 2007 NSSE Writing Items

9. Ability to analyze quantitative problems Entering Mean 5 4.5 4 3.5 3 4.3 Ability to analyze quantitative problems Exiting Math Majors All Others 3.9 3.7 4.7 4.3

10. Mathematics Majors (Seniors) All Others (Seniors) Number of problem sets (problem- based homework assignments) that take you MORE than an hour to complete 2.802.43 Number of problem sets (problem- based homework assignments) that take you LESS than an hour to complete 3.002.22 Institutional contribution: Analyzing quantitative problems 3.602.99 Coursework emphasized: MEMORIZING facts, ideas, or methods from your courses and readings so you can repeat them in pretty much the same form 2.832.76 2007 NSSE Quantitative Items

11. Apply scientific reasoning in problem-solving Entering Mean 5 4.5 4 3.5 3 Physics Majors All Others Apply scientific reasoning in problem-solving Exiting 3.7 3.6 4.9 4.3

12. GPA 3.7-4.03.0-3.692.0-2.99 5.00 4.80 4.60 4.40 4.20 4.00 3.80 Mean of communication (exiting) Mean of communication (entering)

13. GPA 3.7-4.03.0-3.692.0-2.99 Mean of quantitative skills (entering) 5.00 4.80 4.60 4.40 4.20 4.00 3.80 3.60 Mean of quantitative skills (exiting)

14. GPA 3.7-4.03.0-3.692.0-2.99 Mean of critical thinking (entering) 5.00 4.80 4.60 4.40 4.20 4.00 3.80 3.83 3.86 3.92 4.47* 4.56 4.63* *p<.01 Mean of critical thinking (exiting)

15. GPA 3.7-4.03.0-3.692.0-2.99 Mean of technology (entering) 5.00 4.80 4.60 4.40 4.20 4.00 3.80 Mean of technology (exiting)

16. *Mean range: 1=poor to 4=excellent Spring 2005 Summer 2005 Fall 2005 Spring 2006 Summer 2006 Fall 2006 Spring 2007 Summer 2007 Fall 2007 Mean Quality of instruction In your major program 3.443.523.433.423.453.463.433.513.45 Scholarly competency Of faculty in your major program3.623.683.583.593.573.613.563.653.61 4 point scale: Poor to Excellent

19. Utilization of Survey Data Department Level –Identify strengths and weaknesses using comparative data –Identify trends and implications of program changes using longitudinal data –Ability to triangulate survey data with other assessment measures –Starting point for conversations

20. University Level –Identify trends and implications of University policy changes using longitudinal data –Ability to triangulate survey data with other assessment measures –Contribute to strategic planning and policy discussions

21. Conclusion High reliability index Focused on indirect measures of learning outcomes Saved OIR a considerable amount of money and time via online administration Increased response rates Facilitated access to the data Contributed to department and University assessment activities

22. Teresa Ward – tward@gsu.edutward@gsu.edu Bethann Katz – bnorthup@gsu.edubnorthup@gsu.edu Office of Institutional Research, Georgia State University – http://www2.gsu.edu/~wwwire/