1. Lessons in Implementing a Post- Graduation and Learning-Gains Questionnaire Sara Lazenby sllazenby@wisc.edu University of Wisconsin - Madison May 21, 2013 AIR Annual Forum

2. Why implement an institutional exit questionnaire? Prior to 2008, UW-Madison had no institutional exit questionnaire for Bachelor’s degree recipients The need to implement a questionnaire arose due to: – Voluntary System of Accountability (VSA) question on future plans – Higher Education Opportunity Act of 2008 – Essential Learning Outcomes adopted in 2008 – Priority of campus leadership

3. What is the Voluntary System of Accountability? A voluntary initiative developed by the higher education community to meet the following objectives: – Demonstrate accountability and stewardship to the public – Support institutions in the measurement of educational outcomes and facilitate the implementation of effective practices as part of institutional improvement efforts. – Assemble and disseminate information that is transparent, comparable, and understandable – Provide a useful too for students during the college search process Sponsored by the Association of Public and Land- grant Universities and the American Association of State Colleges and Universities More information available at: http://www.voluntarysystem.org http://www.voluntarysystem.org Viewable on the College Portrait: http://www.collegeportraits.org/ http://www.collegeportraits.org/

4. Developing Criteria for an Exit Questionnaire – Phase 1 Include the required VSA question Create a cost-effective and sustainable questionnaire Integrated into the student record Census approach has potential for high response rates Reporting considerations

5. Importance of the questionnaire design Developer coded to be part of our Student Portal As part of the Student Record, student responses are protected by FERPA Placement in ISIS/PeopleSoft means the questionnaire is easily connected to other student information No need to ask additional questions for demographic/academic data

6. -Module (annoyingly) stays live in student’s MyUW portal until student responds -During the questionnaire period, the data is extracted into an InfoAccess data warehouse table and refreshed nightly. This is available to anyone who also has access to the relevant tables -12 Days Questionnaire goes live End of Term (December and May) +21 Days -11 Days: E-mail invitation goes out -4 Days: Reminder e-mail sent Questionnaire deactivated Questionnaire Timeline

7. Many contributions to this project’s success: Questionnaire Development Voluntary System of Accountability AAU Data Exchange Senior Exit Survey Team NSSE University of Illinois UW-Madison Convergence Team AACU’s LEAP UW-Madison Assessment Fund John Stevenson, UW Survey Center Jennifer Dykema, UW Survey Center Jeremy Beach, UW Survey Center Student participants in focus groups Questionnaire Development Voluntary System of Accountability AAU Data Exchange Senior Exit Survey Team NSSE University of Illinois UW-Madison Convergence Team AACU’s LEAP UW-Madison Assessment Fund John Stevenson, UW Survey Center Jennifer Dykema, UW Survey Center Jeremy Beach, UW Survey Center Student participants in focus groups ELOs and Post-Grad Plans Technical Development and Implementation Connie Chapman, Registrar’s Office Michael Fay, DoIT James Helwig, DoIT Rob Lauer, DoIT Wanjiru Pontes, DoIT Stephan Wahe, DoIT Bob Mayville, DoIT Betty McIlwee, DoIT Jim Steele, Registrar’s Office Kay Steiner, InfoAccess, DoIT Technical Development and Implementation Connie Chapman, Registrar’s Office Michael Fay, DoIT James Helwig, DoIT Rob Lauer, DoIT Wanjiru Pontes, DoIT Stephan Wahe, DoIT Bob Mayville, DoIT Betty McIlwee, DoIT Jim Steele, Registrar’s Office Kay Steiner, InfoAccess, DoIT

8. Example of Portal Appearance

9. Examples of the Questionnaire

10. Examples of the Questionnaire – Required VSA Question

11. Examples of the Questionnaire

12. Early Lessons Learned Quick resolutions needed for technical issues Varying response rates by question Some questions did not produce usable data – Salary – Workplace/Next Institution

13. Developing Criteria for an Exit Questionnaire – Phase 2 Need for indirect measures of the Essential Learning Outcomes Questionnaire fatigue considerations Removal of questions that weren’t useful

14. http://www.provost.wisc.edu/content/WI_Exp_ELOs.pdf

15. -Indirect measures of learning, student perceptions - ELOs, UI-UC, NSSE were all influences -UW Survey Center did review and pre- testing, including layout and syntax - Added questions to the post- graduation plans questionnaire in December 2010 and into the future Question Development

18. What are our response rates?

19. Response Rates – Effects of Reminder E-mails December 2012 – Number of responses by day. May 2012 – Number of responses by day.

20. Communicating Findings Time lag before degrees are posted Challenge: Displaying the results in a way that campus can easily understand. What do the results mean?

21. Analyzing and Disseminating data Campus Wide Schools/Colleges Individual Majors for Program Review Disciplinary “Divisions” – Arts and Humanities – Biological Sciences – Physical Sciences – Social Studies

22. Post Graduation Plans – ALL GRADUATES Fall 2011 and Spring 2012 Bachelor’s Degree Recipients Other includes: graduate school part-time, additional coursework, military service, volunteering, and unspecified activities. Based on 3083 responses (54% response rate) If employment selected, which best describes your plans at this time? Based on 2,689 responses (47% response rate) Work Full-Time Work Part-Time Graduate School Full-Time Other What is most likely to be your principal activity/activities after graduation? (Unduplicated) Have accepted a position Considering one or more offers Currently searching for a position Will begin searching for a position after graduation Do you plan to work in Wisconsin after graduation? Yes Undecided No Do you have a valid passport? Yes, from the United States Yes, from another country No SL, Academic Planning and Institutional Research, 10/12Source: Post Graduation Plans Questionnaire Responses) Based on 1853responses (87% response rate among those with work as their primary activity)

23. Post Graduation Plans – Campus Summary Trend in VSA question responses over time What is most likely to be your principal activity/activities after graduation? (Unduplicated)

24. Eighteen questions that provide a self- assessment at graduation, that maps to the essential learning outcomes at UW-Madison. Respondents asked to assess their ability upon entering UW-Madison and at graduation. Results are tabulated as the non-null average for each question Post Graduation Plans – Essential Learning Outcomes

25. All this data, now how do we communicate findings? Display Challenges – Need something visual to help people interpret the data Understanding differences – A lower learning gain on a particular metric wasn’t always “negative” – Putting learning gains in the context of our general education requirements – Differences in learning outcomes match our expectations given major content.

26. SL, Academic Planning and Institutional Research, 10/12 Scale: 1 –Not at all, 2 –A little, 3 –Somewhat, 4 –Very, 5 –Extremely Responses sorted clockwise from the question with the highest response score at graduation for all UW-Madison students. Based on 2,828 to 2,886 responses, depending on question. Post Graduation Plans – ALL GRADUATES “How able were/are you to …”

27. Post Graduation Plans – Biological Sciences Division 11-12 “How able were/are you to …” SL, Academic Planning and Analysis, 8/11 Scale: 1 –Not at all, 2 –A little, 3 –Somewhat, 4 –Very, 5 –Extremely Responses sorted clockwise from the question with the highest response score at graduation for all UW-Madison students. Based on 423 to 431 responses, depending on question.

28. Post Graduation Plans – A Biological Science Major “How able were/are you to …” SL, Academic Planning and Institutional Research, 10/12 Scale: 1 –Not at all, 2 –A little, 3 –Somewhat, 4 –Very, 5 –Extremely Responses sorted clockwise from the question with the highest response score at graduation for all UW-Madison students. Based on 104 to 107 responses, depending on question.

29. Post Graduation Plans – Arts and Humanities Division 11-12 “How able were/are you to …” SL, Academic Planning and Institutional Research, 10/12 Scale: 1 –Not at all, 2 –A little, 3 –Somewhat, 4 –Very, 5 –Extremely Responses sorted clockwise from the question with the highest response score at graduation for all UW-Madison students. Based on 398 to 407 responses, depending on question.

30. Post Graduation Plans – An Arts and Humanities Major “How able were/are you to …” SL, Academic Planning and Institutional Research, 10/12 Scale: 1 –Not at all, 2 –A little, 3 –Somewhat, 4 –Very, 5 –Extremely Responses sorted clockwise from the question with the highest response score at graduation for all UW-Madison students. Based on 63 to 66 responses, depending on question.

31. Post Graduation Plans – Physical Sciences Division 2011-12 “How able were/are you to …” SL, Academic Planning and Institutional Research, 10/12 Scale: 1 –Not at all, 2 –A little, 3 –Somewhat, 4 –Very, 5 –Extremely Responses sorted clockwise from the question with the highest response score at graduation for all UW-Madison students. Based on 366 to 375 responses, depending on question.

32. Post Graduation Plans – A Physical Science Major “How able were/are you to …” SL, Academic Planning and Institutional Research, 10/12 Scale: 1 –Not at all, 2 –A little, 3 –Somewhat, 4 –Very, 5 –Extremely Responses sorted clockwise from the question with the highest response score at graduation for all UW-Madison students. Based on 71 to 73 responses, depending on question.

33. Post Graduation Plans – Social Studies Division 11-12 “How able were/are you to …” SL, Academic Planning and Institutional Research, 10/12 Scale: 1 –Not at all, 2 –A little, 3 –Somewhat, 4 –Very, 5 –Extremely Responses sorted clockwise from the question with the highest response score at graduation for all UW-Madison students. Based on 883 to 903 responses, depending on question.

34. Post Graduation Plans – A Social Studies Major “How able were/are you to …” SL, Academic Planning and Institutional Research, 10/12 Scale: 1 –Not at all, 2 –A little, 3 –Somewhat, 4 –Very, 5 –Extremely Responses sorted clockwise from the question with the highest response score at graduation for all UW-Madison students. Based on 112 to 115 responses, depending on question.

35. Majors with at least 50 responses to question about plans after graduation Biochemistry Biology Business: Accounting Business: Finance, Investment, and Banking Business: Management and Human Resources Business: Marketing Business: Risk Management and Insurance Civil Engineering Communication Arts Computer Sciences Economics English Environmental Studies History International Studies Journalism Mathematics Mechanical Engineering Nursing Political Science Psychology Sociology Spanish Zoology

36. Lessons Learned Have worked with campus partners to try and increase response rates Reminder e-mails! Overlap with “exit surveys” within School/College and Departments Low cost, not a lot of opportunity for changes Stability of questionnaire has advantages Popularity of handheld/tablet devices impact response rates

37. Opportunities and Challenges Going forward … Opportunities to make part of the graduation process Changes in regulations may necessitate a different approach Continued collaboration with other units, trying to help them understand the value of the questionnaire

38. Involved a lot of creative people with a wide range of skills and experience. Inspiration from our peers/competitors. Made use of our existing enterprise systems ISIS/PeopleSoft MyUW student portal InfoAccess (data warehouse) Census approach has potential for high response rate. Cost-effective, sustainable collection and reporting system. So far, it’s working!

39. Questions and Discussion sllazenby@wisc.edu https://apir.wisc.edu/pgp.htm sllazenby@wisc.edu