1. Institut für Forschungsinformation und Qualitätssicherung Graduate surveys as an outcome evaluation Presentation EAIR Forum 2009, July 23-26, Vilnius, Lithuania Dr. René Krempkow Institute for Research Information and Quality Assurance, Bonn (Germany) (former project leader of Freiburg Graduate Surveys, University of Freiburg) Institut für Forschungsinformation und Qualitätssicherung

2. iFQ Outline 1.Relevance of graduate surveys for Quality Management 2.Measuring outcomes of higher education – Influencing factors of professional success 3.Exemplary results 4.Interpretation of results 5.Prospects for analysis

3. Institut für Forschungsinformation und Qualitätssicherung Graduate Surveys

4. Institut für Forschungsinformation und Qualitätssicherung iFQ When integrating graduate surveys in quality management: 1.Adequate interpretation of results only through comparison with other HEIs and more general national data (see Teichler 2003) 2.Account for individual characteristics through institution-specific and programme-specific surveys (see QM-literatur)  Nationwide “Core Questionnaire” with institution- /programme-specific adaptations The Potential of graduate surveys for Quality Management I

5. Institut für Forschungsinformation und Qualitätssicherung iFQ Analyse data to identify determinants of professional success to feed back to Quality Management system Unit of observation: study programmes Account for specific context by interpreting data taking into account influencing factors (e.g. regional economy) Account for specific initial conditions, e.g. % of student parents, socioeconomic background of students (Schomburg/Teichler, 1998) The Potential of graduate surveys for Quality Management II

6. Institut für Forschungsinformation und Qualitätssicherung Source: Schomburg/ Teichler (1998: 165) Two HEI stand out positively To what degree can differences be attributed to different starting conditions of HEIs? This will be discussed on the basis of Freiburg Graduate Surveys and Dresden Graduate Surveys. The Potential of graduate surveys for Quality Management III

7. Institut für Forschungsinformation und Qualitätssicherung iFQ Measuring Outcome with graduate surveys – what does professionell success depend on? Multivariate analysis allows identifying factors influencing professionell success This information is independant from (subjective) self-assessment of graduates (see Krempkow/Pastohr 2006) Example: job experience, study times abroad and specific competences appear to be more important than study duration and grades – That leads to the question: Can and (if yes) What can HEIs do to improve professional success of graduate? Multivariate (correlatory) analysis allows analysing relationships between professional success and aspects of study, including starting conditions and context!

8. Institut für Forschungsinformation und Qualitätssicherung iFQ Influencing factors on professional success Objective and subjective criteria of professional success as indicators of HE outcomes (Teichler/Schomburg (1997: 248) 1.Objective indicators for professional success (e.g. income, hierarchical position) 2.Subjective indicators for professional success (e.g. job satisfaction, autonomy) 3.Objective indicators for transition HE-labour market (e.g. Time of job hunt) 4.Subjective assessment of goodness-of-fit HE – job (e.g. usefulness of study contents, employment adequacy). Freiburg and Dresden Survey: Gross income, job satisfaction and employment adequacy fitting criteria for professional success.

9. Institut für Forschungsinformation und Qualitätssicherung Basic model of influencing factors on professional success INDIVIDUAL CHARACTERIS- TICS + SOCIO- ECONOMIC BACKGROUND CHARACTERIST CS OF HIGHER EDUCATION (PROGRAM) LEARNING OUTCOMES (COMPETENCES) JOB SEARCH, TRANSITION PHASE, JOB ENTRY INDIVIDUAL STATE AT TIME OF SURVEY JOB SEARCH, TRANSITION PHASE, JOB ENTRY PROFESSIONAL SUCCESS / PROFESSIONAL SUCCESS One time study: First repetition of study: 1. threshold 2. threshold

10. Institut für Forschungsinformation und Qualitätssicherung Example of results: stand. Regression coefficients for income Regression coefficients for monthly income of current occupation Technical studies R 2 =.41 adj. R 2 =.00 n=105 Medical studies R 2 =.25* adj. R 2 =.11* n=270 Social sciences R 2 =.20* adj. R 2 =.08* n=316 Grade of matriculation standard,179,223 + -,203* Gender (male=1) -,320 + -,020-,100 Finished vocational training before studying (yes=1),317 +,051,114 Duration of stay abroad,182,116,031 Part time job during semester break,093-,141,167 Part time job during lecture period -,063,104-,130 Availability of program specific profiles,053,111,057 Social and communication skills,066,101-,146 + Duration of study -,422*-,004-,125 Job entrance age -,028,266 + Final degree of study -,165-,012,052 Foreign language skills -,025,080,106 Proficiency in own subject,268,010,127 Ability to develop new ideas and problem solutions -,116-,115-,110 Ability to work efficiently towards a goal -,236,066-,089 Ability to critically rethink own ideas and ideas of others,181-,042-,058 Use of personal contacts for job search,063-,323**,011 Employers searched for graduates with other study focus -,043--,076 Importance of family -,119-,004,101 Importance of career,215-,209*-,270** Figure 3: Regression coefficients for the monthly income of current occupation; + / */ ** = significant on the 10%-/ 5%-/ 1%-level ( “ - “ = variable was not included in the explanatory model)

11. Institut für Forschungsinformation und Qualitätssicherung Example of results: stand. Regression coefficients for income Regression coefficients for monthly income of current occupation Technical studies R 2 =.41 n=105 Medical studies R 2 =.25* n=270 Social sciences R 2 =.20* n=316 Grade of matriculation standard,179,223 + -,203* Gender (male=1) -,320 + -,020-,100 Social and communication skills,066,101-,146 + Duration of study -,422*-,004-,125 Job entrance age -,028,266 + Final degree of study -,165-,012,052 Use of personal contacts for job search,063-,323**,011 Importance of family -,119-,004,101 Importance of career,215-,209*-,270** Figure 3: Regression coefficients for the monthly income of current occupation; + / */ ** = significant on the 10%-/ 5%-/ 1%-level ( “ - “ = variable was not included in the explanatory model) Similar Results in Dresden Graduate Survey with the same analysis scheme, but the social background was more important

12. Institut für Forschungsinformation und Qualitätssicherung iFQ Interpretation of results (Freiburg & Dresden) Influencing income : Previous job training, GPA (Secondary school) and gender, in Medicine: Personal contacts and career oriantation are important Influencing job satisfaction : Technical sciences: Socioeconomic background, previous job training Engineering: Tendendy for women to be less satisfied Humanities: Higher self-assessed expertise and family orientation: lower job satisfaction Influencing job adequacy: Engineering: gender - women have less adequate jobs Prospects for analysis?

13. Institut für Forschungsinformation und Qualitätssicherung iFQ Further information IfQ Bonn: www.research-information.dewww.research-information.de Freiburg Graduate Surveys: www.qm.uni-freiburg.de/projektewww.qm.uni-freiburg.de/projekte Requirements of graduate surveys for measuring HE outcomes: „Leistungsbewertung, Leistungsanreize und die Qualität der Hochschullehre“ (2007) www.universitätsverlagwebler.de/krempkow.html, Onlinepublikation (2005) http://nbn- resolving.de/urn:nbn:de:swb:14-1129208825969-55860 www.universitätsverlagwebler.de/krempkow.htmlhttp://nbn- resolving.de/urn:nbn:de:swb:14-1129208825969-55860 Determinants of professional success: Zeitschrift für Evaluation 1/2006 (www.zfev.de), S. 7-37, bzw. http://www.kfbh.de/downloads/Was_macht_Hochschulabsolven ten_erfolgreich.pdfwww.zfev.de http://www.kfbh.de/downloads/Was_macht_Hochschulabsolven ten_erfolgreich.pdf Dresden Graduate Survey: www.kfbh.de/absolventenstudiewww.kfbh.de/absolventenstudie

14. Institut für Forschungsinformation und Qualitätssicherung Example of results: stand. Regression coefficients

15. Institut für Forschungsinformation und Qualitätssicherung 2nd Example: Job Adequacy