2. Just before we get started… Who are we? How questions will be handled Resources available after the webinar Key QILT Dates & the AGS Item Review 2

3. Employer Satisfaction Survey Quality Indicators for Learning and Teaching

4. Overview 1.Background to the ESS 2.ESS approach and methodology 3.Employer Satisfaction Questionnaire (ESQ) 4.ESS trial and materials ESS 4

5. Background to the ESS The ESS Pilot Project was undertaken in 2013 by the Workplace Research Centre at the University of Sydney. The ESS Pilot team: Developed a conceptual foundation for the survey, Designed a sampling approach, Created and tested a draft survey instrument, Conducted telephone interviewing with graduates and supervisors ESS 5

6. Findings from the ESS Pilot All respondents were very positive about graduate skills Teamwork and interpersonal skills, foundation skills, and adaptive skills were the most highly rated. A number of methodological challenges were identified: Graduates were more reluctant to provide their supervisors’ details than originally anticipated. The methodology was time intensive and costly. Robust results were difficult to produce and will be challenging even with a census. The validity of any inferences is likely to be affected by apparent non-referral bias. ESS 6

7. ESS Pilot recommendations That the ESS methodology be further developed be operated as an adjunct to graduate surveys be based on a census of all respondents to the graduate survey take place between six and twelve months after graduation be conducted as a telephone survey of supervisors questionnaire items and clusters be further investigated ESS 7

8. Background to the ESS The ESS was developed by researchers at the University of Sydney and based on previous surveys of graduate employers. The ESS trial builds on the learning from the pilot test. ESS 8

9. ESS methodology - scope Consistent with the recommendations from the pilot: The ESS will be conducted as an adjunct to the GOS All employed GOS respondents will be asked to provide contact information Not restricted to specific disciplines Postgraduate students are also included Supervisors from all industry areas are in-scope ESS 9

10. ESS timing The pilot recommended surveying 6 to 12 months after graduation Deferring the ESS until after the GOS would Decrease the number of potential respondents - graduates are difficult to contact at 4 months Be costly, requiring that graduates be recontacted to determine current employment status Require graduates to complete an additional survey, as was the case for the pilot ESS 10

11. ESS timing (2) Some graduates many not have had sufficient time to find employment To address this During the annual panel maintenance for the GOS – longitudinal, employment status will be reassessed If employment status has changed (from unemployed to employed) the GOS will be readministered with the ESS If the graduate has changed jobs, the GOS will be readministered with the ESS ESS 11

12. ESS methodology The ESS relies on supervisor contact information provided by recent graduates The methodological trial in late 2014 demonstrated that: Graduates are willing to provide contact details via email Shorter explanations and emails are more effective Different graduates prefer to be approached in different ways ESS 12

13. ESS trial methodology All graduates responding to the GOS trial will be asked to provide supervisor contact information Invitation and explanatory text will be tailored – personalised and depersonalised versions Graduate supervisors will be invited to participate via email two business days after the GOS has been completed Supervisors will be reminded via email twice and then they will be followed-up via telephone ESS 13

14. ESS trial methodology (2) The primary mode will still be telephone but employers have the option of online. Supervisors who do not respond to the initial email invitation or two subsequent reminders will be telephoned. Potential respondents can opt out at any time via email or the 1800 number. ESS 14

15. ESS approach & methodology The methodological approach developed for the ESS pilot has been used as a foundation to ‘join up’ the GOS and the ESS. ESS 15

16. ESQ – 2013 pilot test outcomes The overall rating and the technical skills items performed well. ESQ needs further investigation and development. The items were mainly based on previous employer surveys conducted at single universities. Factor analysis results for some clusters were good (teamwork, disciplinary skills), results for others were less satisfactory (enterprise skills, employability skills, and adaptive skills). ESS 16

17. ESQ – 2013 pilot test outcomes (2) The factor analysis demonstrated items in many of the clusters could be improved, particularly foundation skills adaptive skills and employability skills. Cognitive testing of the clusters and individual items would be beneficial. ESS 17

18. ESQ – Graduate Attributes Scale The pilot recommendations and additional classical test theory and item response theory analysis informed the content review. If all of the clusters are treated as a scale (Graduate Attributes Scale – GAS) the psychometric properties are relatively sound. The item sub-scales or clusters are not robust and fail to meet basic thresholds for reliability. ESS 18

19. ESQ – GAS sub-scales The original sub-scales including the following clusters: Teamwork and interpersonal skills Foundation skills Technical and domain specific skills Employability skills Adaptive skills and attributes, and Enterprise skills ESS 19

20. ESQ – GAS sub-scales (2) To improve the properties of the sub-scales Double barrelled items were simplified Problematic items were reworded Items with greater ‘difficulty’ were included, and Clusters were combined The number of overall items increased and the number of subscales decreased. The overall scale, sub-scales and individual items will be reviewed after the pilot. ESS 20

21. ESQ – GAS sub-scales (3) The item stem for the scale has been simplified to decrease cognitive load ORIGINAL For each skill or attribute, to what extent do you think ’s from prepared or did not prepare them for their current role at the level your workplace requires from a recent graduate? REVISED For each skill or attribute, to what extent do you agree or disagree that ’s from prepared them for their job? Analysis of the rating scale suggested modifications and different versions will be trialled ESS 21

22. Employer satisfaction questionnaire The ESQ has been reviewed and revised based on the recommendations from the 2013 Pilot Test and additional psychometric testing. ESS 22

23. ESS – trial Graduates from the GOS trial institutions will be asked to provide contact information for their supervisors All discipline areas are in-scope All graduates are in-scope Fieldwork will be conducted in parallel with the GOS, extending into December for the telephone interviews ESS 23

24. ESS - next steps The revised ESS methodology and ESQ will be trialled with the GOS ESS operational and content data will be reviewed Psychometric analysis of the GAS will be undertaken If required: The ESS methodology will be refined The ESQ and/or the GAS will be modified ESS 24

25. ESS materials ESS materials available after the webinar include: A revised ESQ A summary of the ESQ content review (and implications for the ESS trial) A recording of this webinar http://www.srcentre.com.au/QILT_Webinar http://www.srcentre.com.au/QILT_Webinar The 2013 ESS Pilot Test report is available from https://education.gov.au/employer-satisfaction- survey https://education.gov.au/employer-satisfaction- survey ESS 25

26. Ways we keep you informed The fortnightly newsletter highlights important dates and activities The QILT calendar contains all operational deadlines Webinars are held each month – recordings are available Summaries of key activities are available prior to the release of final reports Fact sheets are available on key issues The (temporary) QILT resources page is URL http://www.srcentre.com.au/qilt-resources http://www.srcentre.com.au/qilt-resources ESS 26

27. Working with the QILT team Opportunities for feedback are available throughout the process but you are welcome to comment at any time. We are always available to  answer questions (no matter how vague)  provide advice about research methodology or data collection  explain technical aspects of the QILT surveys or the data files  assist with the inclusion of additional questions or populations If you have been liaising with a specific person, please contact them directly, otherwise, qilt@srcentre.com.au is the central contact point. QILT Overview 27

28. To summarise… The foundation for the ESS content and methodology was developed by the Workplace Research Centre at the University of Sydney. The methodology has been refined to increase the number of potential employer respondents. The instrument has been modified to improve the properties of the sub-scales. Findings from the trial will inform the November ESS. ESS 28

29. Thank you! Questions? ESS 29