Just before we get started… Who am I? How questions will be handled Resources available after the webinar What does QILT mean for you? 2

Quality Indicators for Learning and Teaching

4 What does QILT mean for you? We are a private, for-profit company owned by ANU Enterprise Our services include: survey design and execution, qualitative research, survey data management, statistical consulting and analytical reporting. About the Social Research Centre

Overview 1.Where did QILT come from? 2.What are the QILT surveys? 3.How the data is disseminated 4.What you need to do to be ready for QILT What does QILT mean for you? 5

Where did QILT come from? QILT is the outcome of 20 years of work on higher education performance indicators The most recent review suggested the indicators should be:  Fit for purpose  Consistent  Auditable  Transparent  Timely …to provide a robust and reliable measure of teaching performance throughout the Student Life Cycle. What does QILT mean for you? 6

7 The Student Life Cycle

8 What does QILT mean for you? Source: The Department of Education – 2013 UES National Report Entire educational experience rated positively, UES and NSSE, Student satisfaction

9 What does QILT mean for you? Source: The Department of Education – Australian Graduate Survey Employment rates of recent graduates, as a proportion of those available for full-time employment, Graduate employment

Our role in QILT The Social Research Centre was commissioned by the Department of Education as the independent administrator of QILT. This involves:  Collecting data  Reporting on survey outcomes  Creating, monitoring and updating the QILT website. What does QILT mean for you? 10

Where did QILT come from? The Australian Government’s desire to have objective KPIs that demonstrate students’ engagement with and experience of courses, and the quality of those courses from the students’ perspective. What does QILT mean for you? 11

What are the QILT surveys? The QILT survey suite consists of:  The Student Experience Survey (SES), formerly the University Experience Survey (UES)  The Graduate Outcomes Survey (GOS), and  The Employers Satisfaction Survey (ESS) All are cross sectional, point-in-time surveys except the GOS, which is longitudinal. What does QILT mean for you? 12

The Student Experience Survey The University Experience Survey (UES) / Student Experience Survey (SES) involved:  Undergraduate university students  Who are onshore in Australia, and  Commencing their studies or in the final year of their degrees Around 100,000 surveys completed each year Data is collected online. What does QILT mean for you? 13

SES – NUHEI Trial Outcomes The wording of the UES was relevant and appropriate when ‘university’ was replaced with ‘student’ Fifteen NUHEIs participated in a trial of the SES Data collection has now finished and response rates were higher than expected A new approach to incentivisation may have contributed to the enthusiastic response. What does QILT mean for you? 14

The SES in 2015 From 2015 the in-scope population will be expanded to include all  Undergraduate  Onshore  Commencing & completing higher education students Institutions fewer than 200 students will have all of their students included in the survey. What does QILT mean for you? 15

The Student Experience Questionnaire The SEQ was developed in 2011 and 2012 through research, review and consultation It focuses on aspects of the university experience that are  measureable  linked to learning and development outcomes  amendable to institutional influence. What does QILT mean for you? 16

The Student Experience Questionnaire The current SEQ domains include:  Skills Development  Learner Engagement  Teaching Quality  Student Support, and  Learning Resources Institutions can include their own items at the end of the survey on a fee-for-service basis. What does QILT mean for you? 17

Examples of SEQ items DomainItem Skills Development To what extent has your course/program developed your ability to solve complex problems? Learner EngagementTo what extent have you felt prepared for your study? Teaching Quality Overall how would you rate the quality of your entire educational experience this year? Student Support To what extent have you experienced efficient enrolment and admissions processes? Learning Resources Thinking of this year, overall how would you rate the online learning material provided for your course/program? What does QILT mean for you? 18

The Graduate Outcomes Survey The in-scope population is undergraduate and postgraduate higher education students  who completed their qualification in the six months before data collection  as an onshore student Data will be collected online using a similar approach to the SES. What does QILT mean for you? 19

GOS Development Initial feedback has been provide on:  the current use of AGS items and  gaps in national performance indicator data A draft Graduate Outcomes Questionnaire (GOQ) is in development There are a number of ways institutions can participate in the GOS trial (May) and further information will be provided soon. What does QILT mean for you? 20

GOS Development Cognitively testing and a Pilot Test in full dress-rehearsal mode will be conducted in May 2015 Operational data from the GOS Pilot Test will be used to refine the methodology The instrument will be fully validated and recommendations made prior to the first GOS in November 2015 The GOS Plus (the longitudinal component) will commence in What does QILT mean for you? 21

The Employers Satisfaction Survey All supervisors of graduates eligible for the GOS are in-scope for the ESS The 2013 pilot of the ESS highlighted the challenges of contacting graduates’ supervisors The content of the instrument also requires review. What does QILT mean for you? 22

ESS Development Two trials of the ESS are being undertaken:  a methodological trial in conjunction with the October 2014 AGS (currently in analysis)  a content trial (revised Employer Satisfaction Questionnaire – ESQ) in conjunction with the April 2015 AGS Institutions participating in the GOS trial will have the opportunity to take part in the ESS trial. What does QILT mean for you? 23

24 What does QILT mean for you? QILT in the context of the student life cycle

What are the QILT surveys? The three QILT surveys will work together to provide a holistic and coherent picture across the entire student life cycle. What does QILT mean for you? 25

How the data is disseminated There are three main mechanisms that will be used to disseminate the QILT data:  Directly to institutions as: Complete unit record files of your data National unit record benchmarking files  Through national and institutional reports Via the QILT website. What does QILT mean for you? 26

Files available to institutions All participating institutions receive:  A draft data file  A final, coded data file including your institution’s student identifier  A national, anonymised, unit record benchmarking file We work in SPSS and R but also support SAS, STATA and.CSV file formats. What does QILT mean for you? 27

Supporting information Supporting information available to all institutions for each survey includes:  A complete data dictionary  A data map  A copy of the final questionnaire, and  (We hope) A technical summary What does QILT mean for you? 28

Data availability Raw data is available during data collection (but we caution against using this) Draft data – one month after the conclusion of fieldwork Final institutional data – two months after fieldwork has finished Benchmarking files – when the National Report is released. What does QILT mean for you? 29

QILT website QILT website informs student choice through provision of survey results by institution and study area. Website developed in two phases:  Initial release early this year  Developed progressively to August Data quality: Minimum of 25 responses in each cell with confidence intervals. What does QILT mean for you? 30

QILT website The initial indicators will include: SES – Overall Educational Experience item, plus five scale scores CEQ – Overall Satisfaction Indicator, Good Teaching Scale and Generic Skills Scale GDS – employment rates, further study rates, and median salaries What does QILT mean for you? 31

How the data is disseminated You get your own data You get a national benchmarking file The data is publicly available through the QILT website And yes – it’s all free! What does QILT mean for you? 32

What you need to do Lock in the QILT timeline to ‘quarantine’ key dates in your survey calendar Think about whether your admin system is ready to support the QILT survey processes Consider any privacy implications Start early discussions with staff at your institution about QILT and the QILT surveys. What does QILT mean for you? 33

34 What does QILT mean for you? Domain Student Experience Survey August – September Graduate Outcomes Survey April (trial) and October rounds April and October rounds Employer Satisfaction Survey April (instrument trial) and October rounds April and October rounds QILT data collection periods

Your admin data Make sure you have your Submission 1 file for HEIMS is clean and complete! Before each round of data collection (one SES and two GOS’) we will contact you to:  Confirm the students selected for the survey via HEIMS  Exclude students that have deferred or withdrawn  Append student contact information. What does QILT mean for you? 35

Consider any privacy implications Enrollment forms should inform students they will be asked to participate in surveys Restrictions that could affect the release of personal information about students need to be resolved Potential barriers can be overcome via an ‘opt-out’ process. What does QILT mean for you? 36

Start talking about QILT Who needs to be briefed about QILT at your institution? Do you need to involve your marketing people? Do you have a communications plan, particularly for the SES? How can faculty members help reinforce the importance of QILT to students? What does QILT mean for you? 37

Ways we keep you informed A fortnightly newsletter highlights important dates and activities Webinars will be held each month in 2015 Summaries of key activities are available prior to the release of final reports Opportunities for feedback are available throughout the process. What does QILT mean for you? 38

Working with the QILT team We are always available to  answer questions (no matter how vague)  provide advice about research methodology or data collection  explain technical aspects of the QILT surveys or the data files  assist with the inclusion of additional questions or populations If you have been liaising with a specific person, please contact them directly, otherwise, will always work. What does QILT mean for you? 39

What you need to do Make a tidy and complete submission to HEIMS by May Get the message out to students and faculty, so you can hit the ground the running in August. What does QILT mean for you? 40

To summarise… You now have a way to objectively demonstrate the quality of your programs The three QILT surveys work together to provide a holistic picture of the student life cycle You get your own data, you get a national benchmarking file and the data is publicly available through the QILT website Submit your data by May and start to get the QILT message out to students and faculty members. What does QILT mean for you? 41

Finally… Putting your submission in to HEIMS will give you an opportunity fully participate in QILT and to position your organisation nationally and internationally. QILT data provides you with objective evidence to demonstrate to prospective and current students that your courses provide quality learning and teaching outcomes. What does QILT mean for you?