1. MAXIMISING RESPONSE RATES GHEETA KRISHNAN, UTAS BRETT RYAN, GRIFFITH ALBERTO MENDEZ, UTS

2. Getting the Results You Want: A Response Rate Strategy

3. Student Evaluation, Review and Reporting Unit (SERRU) Contents 1. SERRU’s role 2. Survey Implementation 3. What we achieved in response rates –2 snapshots 4. How we improve our methods. 5. How we increase our response rates. 6. How we impress upon our stakeholders.

4. SERRU supports the University to……  Develop a strong evaluative culture to improve the student and staff experience at the University of Tasmania  Establish strategic relationships with a range of stakeholders to ensure quality processes and practices are maintained and improved across the University  Provide staff support on a range of data needs, priorities and projects  Collaborate with people internally, nationally and internationally to build partnerships in the quality of teaching and learning, standards and review

5. What we do at SERRU with our data skill sets….  Survey implementation and analysis  Course and unit data evaluation  Data analysis  Research reporting  Benchmarking  Standards  Policy development  Project management

6. Survey Implementation Develop a strong evaluative culture to improve the student and staff experience at the University of Tasmania  External Surveys  Australian Graduate Survey (AGS)  University Experience Survey (UES)  International Student Barometer (ISB)  Ad hoc national surveys  Internal Survey  eVALUate - the UTAS online student feedback system on units and teaching

7. Australian Graduate Survey - Timeline of Response Rates 2009 – 2013

8. How we compare nationally….

9. Building awareness and engaging students and staff through…  Workshops with Schools and Faculties  Student Advisers’ Newsletter  Student Union Representative  Social Media Engagement  MyLo – Online Student Learning Environment  Electronic Notice Boards

10. Building awareness and engaging students and staff through…  Posters around high traffic areas around campus  Flyers distributed at Graduation Ceremonies  Alumni Newsletter  Bulletin on the Graduation Webpage  Presence on SERRU External Surveys and eVALUate webpages with links to the respective survey sites

11. Maximizing response rate potential through…..  Initial email and letter invitation from relevant University management  Telephone Reminder of 300 calls per day at commencement of survey  Follow up telephone reminder towards the end of the survey round  2 to 3 SMS reminders per round  2 email reminders from Survey Manager  Regular response rate updates to Student Advisers  Offering an incentive  Using the correct terminology e.g. International Student Barometer Survey (for international students)  Maintaining a Survey Register

12. Communicating results to stakeholders……  Council  Senior Management Team (SMT)  Schools and Faculties  Division of Students and Education (Divisional Management Team)  Students

13. For more information on the University of Tasmania’s response rate strategy Please contact: Gheeta Chandra Krishnan External Surveys Coordinator Student Evaluation, Review and Reporting Unit (SERRU) Gheeta.Krishnan@utas.edu.au Gheeta.Krishnan@utas.edu.au http://www.utas.edu.au/student- evaluation-review-and-reporting- unit/

14. Brett Ryan Survey Manager - Griffith University

15. The Australian Graduate Survey captures the thoughts, opinions and outcomes of recent graduates from all Australian universities. Your valuable feedback is completely CONFIDENTIAL and it will: -inform future students on study, career, & employment outcomes -assist the federal government with higher education policy development -inform program improvements here at Griffith -qualify you to receive an engraved Griffith commemorative pen -and enter you into a draw to win other great prizes too! Look out for an email with your personal survey link in November 2014.

16. Email Prep Spending time before you launch is critical Email hierarchy – D&A, PS update, home, business, uni account Cleansing the list: –Check priority 1s Is it a GU account? Are they a continuing student or a staff member? Is it an international student’s education agent? –Check for duplicates –Search for domain names

18. Other emails…. Heads of School Pro Vice Chancellor (International) Deans (Learning & Teaching) Survey Manager –Partial completers –Staff non-responders –Prize draw deadline looming We use multiple senders, with multiple messages.

19. SMS & Paper SMS Postal surveys are expensive so it is important to have good data –We use a similar process as to how we attack the email file International students receive two pieces of Direct Mail; one to their Australian address and one overseas We use a different PURL for people who complete the survey online as a direct result of receiving the letter (to track its effectiveness) And we also use this….

21. The Final Push Telephoning fieldwork –Centralised Telephone Service via GCA The final, final push (read into panic) –the next 200 responses receive a guaranteed… And if you are really desperate…. –Guaranteed offer extended! or –Try again on the phones yourself….but you can only do the GDS.

22. Brett Ryan 07 3735 4394 b.ryan@griffith.edu.au

23. AGS Collection at UTS Graduations Alberto Mendez Survey Coordinator Planning and Quality Unit University of Technology, Sydney

24. The typical AGS situation In AGS 2008 only achieved 48% response rate …  thus unable to publish data outside UTS! In AGS 2009 and 2010 achieved 51%, 52% but …  required huge effort to achieve target (printing and posting 2 or more sets of paper forms)  not able to drill down very far into data Must be an easier way …

25. The rather fortuitous UTS solution AGS commencement dates align well with Graduations:  AGS = April 1 and October 1  UTS Graduations = start of May and start of October Main difficulties:  reluctance from Graduations Office to fit us in  possibly disrupt existing process, delay ceremonies Trial run agreed on, successful*

26. Collection mode structure Thus UTS able to increase AGS collection modes:  hardcopy (postal + Graduations) and online  telephone interviews also briefly adopted, since discontinued So, current order of modes: OctoberApril 1. hardcopy (Graduations)1. online 2. hardcopy (postal)2. hardcopy (Graduations) 3. online3. hardcopy (postal) 4. telephone

27. In the lead up to Graduations Preparation work:  request graduate list from Student Systems  prepare email communications (push incentives)  compile a “mark-off” spreadsheet and assign each graduate their oAGSID  print individualised hardcopy forms and letters and stuff into addressed postal envelopes  align graduates and ceremonies:  sort hardcopy forms into individual ceremony’s box  prepare list of graduates for individual ceremony and mark- off online completions (April only)

28. At Graduations UTS setup:  component of pre-ceremony official process (1 hour):  registration  gowning  AGS  Alumni  AGS physical space:  5 tables, each with 6 chairs  AGS staff:  2 people, spruiker (promoter, welcomer) and form provider  AGS completion time:  typically 5-10 minutes (few if any blanks, incompletes) Graduates + AGS + chocolate = happiness!

30. Typical numbers OctoberApril Total graduates 3,2006,100 Graduation ceremonies 1020 AGS completions … online (before and during Graduations) 02,000 hardcopy (at Graduations) 1,3001,500 hardcopy (after Graduations, postal) 100 online (after Graduations) 300100 AGS response rate 53%61% Chocolates (regular size, 50-60g) 1,0002,500

31. Conclusions  Now seamlessly part of pre-ceremony process, graduates go with the flow, many see it as a ‘must do’  They don’t do it for chocolate but appreciate the gesture  Small number of latecomers with no time left to complete AGS, asked to take AGS and return it in post  Small but annoying features of setup:  relatives/friends wanting to spend time with them (sometimes hard to keep them out of AGS area, seats)  pen stealing

32. Final thoughts  Great mode for maximising responses but requires considerable time investment from dedicated staff – use of casual, un-invested staff can reduce effectiveness  Graduates overwhelmingly happy, good time to target them for participation, but possible mode effect on CEQ scores?

33. Response mode effect on CEQ scales