1.  Data file preparation  Item and scale scores  Missing data  Reporting the CEQ  Change in 2010  CEQ Q&A

2.  Course experience perceptions of graduates who completed coursework degrees ◦ Research degree graduates respond to the PREQ  Feedback on up to two majors ◦ More responses than respondents  Eleven scales underpinned by 49 Likert-type items ◦ Three core scales consisting of 13 items

3.  Good Teaching Scale (GSS) [6]  Generic Skills Scale (GSS) [6]  Overall Satisfaction Item (OSI) [1]  Clear Goals and Standards Scale (CGS) [4]  Appropriate Workload Scale (AWS) [4]  Appropriate Assessment Scale (AAS) [3]  Intellectual Motivation Scale (IMS) [4]  Student Support Scale (SSS) [5]  Graduate Qualities Scale (GQS) [6]  Learning Resources Scale (LRS) [5]  Learning Community Scale (LCS) [5]

4.  Remove PREQ cases  Remove cases with no CEQMAJ ◦ Imputation possible from MAJ1 and MAJ2  Remove cases with no LEVEL  Remove cases that do not fulfil these conditions: ◦ valid response to OSI, or ◦ at least four GTS item responses, or ◦ at least four GSS item responses  All collection methods retained for 2010 CEQ

5.  113,523 respondents  131,603 responses GDSCEQ1CEQ2

6.  Common five-point response scale ◦ 1 = strongly disagree ◦ 2 = disagree ◦ 3 = neither agree nor disagree ◦ 4 = agree ◦ 5 = strongly agree  Some items reverse coded  CEQ Reporting metrics: ◦ 1 = -100; 2 = -50; 3 = 0; 4 = 50; 5 = 100 (CEQ) ◦ 1 = 0; 2 = 0; 3 = 0; 4 = 100; 5 = 100 (PA) ◦ 1 = 0; 2 = 0; 3 = 100; 4 = 100; 5 = 100 (BA)

7.  Mean of item scores  Minimum item scores: ◦ 1 for OSI ◦ 2 for AAS ◦ 3 for AWS, CGS, IMS ◦ 4 for GQS, GSS, GTS, LCS, LRS, SSS  Item scores removed if scale score not computed  CEQ scores are normally distributed

8.  Planned: ◦ Optional scale not included ◦ From < 0.1% to 1.8% of responses  Unplanned: ◦ Item non-response

9.  START ◦ Resource library  Data files  CEQ  2010

10.  START ◦ Resource library  Reports  2010

11.  START ◦ Resource library  Data files  CEQ  2010 ◦ Save target as… ◦ Save link as…

15.  Concern that graduates were misreading the direction of the response scale ◦ Positive comments accompanying negative scores It was awesome!!!

16.  Endpoint-only labels 1992-2009  All points labelled 2010- following resolution by SRG at July 2009 meeting

18.  Yearly changes in CEQ scores typically incremental  Sample composition consistent between 2009-10  Response scale change flagged as potential cause  Discussion paper prepared ◦ Core items ◦ Hardcopy/online responses

19.  Denoting agreement/disagreement makes valence of positive/negative response more explicit (Weijters, 2010) ◦ Respondents generally have a desire to be agreeable (McClendon, 1991) ◦ Fully-labelled response scale results in an upward shift in the response distribution (Guterbock and Hubbard, 2000)  Greater accessibility of labelled response categories is likely to cause a shift away from the midpoint (Simonson, 1989)

22.  ‘Neither agree nor disagree’ or ‘Undecided’?  Ambiguity of unlabelled midpoint

23. Item 200820092010 NMeanSt. Dev.NMeanSt. Dev.NMeanSt. Dev. GTS0171,95721.350.573,67622.051.075,27130.746.2 GTS0371,94424.050.673,65324.351.275,30933.945.6 GTS1071,78523.551.673,54823.952.075,23431.447.2 GTS1571,78624.948.973,51525.249.375,26532.345.1 GTS1671,82825.950.173,54026.350.475,23633.345.9 GTS2771,64116.252.573,40016.653.275,17522.149.3 GSS0671,85226.953.573,59327.153.675,23636.047.2 GSS1471,88142.750.273,57242.651.075,26151.441.4 GSS2371,78637.848.573,45837.849.275,21046.640.7 GSS3271,78342.553.273,44742.353.775,21350.245.1 GSS4271,75734.248.573,45633.749.275,20940.842.2 GSS4371,75239.548.873,47139.149.775,19146.541.4 OSI4971,72339.350.273,42539.250.975,15847.442.8

24.  Shift in response distribution likely due to labelling ◦ Upward shift from midpoint (‘N’) to fourth point (‘A’) ◦ Consistent with literature  2010 beginning new CEQ time series  Positive development for the CEQ: ◦ More consistent responses ◦ More in line with PREQ response scale  CEQ review—response scale likely reassessed