1. Evaluating a Student Rating of Teaching Form John Ogier Survey & Testing Unit (STU) University Centre for Teaching & Learning (UCTL) john.ogier@canterbury.ac.nz Ph 64-3-364-2850

2. A case of quality not being assured! “No single student rating item, nor set of related items will be useful for all purposes” Cashin (1995) “Student ratings should be used to make only crude judgements of instructional effectiveness (exceptional, adequate, and unacceptable).” D’Apollonia & Abrami (1997)

3. UOC Student Surveys Course –Organisation a well organised course –Stimulation helped to stimulate my interest in the course area –Workload The overall workload in this course was reasonable –Difficulty The level of difficulty of this course was reasonable –Overall Overall, this was a good quality course Teaching –Organisation classes were well organised –Communication able to communicate ideas and information clearly –Interest stimulated my interest in the subject –Attitude attitude towards students has been good –Overall Overall, the lecturer is an effective teacher

5. Effective? How appropriate are these 5 Likert scale questions? –Do they define a “good teacher”? –Is that measure reliable? 1-5 scale appropriate? –Some cultures are use to “1” being a good score –Is 3 really “neutral”? Students circle 2 no’s! Students circle a column!! Is it correct to use the mean of a discontinuous scale? Median rather than mean?

6. Other survey “quality” issues Was the class told about it in advance? What extra questions were asked? When in the semester was the survey held? What time was the survey held? How many other surveys were given out? Who administered the survey? How long were they given to complete it? Is it really anonymous? Ad hominem comments!

7. Possible factors Course Organisation Stimulation Workload Difficulty Overall Teaching Organisation Communication Interest Attitude Overall

8. Correlation of Q5 (Overall) mean against the means of Q1, Q2, Q3 & Q4 weighted by responses (capped at 50) OrganisationCommunicationInterestAttitude R 2 = 0.7441R 2 = 0.9222R 2 = 0.8310R 2 = 0.6580 The lecturer is able to communicate ideas and information clearly Overall, the lecturer is an effective teacher

9. Scatterplot & Regression Line - Overall v Communication

10. Where to now? Difficulties faced by tertiary NESB students in English language lectures Cosmopolitan UOC academics – both NESB & ESB students must cope! Finegan, T.A., & Siegfried, J.J. (2000). Are student ratings of teaching effectiveness influenced by instructors’ English language proficiency? –they found in an introductory Economics course that “the student ratings of ESL instructors are, on average, about 0.4 points lower, on a scale of 1 to 5, than the student ratings of native English speaking instructors.” The American Economist, 44, 17-29.

11. NESB by Subjective Classification Surveys (UGrad, 2 or more responses) - 7072 from ESB lecturers and 524 from NESB lecturers Means with 95% Conf. ESBNESB Q2 - Comm3.810 ± 0.0023.216 ± 0.009 Q5 - Overall3.922 ± 0.0023.496 ± 0.009 Q5 Difference ~ 0.43, Q2 ~ 0.60

12. A similar difference, but… What about across –Faculties? –Levels? –Departments?

13. Breakdown of surveys Number of Cases within grouping variables (and no. of lecturers surveyed) ArtsCommerceEngineeringScienceLaw Level Y ESL N 1 (Lect) 14 (8) 476 (153) 12 (5) 98 (31) 038 (15) 42 (12) 585 (167) 029 (10) 2 (Lect) 40 (16) 678 (196) 24 (9) 213 (62) 22 (6) 245 (81) 89 (23) 713 (206) 088 (18) 3 (Lect) 31 (13) 517 (151) 46 (14) 267 (83) 49 (15) 387 (96) 98 (24) 820 (218) 0193 (29) 4 (Lect) 000057 (17) 537 (148) 0000

14. Faculty Means

15. Why the faculty effect? Cashin (1990) mentions a hypothesis that –students’ quantitative skills are less well developed than their verbal skills so that quantitative based courses are more difficult for students and are more difficult for lecturers to teach.

16. By Level

17. Some relevant student comments The following student comments from relevant courses put ‘quantitative skills’ into perspective: ~ “He did the best he could but his accent was still very strong, and made a complex subject even harder to understand.” Ratings: ‘Comms’ = 2, ‘Overall’ = 3 ~ “.. we should have loud, clearly spoken English from the lecturer as otherwise you have to concentrate too hard on just understanding the lecturer.” Ratings: ‘Comms’ = 1, ‘Overall’ = 1 Compare a NESB student rating an ESB lecturer ~ a strong accent, ~ “It’ll be very interesting if the lecturer can speak in Thai.” Ratings: ‘Comms’ = 2, ‘Overall’ = 2.

18. Implications For a “Transmission model” of teaching, do we need anything more than a Communication question!!! ? –Need questions that cover the important aspects of teaching (or courses) with an appropriate weighting. NESB lecturers may have a tougher time gaining promotion and tenure. –Assistance with language? –Alternative communication methods? Assisting ESB & NESB students to understand and cope?

19. The NESB student’s view Econ104 ~ 33% NESB Students Online survey – 353 responses ESB, 165 NESB Overall – ESB 4.58 ± 0.07, NESB 4.28 ± 0.10 –“... And think about the language he use in the test paper some words I even do not know and if he want to use it then use it in his lecture. ECON test is not a word game its aim should be test people how well they learn for ECON!” Ratings: ‘Comms’ = 3, ‘Overall’ = 4

20. Questions? John Ogier john.ogier@canterbury.ac.nz Ph 64-3-364-2850