Let's go formative: Continuous student ratings with Web 2 Let's go formative: Continuous student ratings with Web 2.0 application Burger Christoph & Stefan Stieger School of Psychology University of Vienna General Online Research 09 April 6-8, 2009 University of Vienna, Austria

Introduction Summative course evaluation 1 2 3 4 5 6 7 8 9 10 11 12 13 Student Ratings Controversial but important for the assessment of teaching Mainly summative (at the end of the course) Representative for each course unit? Possible biases Primacy effects (the first lecture units have a greater impact) Recency effects (the last lecture units have a greater impact) No relationship between each unit and the overall judgement at the end of the term. Burger Christoph & Stieger Stefan GOR 2009, Vienna

Introduction Formative course evaluation 1 2 3 4 5 6 7 8 9 10 11 12 13 Formative approach Evaluation after each lecture unit Evaluation messages have to be short Further benefits of formative approach: Feedback for teachers is available directly after each lesson Feedback can already be used for the next lecture unit Courses can be quickly modified during the term Students can develop a feeling of being part of the course Students can directly profit from teacher’s improvement Burger Christoph & Stieger Stefan GOR 2009, Vienna

Introduction Twitter Social networking and micro-blogging service 1 2 3 4 5 6 7 8 9 10 11 12 13 Social networking and micro-blogging service Text-based messages up to 140 characters long Burger Christoph & Stieger Stefan GOR 2009, Vienna

Introduction Advantages of Twitter 1 2 3 4 5 6 7 8 9 10 11 12 13 Messages are instantly available online and delivered to other users who are “following” (i.e. have signed up to receive posts of the user) Only “Followers” (people who got permission of the owner of the messages) can read the messages By using nicknames, answering the questions is completely anonymous Posting messages can be done from everywhere at any time via the Twitter Web-site, SMS, IM or third party applications The teacher can get an overview easily, since all evaluation messages are displayed on his/her profile   Burger Christoph & Stieger Stefan GOR 2009, Vienna

Method Research Questions 1 2 3 4 5 6 7 8 9 10 11 12 13 Research question 1 Is Twitter useful? Research question 2 Formative evaluation = summative offline evaluation Research question 3 Summative offline evaluation = summative online evaluation Research question 4 Formative evaluation -> influence on summative offline evaluation Burger Christoph & Stieger Stefan GOR 2009, Vienna

Method Design Pilot study Does formative evaluation via Twitter work? 1 2 3 4 5 6 7 8 9 10 11 12 13 Pilot study Does formative evaluation via Twitter work? Experimental study Control group: Did NOT evaluate formatively Measures Formative evaluation Offline summative evaluation Online summative evaluation (University’s evaluation unit) Burger Christoph & Stieger Stefan GOR 2009, Vienna

Method Participants and Material 1 2 3 4 5 6 7 8 9 10 11 12 13 Pilot study (26 students) Experimental study (40 students) Formative evaluation 2 closed questions How did you like the lecture today? (1..very good, 9..very bad) How interesting was the lecture today? (1..very interesting, 9..not at all interesting) 3 open questions What was good today? What was bad today? General comments / just wanted to tell you. Closed questions were also part of the summative evaluation! Burger Christoph & Stieger Stefan GOR 2009, Vienna

Method Procedure – Evaluation via Twitter 1 2 3 4 5 6 7 8 9 10 11 12 13 The teacher created a Twitter account After the first lecture unit, each student also created an individual Twitter account using an anonymous nickname Students asked for permission to follow the teacher's messages and vice versa These permissions were granted by both the teacher and students Now the teacher was able to read the students messages on his Twitter profile Furthermore, students could read the teacher’s messages on their profile but not their colleagues’ messages Burger Christoph & Stieger Stefan GOR 2009, Vienna

Results and Discussion Research question 1 1 2 3 4 5 6 7 8 9 10 11 12 13 Is the formative evaluation via Twitter useful? Questions about the formative Evaluation via Twitter (1..totally agree, 5..totally disagree; except last question) Pilot study Experimental study The instructions for the formative evaluation via twitter were comprehensible. 2.0 (1.1) 1.6 (0.7) I got the meaning of the formative evaluation via twitter. 1.7 (1.1) 1.8 (0.9) I got along with Twitters homepage very well. 2.2 (1.1) 1.9 (0.9) Each lecture should be evaluated formatively. 3.0 (1.1) 2.7 (1.4) The teacher responded to our formative feedback adequately. 3.3 (1.0) 2.3 (1.3) All in all, I find the formative evaluation via Twitter (1..very useful, 5..not at all useful). 2.8 (1.0) 2.8 (1.3) Burger Christoph & Stieger Stefan GOR 2009, Vienna

Results and Discussion Research question 2 1 2 3 4 5 6 7 8 9 10 11 12 13 Does a formative evaluation come to the same conclusion as a summative one? 2 closed questions which were both asked in the formative and the offline summative evaluation were compared Formative and summative evaluation do not come to the same conclusion (Not a single significant correlation was found) Which form of evaluation is more reliable? Further research is needed. Burger Christoph & Stieger Stefan GOR 2009, Vienna

Results and Discussion Research question 3 1 2 3 4 5 6 7 8 9 10 11 12 13 Do the results of an offline and online summative evaluation differ? 3 different scales (teacher and course evaluation questionnaire, general questions) No significant differences Burger Christoph & Stieger Stefan GOR 2009, Vienna

Results and Discussion Research question 4 1 2 3 4 5 6 7 8 9 10 11 12 13 Do formative and summative evaluation interact? No significant differences between control and experimental group. Burger Christoph & Stieger Stefan GOR 2009, Vienna

Conclusion Formative evaluation using Twitter was successful. 1 2 3 4 5 6 7 8 9 10 11 12 13 Formative evaluation using Twitter was successful. Results from summative evaluations are stable. (timeframe of 7 days) Formative evaluation does not interact with a summative evaluation. Formative and summative evaluations don’t come to the same conclusion. It is questionable which result is more valid. Further studies should clear that point in more detail. Overall, Twitter is a useful evaluation platform to evaluate courses formatively (on a weekly basis). Because of Twitters simple use and the electronic data handling, the administrative effort keeps within a limit. Burger Christoph & Stieger Stefan GOR 2009, Vienna

THANK YOU FOR YOUR ATTENTION End of Presentation 1 2 3 4 5 6 7 8 9 10 11 12 13 THANK YOU FOR YOUR ATTENTION Burger Christoph & Stieger Stefan GOR 2009, Vienna