1. Let's go formative: Continuous student ratings with Web 2
Let's go formative: Continuous student ratings with Web 2.0 application
Burger Christoph & Stefan Stieger
School of Psychology
University of Vienna
General Online Research 09
April 6-8, 2009
University of Vienna, Austria

2. Introduction Summative course evaluation
Student Ratings
Controversial but important for the assessment of teaching
Mainly summative (at the end of the course)
Representative for each course unit?
Possible biases
Primacy effects (the first lecture units have a greater impact)
Recency effects (the last lecture units have a greater impact)
No relationship between each unit and the overall judgement at the end of the term.
Burger Christoph & Stieger Stefan
GOR 2009, Vienna

3. Introduction Formative course evaluation
Formative approach
Evaluation after each lecture unit
Evaluation messages have to be short
Further benefits of formative approach:
Feedback for teachers is available directly after each lesson
Feedback can already be used for the next lecture unit
Courses can be quickly modified during the term
Students can develop a feeling of being part of the course
Students can directly profit from teacher’s improvement
Burger Christoph & Stieger Stefan
GOR 2009, Vienna

4. Introduction Twitter Social networking and micro-blogging service
Social networking and micro-blogging service
Text-based messages up to 140 characters long
Burger Christoph & Stieger Stefan
GOR 2009, Vienna

5. Introduction Advantages of Twitter
Messages are instantly available online and delivered to other users who are “following” (i.e. have signed up to receive posts of the user)
Only “Followers” (people who got permission of the owner of the messages) can read the messages
By using nicknames, answering the questions is completely anonymous
Posting messages can be done from everywhere at any time via the Twitter Web-site, SMS, IM or third party applications
The teacher can get an overview easily, since all evaluation messages are displayed on his/her profile
Burger Christoph & Stieger Stefan
GOR 2009, Vienna

6. Method Research Questions
Research question 1
Is Twitter useful?
Research question 2
Formative evaluation = summative offline evaluation
Research question 3
Summative offline evaluation = summative online evaluation
Research question 4
Formative evaluation -> influence on summative offline evaluation
Burger Christoph & Stieger Stefan
GOR 2009, Vienna

7. Method Design Pilot study Does formative evaluation via Twitter work?
Pilot study
Does formative evaluation via Twitter work?
Experimental study
Control group: Did NOT evaluate formatively
Measures
Formative evaluation
Offline summative evaluation
Online summative evaluation (University’s evaluation unit)
Burger Christoph & Stieger Stefan
GOR 2009, Vienna

8. Method Participants and Material
Pilot study (26 students)
Experimental study (40 students)
Formative evaluation
2 closed questions
How did you like the lecture today? (1..very good, 9..very bad)
How interesting was the lecture today? (1..very interesting, 9..not at all interesting)
3 open questions
What was good today?
What was bad today?
General comments / just wanted to tell you.
Closed questions were also part of the summative evaluation!
Burger Christoph & Stieger Stefan
GOR 2009, Vienna

9. Method Procedure – Evaluation via Twitter
The teacher created a Twitter account
After the first lecture unit, each student also created an individual Twitter account using an anonymous nickname
Students asked for permission to follow the teacher's messages and vice versa
These permissions were granted by both the teacher and students
Now the teacher was able to read the students messages on his Twitter profile
Furthermore, students could read the teacher’s messages on their profile but not their colleagues’ messages
Burger Christoph & Stieger Stefan
GOR 2009, Vienna

10. Results and Discussion Research question 1
Is the formative evaluation via Twitter useful?
Questions about the formative Evaluation via Twitter (1..totally agree, 5..totally disagree; except last question)
Pilot study
Experimental study
The instructions for the formative evaluation via twitter were comprehensible.
2.0 (1.1)
1.6 (0.7)
I got the meaning of the formative evaluation via twitter.
1.7 (1.1)
1.8 (0.9)
I got along with Twitters homepage very well.
2.2 (1.1)
1.9 (0.9)
Each lecture should be evaluated formatively.
3.0 (1.1)
2.7 (1.4)
The teacher responded to our formative feedback adequately.
3.3 (1.0)
2.3 (1.3)
All in all, I find the formative evaluation via Twitter (1..very useful, 5..not at all useful).
2.8 (1.0)
2.8 (1.3)
Burger Christoph & Stieger Stefan
GOR 2009, Vienna

11. Results and Discussion Research question 2
Does a formative evaluation come to the same conclusion as a summative one?
2 closed questions which were both asked in the formative and the offline summative evaluation were compared
Formative and summative evaluation do not come to the same conclusion (Not a single significant correlation was found)
Which form of evaluation is more reliable?
Further research is needed.
Burger Christoph & Stieger Stefan
GOR 2009, Vienna

12. Results and Discussion Research question 3
Do the results of an offline and online summative evaluation differ?
3 different scales (teacher and course evaluation questionnaire, general questions)
No significant differences
Burger Christoph & Stieger Stefan
GOR 2009, Vienna

13. Results and Discussion Research question 4
Do formative and summative evaluation interact?
No significant differences between control and experimental group.
Burger Christoph & Stieger Stefan
GOR 2009, Vienna

14. Conclusion Formative evaluation using Twitter was successful.
Formative evaluation using Twitter was successful.
Results from summative evaluations are stable. (timeframe of 7 days)
Formative evaluation does not interact with a summative evaluation.
Formative and summative evaluations don’t come to the same conclusion. It is questionable which result is more valid. Further studies should clear that point in more detail.
Overall, Twitter is a useful evaluation platform to evaluate courses formatively (on a weekly basis). Because of Twitters simple use and the electronic data handling, the administrative effort keeps within a limit.
Burger Christoph & Stieger Stefan
GOR 2009, Vienna

15. THANK YOU FOR YOUR ATTENTION
End of Presentation
THANK YOU FOR YOUR ATTENTION
Burger Christoph & Stieger Stefan
GOR 2009, Vienna