1. Design and Evaluation of an Adaptive Incentive Mechanism for Sustained Educational Communities
Ran Cheng and Julita Vassileva
User Modeling & User-Adapted Interaction
Presented by Rosta Farzan
PAWS Group Meeting
April 13, 2007

2. Problem Insufficient user participation in online communities
Provide incentives to encourage participation
Too much participation especially of low-quality

3. Goal
Proposing an incentive mechanism regulating quantity and quality of user contribution and ensuring a sustainable user participation

4. Outline Related work Brief introduction to Comtella Proposed mechanism
Collaborative quality evaluation mechanism
Brief introduction to Comtella
Proposed mechanism
Implementation of proposed mechanism in Comtella
Evaluation
Discussion

5. Collaborative Quality Evaluation Mechanism
Active and sustained user participation while avinding information overload requires quality control
Decentralized moderation
Real world example
Measuring the quality of journals or papers by counting the time they were cited
Online communities
Counting the number of clicks on each item
Problem
Not always a click means a positive attitude
Explicit user rating
E.g. rating process in Slashdot
Problem – “rich get richer”
Unfair score for items with insufficient attention
Lower initial rating
Contributed late in discussion

6. Requirement of Incentives
Time based community need
New contributions in the early period of discussion
Ratings of the contribution when many contributions are collected
Different users have different contribution patterns
Encourage higher participation for users who contributed few high-quality resources
Inhibit contributions from users who contributed many low-quality resources
Overall community need

7. Comtella Developed at University of Saskatchewan
Online community for sharing URLs of class-related web-resources (bookmarks)

8. Proposed Incentive Mechanism
Mechanism encouraging users to rate resources
Adaptive reward mechanism

9. Encouraging Users to Rate
All users can rate others’ contribution
Each user receives a limited number of rating points to give out
Users with higher level membership receives more points
Initial rating of zero independent of level of resource provider
Ensuring all contributions have equal chance to be read and rated
Summative rating
Incentive: Virtual currency (c-points)
Limited initial c-points
Awarding users for rating resources
Depending on user’s reputation
Can be invested to promote the initial visibility

10. Sorted by c-points
Summative Ratings
Report duplicate, broken, or special permission links

11. Adaptive Reward Mechanism
Hierarchical membership
Adapt rewards for different forms of participation
Quality of users’ participation so far
Community need
Personalized motivational message
Stating specific performance goals
Calculating adaptive rewards
Community Model
Individual Model

12. Community Model Expected # of total contributions
Set by community admin
Community Reward Factor
Usefulness of new resources

13. Individual Model Contribution reputation Current membership level
Sharing
Average summative rating of all shared resources
Rating
Quality ~ Difference between the specific rating and the average of all ratings the resource gets eventually
Smaller difference  Higher quality
Current membership level

14. Individual Model
Expected # of resource contribution
Reward Factor

15. Adaptive Reward Mechanism

16. Implementation of proposed mechanism in Comtella

17. Evaluation Questions to be answered
Will the users in the test group rate articles more actively?
How well will the summative ratings reflect the real quality of the articles?
Will the users tend to share resources earlier in the week?
Will the actual number of contributions be close to the desired one?
Will the users share the number of articles that is expected from them?
Will the users contribute a higher percentage of high-quality articles?
Will there be information overload?

18. Participant Class on Ethics and Information Technology
Share web-articles related to the topic of each week
31 4th year undergraduate students
Test group: 15
Control group: 16
Randomly assigned while controlling gender and nationality

19. Result Users in the test group were more active in rating articles
The articles with higher ratings were more likely to be chosen by users to summarize
The users in the test group were more satisfied with the summative ratings received by their articles
The users in the test group tended to share resources earlier in the week
No big difference between total number of shared articles across the two groups
In both groups, the users’ attitude towards the quality of the articles were generally neutral
No information overload problem – the overall contribution did not exceed the community need
The users in the test group were more active in terms of logging on the system and reading articles – sustainability

20. Discussion/Limitations
Choosing the Parameters for the mechanism
Expected sum of contributions for each topic
Threshold for reputation values
Community reward function
Narrow scale of rating (+1 or -1)
Less cognitive load
No option to rate different aspect of a paper
Popularity of topic, originality, interestingness
Measure of quality
Average rating means average taste rules
In educational context tends to be superficial, easy to read articles

21. Discussion/Connection to Our Work
Trying personalized motivational message in CourseAgent or CoPE
Good evaluation questions
“Some people are easy to be motivated by glory and recognition”
Is there any cognitive tool to measure this?
Compare Comtella with CoPE
CoPE has the option to write summary and read summary written by others
More personal benefit