Adaptive Rewards Mechanism for Sustainable Online Learning Community

1
Adaptive Rewards Mechanism for Sustainable Online
Learning Community

• Ran Cheng and Julita Vassileva
• Computer Science Department University of
  Saskatchewan, Canada

2
Outline

• The problem
• Comtella a link-sharing online community
• Encouraging students to rate links
• Adaptive rewards mechanism
• Community model and individual model
• Adaptive rewards
• Case study
• Conclusions

3
The problem

• Why encouraging participation in online
  communities?
• The majority of systems remain unused
• Incentive mechanism is needed
• To encourage participation
• However, excessive participation can kill a
  community
• To ensure sustainability the incentive mechanism
  needs to
• Discourage excessive participation
• Ensure a way to measure the quality of
  contributions
• Encourage timely contributions (when needed by
  the community most) and good contributions (from
  people who tend to have higher standards)

4
Comtella

• Brief history
• 2002 P2P (Gnutella) client for sharing research
  papers (as files) in the MADMUC lab (Vassileva _at_
  CoopIS02, Bretzke Vassileva _at_ UM03)
• 2004 Modified to support a 4th year students in
  a class on Ethics and IT to share URLs related to
  each topic/ week) (Cheng Vassileva _at_ ITS2004,
  _at_HICSS05)
• 2005 Modified as centralized web-based online
  community (try it out at the Interactive Event
  today 1400-1500 in room C.611A)

5
Motivation Strategies in Comtella

• Rewarding participatory acts with points and
  status (membership)
• Sharing new links, rating links
• Points accumulate and result in higher membership
  for the user
• Visualization of the community showing user
  participation allowing different views for each
  kind of activity
• Results - case study in 2004
• Very effective in increasing participation in
  sharing new papers
• But also some low quality papers excessive
  participation, high cognitive load

Silver
Bronze
Gold
Plastic
Cheng R., Vassileva, J. (2005) User Motivation
and Persuasion Strategy for Peer-to-peer
Communities. HICSS'2005 Vassileva, J. (2004)
Harnessing P2P Power in the Classroom.
Proceedings Intelligent Tutoring Systems,
ITS'2004
6
Summary so far

• Measuring and rewarding participation can
  stimulate participation
• However, quality tends to deteriorate (some
  students game the system)
• Therefore,
• excessive participation should not be encouraged
• a way for measuring quality and rewarding
  high-quality contributions is needed

7
Encouraging students to rate links

• Modeled after Slashdot
• Students with high membership get more ratings to
  give out more power
• In addition, each act of rating is rewarded
• By earning c-points (a kind of virtual currency)
• By earning participation points (towards higher
  membership)

8
Measuring the Quality of Contributions

• All contributed URLs start at 0
• unlike Slashdot, where it depends on the karma
  of the user who submitted the post
• The final rating is the sum of all received
  ratings
• How to judge the quality of a rating?
• based on its similarity with other ratings for
  the same link
• How to prevent gaming the system?
• allow sorting search results by average rating
• but do not show the average rating

9
Incentives for rating (1)

• Each act of rating earns c-points
• The user can invest the c-points earned to
  increase the visibility of the links he/she
  contributes in the list of search results

10
Incentives for rating (2)

• By rating links users also earn participation
  points towards their membership
• The number of points earned depends on the
  quality of each rating
• Tendency towards the average taste?
• Is this the taste of the community?

11
Adaptive Rewards Mechanism

• Adapting to what?
• To the current needs of the community
• To the individual quantity and quality standards
  of the user
• Adapting what?
• The participation points earned by different
  activities (economic incentives)
• The personal message to the user at login

12
Skip details
13
Community Model

• Community reward factor (depends on time since
  introducing topic) - Fc

• Expected / desirable number of links for topic
  (set by instructor) Qc
• Depends on topic, the time in the semester,
  busyness of students etc.

14
Individual Model (1)

• The user reputation in bringing high quality
  links RI the average summative rating of all
  links contributed by the user
• Users with high RI should be encouraged to
  contribute more at any time
• Users with low RI should be discouraged, unless
  the topic is still fresh and lacks links.
• The expected number of links from individual user
  for the current topic

15
Individual model (2)

• Individual reward factor FI
• User reputation in giving high-quality ratings
• Computed as 1/EI where

16
Adaptive rewards mechanism

• Varying weights Wi(t) for particular forms of
  participation
• The user contribution is computed as
• where t the time of contribution
• The weights depend on the state of the users
  individual model and on the community model at
  the moment of contribution
• Weight for sharing links
• Wieght for giving ratings proportional to the
  user reputation for giving high quality ratings
  1/EI

17
Adaptive messages to the user

• Communicating the rewards that the user will get
  for each type of action at the time of login.
• user who shares many papers of poor quality gets
  a low RI and a small QI , therefore little reward
  for subsequent contributions.
• the related message would be to contribute less
  in next period but improve the quality of her
  contributions.
• Show it

18
Case study

• Comtella used in the Ethics and IT class at the
  UofS
• Jan-April 2005
• 32 students
• Two groups of 16 throughout the term
• Test - Comtella 1 with adaptive rewards
  mechanism and c-points
• Control - Comtella 2 no adaptive rewards
  mechanism, no c-points
• Groups formed to have equal gender and Canadian /
  foreign representation
• Test and Control groups are separate communities
• no interaction of shared links, ratings etc.
• However, students were in the same classroom for
  lectures project teams across both groups
• We compared the numbers of contributions in each
  group (links, ratings)
• Post-study online questionnaire

19
Questions and answers (1)

• Did the users in the test group (Comtella 1) give
  more ratings?
• Yes nearly twice as much as Comtella 2 1065 vs.
  613 ratings (significant)
• Did the summative ratings in Comtella 1 reflect
  better the quality of the contributed links?
• Yes in Comtella 1, 56 (9 users) felt that the
  final summative ratings that their links received
  reflect fairly their quality, while in Comtella
  2, only 25 (4 users) thought so.
• Did the users in Comtella 1 tend to share links
  earlier in the week?
• Yes users in Comtella 1 shared 71.3 of their
  contributions in the first 3 days after
  introducing the topic users in Comtella 2 shared
  60.6 of their contributions in the first 3 days.
  The difference was significant for all topics
  and ranged between 7-14.
• Did the users in Comtella 1 share the number of
  links that was expected from them?
• To some extent about half of the users did.

20
Questions and answers (2)

• Did the users in Comtella 1 participate more
  actively in general?
• Yes they read more papers (3419 vs. 2416) and
  logged in the system more frequently (1714 vs.
  982).
• Is there a significant difference with respect to
  the total number of contributed links between the
  test and the control group?
• No 613 in Comtella 1 versus 587 in Comtella 2
• There was no excessive paper contribution in
  either case.
• Did the users in Comtella 1 contribute links with
  higher quality?
• Is there a correlation between the ratings of
  links and the times they were chosen for
  summarizing?
• Yes strong correlation
• 0.861 for links with rating gt 0,0.928 for links
  with rating gt1, 0.983 for links with rating gt 2,
  1 for links with rating gt 3

21
Discussion

• Incorporating an incentive mechanism can
  stimulate a desired behaviour in an online
  community
• in our case, the c-points stimulated ratings
• can be useful for collaborative filtering systems
• An adaptive rewards mechanism can orchestrate a
  desired pattern of collective behaviour
• in our case, the time-adaptation of the rewards
  stimulated users to make contributions earlier
• It is important to make the user aware of the
  rewards for different actions at any given time
• More experiments are needed to show effect of
  adaptation to individual pattern and impact on
  the quality / quality perception

22
Future work

• Deal with the subjectivity of ratings
• Adapt the mechanism to motivate contributions in
  a different kind of community,
• a research group or a group of graduate students
  sharing research papers,
• a community of researchers (the AIED and /or the
  UM community?)
• a community for girls and women in science and
  engineering

23
More information

• Try Comtella at
• http//svaroy.usask.ca8080/aied
• Try Comtella used in the case study
• http//svaroy.usask.ca8080/ Login using
  comtellaguest as username and password
• See other papers, follow up project
• http//bistrica.usask.ca/madmuc/peer-motivation.ht
  m
• Or just Google for Comtella

24
back
25
(No Transcript)
26
(No Transcript)
27
(No Transcript)
28
(No Transcript)
29
(No Transcript)
30
Back