Evaluating Similarity Measures: A Large-Scale Study in the orkut Social Network

1
Evaluating Similarity Measures A Large-Scale
Study in the orkutSocial Network

• Ellen Spertus
• spertus_at_google.com

2
Recommender systems

• What are they?
• Example Amazon

3
Controversial recommenders

• What to do when your TiVo thinks youre gay,
  Wall Street Journal, Nov. 26, 2002

http//tinyurl.com/2qyepg
4
Controversial recommenders

• What to do when your TiVo thinks youre gay,
  Wall Street Journal, Nov. 26, 2002

http//tinyurl.com/2qyepg
5
Controversial recommenders

• What to do when your TiVo thinks youre gay,
  Wall Street Journal, Nov. 26, 2002

http//tinyurl.com/2qyepg
6
Controversial recommenders

• Wal-Mart DVD recommendations

http//tinyurl.com/2gp2hm
7
Controversial recommenders

• Wal-Mart DVD recommendations

http//tinyurl.com/2gp2hm
8
Controversial recommenders

• Wal-Mart DVD recommendations

http//tinyurl.com/2gp2hm
9
Googles mission

• To organize the world's information and make it
  universally accessible and useful.

10
communities
11
Community recommender

• Goal Per-community ranked recommendations
• How to determine?

12
Community recommender

• Goal Per-community ranked recommendations
• How to determine?
• Implicit collaborativefiltering
• Look for common membership between pairs of
  communities

13
Terminology

• Consider each community to be a set of members
• B base community (e.g., Pizza)
• R related community (e.g., Cheese)
• Similarity measure
• Based on overlap BnR

14
Example Pizza
15
Example Pizza
16
Terminology

• Consider each community to be a set of members
• B base community (e.g., Wine)
• R related community (e.g., Linux)
• Similarity measure
• Based on overlap BnR
• Also depends on B and R
• Possibly asymmetric

17
Example of asymmetry
18
Similarity measures

• L1 normalization
• L2 normalization
• Pointwise mutual information
• Positive correlations
• Positive and negative correlations
• Salton tf-idf
• Log-odds

19
L1 normalization

• Vector notation
• Set notation

20
L2 normalization

• Vector notation
• Set notation

21
Mutual information positive correlation

• Formally,
• Informally, how well membership in the base
  community predicts membership in the related
  community

22
Mutual information positive and negative
correlation
23
Salton tf-idf
24
LogOdds0

• Formally,
• Informally, how much likelier a member of B is to
  belong to R than a non-member of B is.

25
LogOdds0

• Formally,
• Informally, how much likelier a member of B is to
  belong to R than a non-member of B is.
• This yielded the same rankings as L1.

26
LogOdds
27
Predictions?

• Were there significant differences among the
  measures?
• Top-ranked recommendations
• User preference
• Which measure was best?
• Was there a partial or total ordering of measures?

28
Recommendations for I love wine (2400)
29
Experiment

• Precomputed top 12 recommendations for each base
  community for each similarity measure
• When a user views a community page
• Hash the community and user ID to
• Select an ordered pair of measures to
• Interleave, filtering out duplicates
• Track clicks of new users

30
Click interpretation
31
Click interpretation
32
Overall click rate (July 1-18)
Total recommendation pages generated 4,106,050
33
Overall click rate (July 1-18)
34
Overall click rate (July 1-18)
35
Analysis

• For each pair of similarity measures Ma and Mb
  and each click C, either
• Ma recommended C more highly than Mb
• Ma and Mb recommended C equally
• Mb recommended C more highly than Ma

36
Results

• Clicks leading to joins
• L2 MI1 MI2 IDF L1 LogOdds
• All clicks
• L2 L1 MI1 MI2 IDF LogOdds

37
Positional effects

• Original experiment
• Ordered recommendations by rank
• Second experiment
• Generated recommendations using L2
• Pseudo-randomly ordered recommendations, tracking
  clicks by placement
• Tracked 1.3 M clicks between September
  22-October 21

38
Results single row (n28108)
Namorado Para o Bulldog
39
Results single row (n28,108)
p.12 (not significant)
40
Results two rows (n24,459)
41
Results two rows (n24,459)
p lt .001
42
Results 3 rows (n1,226,659)
43
Results 3 rows (n1,226,659)
p lt .001
44
Users reactions

• Hundreds of requests per day to add
  recommendations
• Angry requests from community creators
• General
• Specific

45
Amusing recommendations
C
46
Amusing recommendations
C
Whats she trying to say? For every time a woman
has confused you
47
Amusing recommendations
Chocolate
48
Amusing recommendations
Chocolate
PMS
49
Allowing community owners to set recommendations
50
Allowing community owners to set recommendations
51
Manual recommendations

• Eight days after release
• 50,876 community owners
• Added 267,623 recommendations
• Deleted 59,599 recommendations
• Affecting 73,230 base communities and
• 111,936 related communities
• Open question How do they compare with automatic
  recommendations?

52
Future research 1

• Determining similar users based on common
  communities
• Is it useful?
• Will the measures make the same total order?

(9 users)
53
Other types of information

• Distance in social network
• Demographic
• Country
• Age
• Etc.

54
Future research 2

• Per-user community recommendations
• Using social network information
• Using profile information (e.g., country)

55
Future research 2

• Per-user community recommendations
• Using social network information
• Using profile information (e.g., country)

56
Future research 2

• Per-user community recommendations
• Using social network information
• Using profile information (e.g., country)

57
Future research 3

• Do we get the same ordering for other domains?

L2 MI1 MI2 IDF L1 LogOdds
58
Acknowledgments

• Mehran Sahami
• Orkut Buyukkokten
• orkut team

59
Bonus material
60
Self-rated beauty

• beauty contest winners
• very attractive
• attractive
• average
• mirror-cracking material

61
Self-rated beauty men

• beauty contest winners 8
• very attractive 18
• attractive 39
• average 24
• mirror-cracking material 11

62
Self-rated beauty women

• beauty contest winners 8
• very attractive 16
• attractive 39
• average 27
• mirror-cracking material 9

63
Self-rated beauty by country

• Most beautiful
• men
• women
• Least beautiful
• men
• women

64
Self-rated beauty by country

• Most beautiful
• men Syrian
• women Barbadian
• Least beautiful
• men Gambian
• women Ascension Islanders

65
Ratings by others

• Karma
• trustiness
• sexiness
• coolness
• How do these correlate with age?

66
Ratings by others
67
Friend counts
68
Self-rated best body part