Recommending Recommenders

1
Recommending Recommenders

• A Comparison of Human and Automated Systems
• for Book and Movie Recommendations

Kirsten Swearingen December 13, 2000 Prof. Sinha
IS 271
2
Motivation

• Designing a book recommendation service for
  childrenresearching best practices
• Looked at existing recommendation systems
• E-commerce
• Stand-alone
• Wanted to know
• How useful are they?
• How usable are they?
• How do their recommendations compare to those
  provided by human beings?

3
The Experiment Design

• 10 participants in experiment
• 5 tested 3 book recommender systems
• Amazon Recommendations Wizard
• Sleeper
• RatingZone Quick Picks
• 5 tested 3 movie recommender systems
• Amazon Recommendations Wizard
• MovieCritic
• Reel Videos Movie Matches
• Tasks
• Register and rate items
• Review recommendation set and evaluate

4
So Whats a Useful Recommendation?

• Not previously read/viewed
• Interested in reading/viewing
• May or may not have heard of item
• Simple evaluationnot identifying specific degree
  of interest

5
Measures

• Objective
• Time to register
• Time to find at least one useful recommendation
• Number of useful recommendations
• Subjective (participants opinions)
• How useful was system?
• How easy to use?
• How did interface elements affect experience?
• Would they use it again? Recommend it to someone
  else?

6
The Human Element

• Each participant also received a set of 3
  recommendations from 3 friends
• As with systems, reviewed set and identified
  useful recommendations

7
A Few Problems

• Small number of participants
• Lack of information about participants reading
  and viewing patterns
• Might have been better predictor of system
  usefulness than age, gender, Internet use
• Fairly homogeneous test group
• 3 5 years Internet experience (90)
• Ages 25 34 (90)
• SIMS students (70 overall, 100 of book testers)
• Design used nominal scale for some evaluation
  questions

8
The Bottom Line
9
Systems vs. Friends

• Friends best on average highest of useful
  items

• But in post-test interviews 50 of subjects said
  a system gave them the best recommendations.

10
Different Results for Books and Movies

• Systems fared slightly better in movie domain,
  friends significantly worse

11
One Book System Did Poorly
2 subjects found no useful recs at RatingZone
12
Not Enough Information?
13
Useful Recs Overall Heard of vs. Not Heard of

• Friends better in heard of category
• Most systems better for not heard of

14
Did recommended items both read liked predict
useful recs?
15
Time to find useful recs comparable for most
individuals
16
but some time differences between systems.
17
Interface Factors Books
Mostly Positive Effects
18
Interface Factors Movies
Some Negative Effects
19
Majority of systems rated useful or very useful
20
and easy to use
21
but not all are recommended.

• Sleeper and MovieCritic average highest
• Required the most ratings
• Split opinions on Amazon and Reel
• Required the fewest ratings

22
Amazon vs. Amazon

• Books/movies about the same of useful recs

• Book system more likely to be used in future

23
Conclusions

• Book and movie domains differ
• Friends win overall, but systems are useful too
• Connection between reading/viewing patterns and
  user satisfaction with systems
• Requiring more initial ratings does not seem to
  adversely affect user satisfaction (might even
  increase it)
• Information about item is key to usefulness of
  recommendation

24
Future Steps

• Run study with larger, more diverse pool of
  participants
• Focus on one domain books
• Gather more specific information
• Degree of interest in recommended item
• Reason for interest
• System elements that help them decide whether
  they are/arent interested in item
• Follow-up study to see if rec. was a good match

25
Thanks to all who participated!

• Questions? Comments?