Datamining the Internet: Alexa

1
Datamining the Internet Alexa

• Brewster Kahle
• President, Alexa Internet
• brewster_at_alexa.com

2
To Answer Any Question...

• Know a lot
• Know what is important
• Be right enough

Alexa The web navigation service that learns
from people
3
Know Alot Other Repositories

• Library of Alexandria 800GB (400k scrolls _at_2MB)
• Library of Congress 20TB (20M books, ascii)
• Dialog Information Service 3-5TB
• Video Store 8TB (5k videos, 1GB/hr)
• Public Branch Library 3TB (300k scanned books)
• Radio Station 1TB (15k hrs of music)
• . . . Alexas Internet Archive 10TB

4
Know A lot Gathering

• Web Snapshot on T3 in 20 days
• Users Paths essential as well

5
8 Terabytes so far
6
Web Stats

• 1million sites, doubling every 6 months (millions
  of authors)
• More videos, dynamic pages, Java etc.
• 15 links on each page

7
Storage
Snapshot of the Web on Tape Jukebox costs 80k
8
Knowing what is ImportantMining the WWW for
Quality

• Content 100 million pages
• Link Structure 750 million links
• Usage paths many 100 million hits

9
Be Right Enough being useful

• Competition
• Directories
• Biggest only links to lt 1 of the WebPages
• Search Engines
• Returning 1000s of hits (sometimes millions)
• Trends
• Move to channels of less content, but good
• limit crawling (50M pages and holding)

10
Be Right Enough Alexa

• Where am I?
• Where do I want to go?
• Alexa
• Can I trust this information
• What should I look at next?

11
(No Transcript)
12
(No Transcript)
13
(No Transcript)
14
Travel Agents
15
Conde Naste Travel
16
Ford Vehicles Homepage
17
Fords Mustang Page
18
Independent Mustang Page
19
Surrealism Page
20
Women Surrealists
21
Archive in action
22
Alexa Conclusion