Search Engines Information Ranking and Retrieval

1
Search EnginesInformation Ranking and Retrieval

• Summer Semester 2004
• Internet Technologies Course, University of
  Hannover, CS Dept.

2
Overview of the Web
3
Top Online Activities(Jupiter Communications,
2000)
(a) Source Jupiter Communications.
4
Pew Study (US users July 2002)

• Total Internet users 111 M
• Do a search on any given day 33 M
• Have used Internet to search 85
• http//www.pewinternet.org/reports/toc.asp?R
  eport64

5
Search on the Web

• CorpusThe publicly accessible Web static
  dynamic
• Goal Retrieve high quality results relevant to
  the users need
• (not docs!)
• Need
• Informational want to learn about something
  (40)
• Navigational want to go to that page (25)
• Transactional want to do something
  (web-mediated) (35)
• Access a service
• Downloads
• Shop
• Gray areas
• Find a good hub
• Exploratory search see whats there

Relativity theory
United Airlines
Car rental Finland
6
Results

• Static pages (documents)
• text, mp3, images, video, ...
• Dynamic pages generated on request
• data base access
• the invisible web
• proprietary content, etc.

7
Terminology
URL Universal Resource Locator

• http//www.cism.it/cism/hotels_2001.htm

Host name
Access method
Page name
8
Scale

• Immense amount of content
• 2-10B static pages, doubling every 8-12 months
• Lexicon Size 10s-100s of millions of words
• Authors galore (1 in 4 hosts run a web server)

http//www.netcraft.com/Survey
9
Diversity

• Languages/Encodings
• Hundreds (thousands?) of languages, W3C
  encodings 55 (Jul01) W3C01
• Home pages (1997) English 82, Next 15 13
  Babe97
• Google (mid 2001) English 53
• Document query topic
• Popular Query Topics (from 1 million Google
  queries, Apr 2000)

10
Rate of change

• Cho00 720K pages from 270 popular sites sampled
  daily from Feb 17 Jun 14, 1999

11
Web idiosyncrasies

• Distributed authorship
• Millions of people creating pages with their own
  style, grammar, vocabulary, opinions, facts,
  falsehoods
• Not all have the purest motives in providing
  high-quality information - commercial motives
  drive spamming - 100s of millions of pages.
• The open web is largely a marketing tool.
• IBMs home page does not contain computer (could
  be in news though)

12
Other characteristics

• Significant duplication
• Syntactic - 30-40 (near) duplicates
  Brod97, Shiv99b
• Semantic - ???
• Complex graph topology
• 8 links/page in the average
• Not a small world bow-tie structure Brod00
• More on these corpus characteristics later
• how do we measure them?

13
Web search users

• Ill-defined queries
• Short
• AV 2001 2.54 terms avg 80 lt 3 words)
• Imprecise terms
• Sub-optimal syntax (80 queries without operator)
• Low effort
• Wide variance in
• Needs
• Expectations
• Knowledge
• Bandwidth

• Specific behavior
• 85 look over one result screen only (mostly
  above the fold)
• 78 of queries are not modified (one
  query/session)
• Follow links the scent of information ...

14
Web search engine history
15
Evolution of search engines

• First generation -- use only on page, text data
• Word frequency, language
• Second generation -- use off-page, web-specific
  data
• Link (or connectivity) analysis
• Click-through data (What results people click on)
• Anchor-text (How people refer to this page)
• Third generation -- answer the need behind the
  query
• Semantic analysis -- what is this about?
• Focus on user need, rather than on query
• Context determination
• Helping the user
• Integration of search and text analysis

1995-1997 AV, Excite, Lycos, etc
From 1998. Made popular by Google but everyone
now
Still experimental
16
First generation ranking

• Extended Boolean model
• Matches exact, prefix, phrase,
• Operators AND, OR, AND NOT, NEAR,
• Fields TITLE, URL, HOST,
• AND is somewhat easier to implement, maybe
  preferable as default for short queries
• Ranking
• TF like factors TF, explicit keywords, words in
  title, explicit emphasis (headers), etc
• IDF factors IDF, total word count in corpus,
  frequency in query log, frequency in language

17
Second generation search engine

• Ranking -- use off-page, web-specific data
• Link (or connectivity) analysis
• Click-through data (What results people click on)
• Anchor-text (How people refer to this page)
• Crawling
• Algorithms to create the best possible corpus

18
Connectivity analysis

• Idea mine hyperlink information in the Web
• Assumptions
• Links often connect related pages
• A link between pages is a recommendation
• people vote with their links

19
Third generation search engine answering the
need behind the query

• Query language determination
• Different ranking
• (if query Japanese do not return English)
• Hard soft matches
• Personalities (triggered on names)
• Cities (travel info, maps)
• Medical info (triggered on names and/or results)
• Stock quotes, news (triggered on stock symbol)
• Company info,
• Integration of Search and Text Analysis

20
Answering the need behind the queryContext
determination

• Context determination
• spatial (user location/target location)
• query stream (previous queries)
• personal (user profile)
• explicit (vertical search, family friendly)
• implicit (use AltaVista from AltaVista France)
• Context use
• Result restriction
• Ranking modulation

21
The spatial context - geo-search

• Two aspects
• Geo-coding
• encode geographic coordinates to make search
  effective
• Geo-parsing
• the process of identifying geographic context.
• Geo-coding
• Geometrical hierarchy (squares)
• Natural hierarchy (country, state, county, city,
  zip-codes, etc)
• Geo-parsing
• Pages (infer from phone nos, zip, etc). About
  10 feasible.
• Queries (use dictionary of place names)
• Users
• From IP data
• Mobile phones
• In its infancy, many issues (display size,
  privacy, etc)

22
Helping the user

• UI
• spell checking
• query refinement
• query suggestion
• context transfer

23
Context sensitive spell check
24
Deeper look into a search engine
25
Typical Search Engine
26
Typical Search Engine (2)

• User Interface
• Needed to take the user query
• Index
• Database/repository with the data to be searched
• Search module
• Transforms query to understandable format
• Does matching with the index
• Returns the results as output with information
  needed

27
Typical Crawler Architecture
28
Typical Crawler Architecture (2)

• Retrieving Module
• Retrieve each document from the Web and give it
  to the Process module
• URL Listing Module
• Feeds the Retrieving Module using its list of
  URLs
• Process Module
• Processes data from the Retrieving Module
• Sends new discovered URLs to the URL Listing
  Module
• Sends the Web page text to the Format Store
  Module
• Format Store Module
• Converts data to better format and store it into
  the index
• Index
• Database/repository with the useful data retrieved

29
Putting some order in the WebPage Ranking
30
Query-independent ordering

• First generation using link counts as simple
  measures of popularity.
• Two basic suggestions
• Undirected popularity
• Each page gets a score the number of in-links
  plus the number of out-links (325).
• Directed popularity
• Score of a page number of its in-links (3).

31
Query processing

• First retrieve all pages meeting the text query
  (say venture capital).
• Order these by their link popularity (either
  variant on the previous page).

32
Pagerank scoring

• Imagine a browser doing a random walk on web
  pages
• Start at a random page
• At each step, go out of the current page along
  one of the links on that page, equiprobably
• In the steady state each page has a long-term
  visit rate - use this as the pages score.

1/3 1/3 1/3
33
The Adjacency Matrix (A)

• Each page i corresponds to row i and column i of
  the matrix.
• If page j has n successors (links), then the ijth
  entry is 1/n if page i is one of these n
  successors of page j, and 0 otherwise.

34
Not quite enough

• The web is full of dead-ends.
• Random walk can get stuck in dead-ends.
• Makes no sense to talk about long-term visit
  rates.

??
All pages will end up with rank 0 !
35
Spider Traps Easy SPAM

• One can easily increase its rank by creating a
  spider trap

MS will converge to 3, i.e. get all !
36
Solution - Teleporting

• At each step, with probability c (10-20), jump
  to a random web page.
• With remaining probability 1-c (80-90), go out
  on a random link.
• If no out-link, stay put in this case.

37
Example

• Suppose c 0.2 (20 probability to teleport to a
  random page)

• Converges to n 7 / 11, m 21 / 11, a 5 / 11
• Scores could be normalized after each iteration
  (to sum to 1)

38
Pagerank summary

• Preprocessing
• Given graph of links, build matrix P.
• From it compute a.
• The entry ai is a number between 0 and 1 the
  pagerank of page i.
• Query processing
• Retrieve pages meeting query.
• Rank them by their pagerank.
• Order is query-independent.

39
The reality

• Pagerank is used in google, but so are many other
  clever heuristics

40
Topic Specific Pagerank Have02

• Conceptually, we use a random surfer who
  teleports, with say 10 probability, using the
  following rule
• Selects a category (say, one of the 16 top level
  ODP categories) based on a query user -specific
  distribution over the categories
• Teleport to a page uniformly at random within the
  chosen category
• Sounds hard to implement cant compute PageRank
  at query time!

41
Non-uniform Teleportation
Sports
Teleport with 10 probability to a Sports page
42
Interpretation of Composite Score

• For a set of personalization vectors vj
• ?j wj PR(W , vj) PR(W , ?j wj vj)
• Weighted sum of rank vectors itself forms a valid
  rank vector, because PR() is linear wrt vj

43
Interpretation
Sports
10 Sports teleportation
44
Interpretation
Health
10 Health teleportation
45
Interpretation
Health
Sports
pr (0.9 PRsports 0.1 PRhealth) gives you 9
sports teleportation, 1 health teleportation
46
Topic Specific Pagerank Have02

• Implementation
• offlineCompute pagerank distributions wrt to
  individual categories
• Query independent model as before
• Each page has multiple pagerank scores one for
  each ODP category, with teleportation only to
  that category
• online Distribution of weights over categories
  computed by query context classification
• Generate a dynamic pagerank score for each page -
  weighted sum of category-specific pageranks

47
How big is the web?
48
What is the size of the web ?

• Issues
• The web is really infinite
• Dynamic content, e.g., calendar
• Soft 404 www.yahoo.com/ltanythinggt is a valid
  page
• Static web contains syntactic duplication, mostly
  due to mirroring (20-30)
• Some servers are seldom connected
• Who cares?
• Media, and consequently the user
• Engine design
• Engine crawl policy. Impact on recall.

49
What can we attempt to measure?

• The relative size of search engines
• The notion of a page being indexed is still
  reasonably well defined.
• Already there are problems
• Document extension e.g. Google indexes pages not
  yet crawled, by indexing anchortext.
• Document restriction Some engines restrict what
  is indexed (first n words, only relevant words,
  etc.)
• The coverage of a search engine relative to
  another particular crawling process.
• The ultimate coverage associated to a particular
  crawling process and a given list of seeds.

50
Statistical methods

• Random queries
• Random searches
• Random IP addresses
• Random walks

51
Some Measurements
Source http//www.searchengineshowdown.com/stats/
change.shtml
52
Shape of the web
53
Questions about the web graph

• How big is the graph?
• How many links on a page (outdegree)?
• How many links to a page (indegree)?
• Can one browse from any web page to any other?
  How many clicks?
• Can we pick a random page on the web?
• (Search engine measurement.)

54
Why?

• Exploit structure for Web algorithms
• Crawl strategies
• Search
• Mining communities
• Classification/organization
• Web anthropology
• Prediction, discovery of structures
• Sociological understanding

55
Algorithms

• Weakly connected components (WCC)
• Strongly connected components (SCC)
• Breadth-first search (BFS)
• Diameter

56
Web anatomy Brod00
57
Distance measurements

• For random pages p1,p2
• Prp1 reachable from p2 1/4
• Maximum directed distance between 2 SCC nodes
  gt28
• Maximum directed distance between 2 nodes, given
  there is a path gt 900
• Average directed distance between 2 SCC nodes
  16
• Average undirected distance 7

58
Power laws on the Web

• Inverse polynomial distributions
• Prk c/k? for a constant c.
• ? log Prk c - ? log k
• Thus plotting log Prk against log k should give
  a straight line (of negative slope).

59
Zipf-Pareto-Yule Distributions on the Web

• In-degrees and out-degrees of pages
• Kuma99, Bara99, Brod00
• Connected component sizes Brod00
• Both directed undirected
• Host in-degree and out-degree Bhar01b
• Both in terms of pages and hosts
• Also within individual domains
• Number of edges between hosts Bhar01b

60
In-degree distribution

• Probability that
• a random page has
• k other pages
• pointing to it is
• k -2.1 (Power law)

Slope -2.1
61
Out-degree distribution
Probability that a random page points to k other
pages is k -2.7
Slope -2.7
62
Thank You!