6132009

1
SavvySearch Engine

• First Law of Search EnginesThe World Wide Web
  may appear to be a visual medium, but it is
  indexed textually.

2
Introduction

• We are going to briefly talk about Intelligent
  Meta Search Engine
• Kalyan Boggavarapu,Graduate student at Lehigh
  University.

3
Which is the best search engine?

• A) Google
• B) Yahoo
• C) AltaVista
• D) AskJeeves
• E) ResearchIndex
• Not long ago I embarked on a search for the
  perfect search engine. There is something
  Zen-like about the notion. Could a search
  conducted on the World Wide Web come up with the
  best search engine for the World Wide Web?"
• 
  Ernest Ackermann Karen Hartman
  

4
Do we need more than one search Engine?

• Yes / No
• Some times we may need more than one search
  engines
• The fact the web is huge and a single search
  engine cannot cover all the contents of the web.
• A Search Engine may contain a part or a subset of
  the web.

5
Search Engine Requirement Specific
6
Meta Search Engines

• Def
• Search the databases of multiple sets of
  individual search engines simultaneously, from a
  single site and using the same interface
• Metasearch engines do not crawl the web compiling
  their own searchable databases
• provide a quick way of finding out which engines
  are retrieving the best results for you in your
  search
• Knowledge is of two kinds. We know a subject
  ourselves, or we know where we can find
  information upon it."             --Samuel
  Johnson, 1744

7
What are pros and cons of meta search?

• Less search options you are usually at the
  mercy of the metasearch engine as far as how the
  search is configured and conducted.
• Few query Google which has the largest database
  of the web.
• They rely on small net search engines, web
  directories and pay per click engines
• They cannot increase the precision

8
Use and Example

• When do we use meta search engine?
• in a hurry
• simple search
• not having any luck
• Example of a meta search engine
  http//www.debriefing.com/
• You can't expect to hit the jackpot if you don't
  put a few nickels in the machine. "          
    --Flip Wilson, 1971

9
Problem and Solution

• Problem
• Current meta search engines can lead to tragedy
  of common means, individuals best interests run
  counter to the societys.
• If they offer to search all the possible search
  engines to satisfy a request, then, the process
  may waste Web resources network load and search
  engine computation.
• Our Solution
• Query Specific Targeting those search engines
  which are likely to return valid results.
• Load Sensitive responding to changing load
  demand on the web

10
Features of Savvy Search Engine

• Submitting a Query
• Processing a Query
• Results processing
• Ranking Search engines
• Recent Performance
• Rank calculation
• Presenting results

11
Submitting a Query
12
2) Ranking Search Engines

• Before(the search)
• Long term performance
• Search engine Rank table

13
Ranking search engine

• After
• Rq,s Qq,s ( Ps,h Ps,r )
• Qq,s S ( (Mt,s It) / Squareroot(Ts) )
• Where
• Rq,s Rank of a search engine for a query q.
• Qq,s Query score.

14
What are these variables?

• Rq,s gives the rank of the search engine
• Ps,h is the penalty if number of hits threshold number of hits
• Ps,r is the penalty response time threshold
  response time.
• "Any sufficiently advanced technology is
  indistinguishable from magic."          
    --Arthur C. Clarke, Profiles of the Future,
  1962

15
Presenting Results

• Output
• 10 search results are presented description
  search engines scanned alternate strategies.
• Eg q Shakespeare's Tempest.
• Savvy searched ?3 Web-crawler, Lycos and
  Yellow-pages
• (10 results ) ( 3 search engines ) 30 total
  results.
• Selects 10 out of 30.
• The average number of searches for a query is
  1.42 is measured.

16
What about Learning Algorithm

• Our observation
• Before training (visits represents relevant
  and No results represents selecting a bad search
  engine)
• After Training for 100 days, 5000 words
• "Research is the process of going up alleys to
  see if they are blind."             --Marsten
  Bates, 1967

17
How do we combine the results from Multiple
Search Engines?

• Document score for each document are returned
  from the search engines.
• We then normalize these Documents scores with
  our Scores (Rank of Search engine).
• Eg D1 from S1 is 70
• D2 from S2 is 40
• R1 (Rank of search engine) 0.0005
• R2 (Rank for second search engine ) is 0.001
• order of Results 1. D2
• 2. D1

18
Which would You Prefer ?

• After you heard so much about the meta search
  engines , Intelligent meta search engines ,
  general large search engines (the one you usually
  use). Which one would you prefer next time. ?

19
The Original Source

• SavvySearch A Meta-Search Engine that Learns
  which Search Engines to Query by Adele E.Howe
  and Daniel Dreilinger AI Magazine.
• Currently the current
• SavvySearch.com Search.com

20
Questions.

• Your Turn!
• Kalyan Boggavarapu,
• CSC Dept,
• Lehigh University.

21
To do
22
How do we accomplish our goal

• Appropriate Tracks the long term performance of
  search engines.
• Worth monitor recent performance of search
  engines to determine which are worth trying to
  contact for specific queries.
• "The Web is like the game of Othello.
  Remember?'a moment to learn, a lifetime to
  master.' "             --Lou Rosenfeld,
  University of Michigan, SLIS, n.d.

23
How do we know - Query Score?

• The search engine supplies us with
• Term frequency in the page
• Number of times the terms appears in the whole
  corpus/collection/database.

24
Types of Meta Search Engines

• Simple which allow the user to select which
  search engine to be contacted. Eg Profusion
• Automated selects three of six search engines
  egProfusion
• Post analysis - downloads and analysis the links
  returned by the search engines to prune out
  unavailable and irrelevant links. eg MetaCrawler
  , Inquires.

25
1) Resource Reasoning

• Concurrently querying the search engines to
  achieve speed.
• How many search engines to contact?
• Depends on 1) Network load 2) CPU load
• Network Load from the values of that network at
  same time of the day in the past.
• Visits number of links explored by the user
• No Results search engine failed to return
  links
• What is the effect of the above two results?
• These values are used in further calculations.

26
Personalization

• What is personalization?
• Tuning the search strategies specific to person
• How is this achieved?
• Every person is categorized into one of the 8
  categories.
• We implement different rules of each of the 8
  categories.
• Person ? category mappings are stored for future
  references