Web Search Engines

1
Web Search Engines

• INFL 100
• 10.30.06

2
Web Search Engines

• Facts and Figures
• Search Engines- How do they work?
• Meta-Search Engines
• Specialized Search Engines

3
Size of the Web

• The web is ENORMOUS (estimates)
• 1997- 125 million pages
• 1999- 320 million pages
• 2001- 1-3 billion pages
• 2003- 4 billion - ??

4
Web Search Engines

• What is a web search engine and how do they work?
  
  
• A web search engine is an interactive tool that
  enables users to locate information available via
  the World Wide Web.
  
• Search Engines
• Google, Yahoo, Northern Light, 100s more
• Meta-Search Engines
• Metacrawler, Dogpile

5
Web Search Engines

• When you use a search engine, you are not
  searching the web directly. The web is the
  totality of many web pages residing on many, many
  computers or servers throughout the world. Your
  computer can not possibly find or go to them all.
  
• You can access any one of several intermediate
  databases that link you to selections of web
  pages. (i.e. search engines)
  

6
Web Search Engines

• A search engine is made up of 4 main parts
• 1.) the engine's spiders 2.) the indexing
  program 3.) the retrieval engine 4.) the
  graphical interface
  

7
Web Search Engines

• 1. Spiders
• Most databases are created automatically using
  spiders (also known as robots or crawlers)
  
• Spiders go out to the Web and identify new sites
  that are to be added to the database and identify
  sites already covered that have changed.
  
• Generally, the more popular sites are crawled
  more often and more thoroughly.
  
• When you use a Web search engine, you search the
  compiled database, not the actual Web.

8
Web Search Engines

• Freshness refers to how frequently the
  information in the database is updated.
  
• If database has not been updated for two months,
  users are actually searching the Web as it was
  two months previously.
  
• It is sometimes hard to tell how fresh a database
  really is.
  
• Some prominent parts of a database may be updated
  more often than the bulk of the database.

9
Web Search Engines

• 2. The Indexing Program
• What is included? Some search engines claim to
  index every word from every page. But what do
  they consider a word?
  
• Stop words are often left out - articles and
  conjunctions. However, some engines don't index
  high frequency words such as web and Internet, or
  numbers.
• Some do not index frames - causing the loss of
  many relevant hits
  
• -example Frames

10
Web Search Engines

• 3. The Retrieval Engine
• The retrieval engine is the program that receives
  your query and
  
• searches the database to identify and deliver
  records that match
  
• your query.
• Two Major things are happening
• the retrieval engine identifies the matching
  records via a retrieval algorithm
  
• the retrieval engine arranges the retrieved items
  in a particular order to be displayed to you the
  user
  
• Retrieval Algorithms utilize matching criteria to
  determine which
  
• records contain particular words, phrases, and
  combinations. They
  
• may also match other user-specified criteria such
  as whether a
  
• page contains audio or film images.

11
Web Search Engines

• 4. The Graphical Interface (GUI)
• Users see and interact with an HTML-based
  interface.
  
• The interface allows the user to input a query
  but also provides a space for advertisers, add-on
  features such as directories, sites for stock
  market information, links to chat rooms, free
  email, etc. and links to help pages.
• May also offer special interfaces for searching a
  single type of file audio, image, video, etc.
  

12
Meta-Search Engines

• What are Meta-Search Engines? What are their
  limitations?
  
• A meta-search engine allows you to search several
  search engines and their databases at one time,
  often very quickly.
  
• Meta-search engines can be useful if you are
  looking for a unique term or phrase.
  
• BEWARE!
• Meta-search engines only spend a short time in
  each database
  
• and often retrieve only 10 of any of the results
  of any
  
• database queried
• Metacrawler Dogpile

13
General-Purpose Versus Specialized Web Search
Engines

• General-purpose Web search engines (AltaVista,
  Lycos, Google, et al.) retrieve hits on just
  about any topic from anywhere on the open Web.
  
• Specialized Web search engines may search only a
  single Web site or single area of interest.
  
• FirstGov retrieves only U.S. government
  information
  
• NewsIndex retrieves only current news items.
• Specialized Web search engine may be more precise
  than engines that search the open Web.
  

14
Disadvantages

• The Deep Web
• Large sections of the Web are invisible to Web
  search engines.
  
• All proprietary Web-based information
  resourcesAcademic Search Elite, biography
  Resource Center, etc.cannot be searched via Web
  search engines.
• Many open-access resources, such as PubMed,
  cannot be searched via Web Search engines.
  
• No search engine of any type can claim to search
  the entire Web.
  

15
Disadvantages

• Pay for Position Advertising
• Owners of Web sites may pay owners of Web search
  engines for prominent positioning.
  
• Widgets Unlimited might pay lycos.com so that
  WidgetsUnlimited.com is at the top of the list
  for any search involving the word widget.

16
Disadvantages

• Pay for Position Advertising
• Web search engines accept advertising.
• An ad for sinus medicine appears at the top of
  the results page following a search on the phrase
  runny nose.
  
• Pay-for-positioning and advertising compromise
  objectivity of Web search engines.
  
• Is this the best information available, or simply
  the best financed information?