Search Engines and the Invisible Web

1
Search Engines and the Invisible Web

• Steven M. Cohen
• Rivkin Radler, LLP

Growing Great Librarians NYLA/YSS Conference
2
What We Will Discuss

• Who Cares about this stuff?
• Whats New in the Search Engine Biz.
• Have you seen these?
• Why not just Google it?
• The Invisible Web

3
Who cares?
4
Why it Matters

• You help children and Young Adults find
  information.
• They trust your abilities.
• You owe to them your knowledge of search engines.
• You are their search engine.
• Librarian, know thyself.

5
(No Transcript)
6
(No Transcript)
7
Whats Happening in the Biz
8
Whats Happening in the Biz!

• Northernlight is finished.
• Alltheweb and Altavista have been bought by
  Overture.
• Meta search engines are making a comeback.
• The Im bigger than you are fight is over.
• Engines are trying to better their algorithms.

9
Whats Happening (contd)

• Everything is Google, Google, Google.
• Ask Jeeves is still going strong.
• Yahoo is still in the running.
• Advanced search sections are becoming more
  advanced.
• Subject specific engines are all the rave.

10
Have you seen these engines?
11
Teoma
12
Teoma
13
GigaBlast
14
GigaBlast
15
IxQuick
16
IxQuick
17
Feedster
18
Feedster
19
Why Not Just Google?
20
Why Not Just Google?

• Google only carries a small portion of the Web.
• Do you rely on only one reference book?
• Other engines might have indexed what Google
  missed.
• Googles database may not be as fresh.
• What if

21
(No Transcript)
22
Think Im Wrong About Google?

• Pick a Random URL
• Place it in the Google search box with info
  before the URL
• If it comes up, it is indexed by Google
• If not, then it is not indexed

23
http//deseretnews.com/movies/view/1,1257,90000047
,00.html
24
Not indexed by Google, but...
25
http//www.mrqe.com
26
(No Transcript)
27
The Invisible Web
28
The Invisible Web

• Web information that does not get indexed by the
  major search engines.
• Hidden mostly in databases or have robot.txt file
  attached
• Data created on the fly from the backend
  (cgi-bin, etc)
• More than ¾ of information on the Web is part of
  the IW.

29
Examples of the IW

• Online telephone and address databases
• News engines
• Professional look-up services (AMA)
• Movie and Book Reviews
• Education databases (ERIC)
• Medical databases (Medline)

30
http//www.accessdata.fda.gov/scripts/oc/cftobacco
/search.cfm
31
http//legacy.library.ucsf.edu/
32
http//www.dawd.com
33
http//www.schools.com/ssearch/schoolsearch.cfm
34
http//www.jcsm.com/areacode.asp
35
Invisible Web Resources

• http//www.invisibleweb.com
• http//www.invisible-web.net
• http//www.profusion.com
• http//www.academicinfo.net
• http//www.lii.org
• http//infomine.ucr.edu
• http//www.completeplanet.com