Lost In Cyberspace: Search Engine User Errors

1
Lost In Cyberspace Search Engine User Errors

• Presented by Maria Serapiglia
• November 28, 2001

2
Why Study Search Engines?

• Intensity of information on the Internet
• Powerful tools for accessing information
• Rely on categorization and user input
• Success intersection of the two

3
Smarter Search Engines?

• Possibilities for artificial intelligence
  application to search engine technology
• Engine that learns about your search as it goes
• Makes suggestions for successive searches to
  create effective results

4
My Research Project

• Objective Begin to understand common errors made
  by Internet search engine users
• Hypothesis Users have difficulty using search
  engines effectively, particularly when iterative
  searches are necessary.
• Result low success, high frustration

5
Method

• Users conducted two search tasks
• One relatively simple, but not totally obvious
• One more complex, requiring successive searches
• Six subjects
• Representing a range of Internet/search engine
  experience levels
• Half male/half female, age range 24-54

6
Data Collection

• Pre-task survey
• Field observation
• Direct questioning in field
• Post-task survey

7
Pre-task Survey

• Experience using computers
• Technology comfort level
• Search engine use
• Frequency of searching
• For what type of information
• Basic demographics
• Age, gender, education

8
Field Observation

• Researcher observation and screenshots
• Actions taken
• Terms entered
• Length of time spent on a page
• Reaction to results
• Affect, frustration level, etc.

9
Direct Questioning

• Actions and reactions
• Tell me what youre thinking/keep talking
• Why are you using that term?
• What do you think of these results?
• What will you do next?
• How did you decide you were done?

10
Post-Task Survey

• How was the experience for you?
• How did your results compare to your
  expectations?
• What was frustrating or hard about the searching?
• How could the search engine have helped you be
  more successful?

11
Findings Participant Ability

• Ability to successfully complete searches varied
  widely
• Success consistent with experience level
• Success consistent with comfort level
• Simple task accomplished more easily than complex
  search

12
Findings Functional Errors

• Failure to use sign to add terms
• Not using quotes to group words together
• Replacing terms instead of adding
• Getting lost in results, losing thread of search

13
Findings Info-Processing Errors

• Starting too broad or vague
• Difficulty identifying relevant terms for
  refining search
• Not considering quantity of results when
  evaluating success of search
• Ill-defined criteria for determining quality of
  results or evaluating source validity

14
Findings User Satisfaction

• Users typically expected the search engine to do
  better than it did
• Iterative searches for complex information
  generated the most frustration
• Greater number of hits increased
  dissatisfaction with search results

15
User Differences

• Less-experienced searchers
• More functional errors than experienced
• Did not use tools
• Replaced rather than added terms
• Greater frustration with iterative searching
• Experienced searchers
• Tended to use tools to enhance search
• Used more specific terms, added terms

16
Notable Finding

• Information processing errors were equal for both
  types of user
• Expectations for search engines clearly were not
  met
• All participants expressed in some way that the
  search engine could have provided a more
  thoughtful response

17
Conclusions

• Errors are common in search engine use
• Common functioning errors might be preventable
  with better user interface
• Iterative searches are difficult for users
• Search engine effectiveness could be improved
• Think more like users
• Users understand engines logic

18
Issues for Further Study

• Quantitative validation of observations
• Larger sample to understand impact of experience
• Differences between search engines?