What do Email System Administrators Do

1
What do E-mail System Administrators Do?

• William Kakes
• Calvin Ling
• Leonard Chung
• Aaron Brown
• EECS Computer Science DivisionUniversity of
  California, BerkeleyJune 2003

2
Motivation

• Running studies on Undo
• Need to ensure that human administrators find it
  useful. That they
• understand how to use the tool.
• use them when appropriate.
• know to not attempt to use them when
  inappropriate.
• Initial target domain e-mail services.
• Need a good understanding of the problems faced
  by real administrators!

3
Approach Survey

• Online survey of practicing e-mail sysadmins.
• Distributed to SAGE mailing list and local
  contacts.
• Incentive for completing survey.

4
Survey Design

• Survey questions covered 5 areas
• Demographics.
• Profile of common administrative tasks.
• Identification of challenging tasks.
• Anecdotal problem reports.
• Free response.
• Used interviews to develop survey.

5
Incentive Encourage Participation

• Provide desirable, community-specific incentive.
• Chance to win 1 of 5 50 gift certificates to
  Amazon.com.
• or a copy of Computer Architecture A
  Quantitative Approach!
• Allow (and guarantee) anonymity.
• Very important when asking about peoples
  mistakes.
• Protect whistle-blowers, corporate reputations.
• Results stored by randomly-generated ID.

6
Incentive Make it Easy to Take

• Basic HTML format.
• Text only, no Javascript, etc. required.
• Appeal to broadest base of sysadmins.
• Participant guidance.
• Likert-scale questions whenever possible.
• Every question had I dont know option.
• Speed.
• Single-page.
• Open-ended questions constrained.

7
Selected Questions

• What is your supported environment?
• How many users?
• Daily volume?
• Number of complaints?
• Amount of downtime theyll accept?
• Scheduled vs. unscheduled?
• What tasks?
• Diagnosis vs. repair?
• Number of problems?

8
First Trial Response

• Distributed to SAGE mailing list and local
  contacts.
• 68 responses received.
• mostly complete with useful answers.
• Only 21 I dont know responses total.
• Many responses from educational admins.
• Yet still lots from within industry.

9
Selected Results Demographics

• Typical administrator experience 5 yrs.
• Running on a variety of platforms, mostly
  Unix-based.
• Mainly serving internal users.
• Typical user base of 100-2500 1K-1M messages
  per day.
• Messages per day results pretty evenly spread
  along this range.

10
Results Acceptable Downtime
11
Results Acceptable Downtime (2)
12
Results Acceptable Downtime (3)

• Discussion
• Users have less tolerance for unscheduled than
  scheduled downtime, but not by much.
• Rather low tolerance in either case.
• Reminder this is for regular hours.
• Most users dont care about off-hours much.

13
Results Critical Affected User
14
Results Duties and User Requests
15
Results Number of Problems
16
Results Diagnosis vs. Repair
17
Results Confidence Time
18
Results Task Profiles

• Breakdown of common and challenging tasks

Common Tasks
Challenging Tasks
(151 total)
(68 total)
User Ed. (lt1)
Other(3)
Monitor/Test (10)
User Ed. (4)
Other (6)
Upgrades/Patches (12)
PlatformChange/Upgrade(26)
ArchitectureChanges (7)
Backup Restore (3)
Repairs(15)
FilterInstallation(37)
Configuration(56)
Config. (13)
Tool Dev.(1)
Tool Dev. (6)
19
Results E-mail Problem Anecdotes

• Breakdown of causes of service outages

All Problems
Lost E-mail
(12 reports)
(60 reports)
Unknown (lt2)
Unknown (8)
Configurationproblems (13)
Configurationproblems (25)
DoS/ResourceExhaustion (22)
Hardware/Envt (17)
Upgrade-related (8)
Operator error (7)
Software error (8)
User error (lt2)
Upgrade-related (17)
Hardware/Envt (20)
Externalresource(15)
Externalresource (8)
Network(5)
Operator error (8)
Usererror (8)
Software error (7)
20
In Their Own Words

• High (often unrealistic) user expectations.
• Users do not understand email they expect it to
  be instantaneous, reliable, and single-hop.
• Users have an intimate relationship with their
  emailwhen email is involved users get quite
  excitable.
• Many of our users would be far less affected by
  losing their phone for a week than by losing
  email access for a day (or even hours).

21
In Their Own Words (2)

• Spam is the key problem for e-mail admins.
• Spamis our largest problem. Our machines are
  well spec'd and very stable, until they are hit
  with too much email all at once!
• I want to find the spammers and toss them off a
  12 story building, then drive over them with a
  truck a few times.
• Spammers should be charged for every piece of
  junk mail we have to accommodate in load
  planning, users assistance, and bounce tracing.
  Id be able to buy Bill Gates at only 0.01 per
  junk message.
• but problems with viruses and worms, too.

22
In Their Own Words (3)

• New technologies are useful
• Automation of all sorts has been improving
  performance and increasing reliability of our
  servers.
• If you choose the right software, you dont
  generally have any problems.
• Email is old hat, actually.
• more are needed
• Testing correctness of configurations is one of
  the hardest things to do.
• No matter how much work you do with it, you are
  never sure that it is secure and stable.
• sendmail.cf is a bitch

23
Discussion Results

• Configuration issues dominate for e-mail.
• Primarily SPAM- and virus-management, and
  platform deployment tasks.
• Errors here can cause degraded service and lost
  mail.
• Little system support to make these tasks easy
• No undo capability for configuration changes or
  upgrades, difficult to understand impact of
  configuration changes.
• Can take days to weeks for administrators to be
  confident.

24
Discussion Results (2)

• Users have less tolerance for unscheduled than
  scheduled downtime.
• but not by much.
• Diagnosis takes more time than repair.
• Another area with little system support.
• SPAM-related DoS a significant problem.

25
Discussion Results (3)

• E-mail administration can be high stress.
• As soon as any users start seeing problems, must
  fix them, and quickly.
• Errors can be silent.
• Undo capability could alleviate some of this.

26
Conclusions

• Surveys are a cheap and easy way to get insight
  into the world of practicing sysadmins.
• But must be created with unique characteristics
  of sysadmin community in mind.
• E-mail administrators challenged by configuration
  and deployment issues.
• Mostly related to SPAM control.
• Configuration management, diagnosis are key areas
  to tackle to improve sysadmin user experience.

27
What Next?

• Use survey results to develop realistic and
  representative scenarios.
• Run experiments with Undo tool, placing
  participants into these scenarios.
• These experiments will be conducted soon!
• Hopefully this summer in Berkeley.

28
What do E-mail System Administrators Do?

• Acknowledgements
• Berkeley/Stanford ROC Research Group
• Professor David Patterson
• For more info
• http//roc.cs.berkeley.edu/projects/emailsurvey/
• abrown_at_cs.berkeley.edu

29
Backup Slides
30
Results Number of Users
31
Results E-mail Volume