End-User Software Engineering for System Administrators

1
End-User Software Engineering for System
Administrators

• Allen Cypher, Eben Haber, Eser Kandogan
• USER Group, Computer Science

2
Overview

• Field Studies of System Management
• Autonomic Task Manager for Administrator/A1
  System Management Tool Development
• Studies on Policy-based Autonomic Computing
• Field Studies
• Experimental Studies
• Ongoing and Future Work
• System Simulation Studies
• Activities for IT Management

3
System Management Today

• Complexity
• Large scale operations, complex layered IT
  infrastructures, variety of system interactions,
  many system management tools, various notions of
  configurations, permissions, etc.
• Risk
• Contractual obligations, high availability
  demands, unacceptable loss of data, dependence of
  modern life
• Cost
• People cost dominating over hardware and
  software, obstacle to future technology

We upgrade and patch systems all the time. We
have about 1,000 production systems that are Sun,
HP , Linux, and Dell."
4
Field Studies of System Management

• Web Hosting Southbury
• 1 Week

• Web Hosting Southbury
• 1 Week

• Web Hosting, Data Management, Operating System,
  Security, Storage, Data Center Operations
• 14 Visits, 6 sites
• Surveys ( 100 people)
• Observations ( 50 days)
• Video ( 300 hours)
• Interviews ( 30 people)
• Diary ( 10 months)
• Qualitative and quantitative analysis

• Data Management
• Poughkeepsie
• 3 Days

• Data Management
• Charlotte
• 3 Days

• Web Hosting
• Boulder
• 3 Days 1 Eve

• Web Hosting
• Boulder
• 1 Week

• Operating system
• Boulder
• 3 Days

• Security
• Urbana
• 3 Days

• Storage
• Boulder
• 3 Days

• Security
• Urbana
• 1 Week

• Storage
• Greenbelt
• 3 Days

• Data Center
• Boulder
• 3 Days

• Storage
• Greenbelt
• 3 Days

• Data Center
• Boulder
• 3 Days

5
A Case from the Field
How can I put a number and date on the same
line?
6
How can I put a number and date on the same
line?

• Task Resolve customer issue so that problem does
  not repeat itself for three consecutive days
• People Rob (web admin), Jack (architect), Andy
  (operating system), Managers, Tech Support, etc.
• Problem Intermittent connection spikes between
  web server and application server that lead to
  unresponsive application

7
The Crit Sit
8
How can I put a number and date on the same
line?

• Troubleshooting intermittent, multi-system
  problems is hard.
• Sysadmins build tools to increase shared
  situational awareness.
• Tool building is a collaborative effort too.
• There is a variety of tool building expertise
  among admins.
• Existing scripting languages aren't aimed at
  non-programmers.
• Scripting languages dont handle input output
  and errors very well.

9
Issues in Tools and Tool Building

• Diversity Every shop is different with its own
  processes and infrastructure. Vendor tools
  sometimes are just not doing it.
• Integration There isnt one tool that does all
  the things I need to do. Every vendor provides
  its own management utilities.
• Transparency I tend to learn the guts of things
  in a CLI as close to the heart of the matter as
  possible , and then translate it into a GUI
• 24x7 Coverage We have scripts that monitor our
  systems and notify us when something goes wrong.
  We cant be on the job all the time!
• Trust I prefer the CLI. These tools seem to be
  the most truthful and accurate for
  administration.
• Efficiency I need to reset 150 passwords every
  week!
• Risk We cant afford to make mistakes as we
  move from test to staging to production servers.