1. What do E-mail System Administrators Do? William Kakes Calvin Ling Leonard Chung Aaron Brown EECS Computer Science Division University of California, Berkeley June 2003

2. Slide 2 Motivation Running studies on Undo – Need to ensure that human administrators find it useful. That they: » understand how to use the tool. » use them when appropriate. » know to not attempt to use them when inappropriate. – Initial target domain: e-mail services. Need a good understanding of the problems faced by real administrators!

3. Slide 3 Approach: Survey Online survey of practicing e-mail sysadmins. Distributed to SAGE mailing list and local contacts. Incentive for completing survey.

4. Slide 4 Survey Design Survey questions covered 5 areas: – Demographics. – Profile of common administrative tasks. – Identification of challenging tasks. – Anecdotal problem reports. – Free response. Used interviews to develop survey.

5. Slide 5 Incentive: Encourage Participation Provide desirable, community-specific incentive. – Chance to win 1 of 5 $50 gift certificates to Amazon.com. – …or a copy of Computer Architecture: A Quantitative Approach! Allow (and guarantee) anonymity. – Very important when asking about people’s mistakes. – Protect whistle-blowers, corporate reputations. – Results stored by randomly-generated ID.

6. Slide 6 Incentive: Make it “Easy to Take” Basic HTML format. – Text only, no Javascript, etc. required. – Appeal to broadest base of sysadmins. Participant guidance. – Likert-scale questions whenever possible. – Every question had “I don’t know” option. Speed. – Single-page. – Open-ended questions constrained.

7. Slide 7 Selected Questions What is your supported environment? How many users? – Daily volume? – Number of complaints? – Amount of downtime they’ll accept? » Scheduled vs. unscheduled? What tasks? – Diagnosis vs. repair? – Number of problems?

8. Slide 8 First Trial: Response Distributed to SAGE mailing list and local contacts. 68 responses received. – …mostly complete with useful answers. – Only 21 “I don’t know” responses total. Many responses from educational admins. – Yet still lots from within industry.

9. Slide 9 Selected Results: Demographics Typical administrator experience: 5+ yrs. – Running on a variety of platforms, mostly Unix-based. – Mainly serving internal users. Typical user base of 100-2500; 1K-1M messages per day. – Messages per day results pretty evenly spread along this range.

10. Slide 10 Results: Acceptable Downtime

11. Slide 11 Results: Acceptable Downtime (2)

12. Slide 12 Results: Acceptable Downtime (3) Discussion: – Users have less tolerance for unscheduled than scheduled downtime, but not by much. » Rather low tolerance in either case. – Reminder: this is for regular hours. » Most users don’t care about off-hours much.

13. Slide 13 Results: Critical Affected User %

14. Slide 14 Results: Duties and User Requests

15. Slide 15 Results: Number of Problems

16. Slide 16 Results: Diagnosis vs. Repair

17. Slide 17 Results: Confidence Time

18. Slide 18 Results: Task Profiles Breakdown of common and challenging tasks Common TasksChallenging Tasks Configuration (56%) Upgrades/ Patches (12%) Tool Dev. (1%) Repairs (15%) Backup & Restore (3%) Monitor/ Test (10%) User Ed. (<1%) Other (3%) Filter Installation (37%) Platform Change/ Upgrade (26%) Tool Dev. (6%) Config. (13%) Other (6%) User Ed. (4%) Architecture Changes (7%) (151 total)(68 total)

19. Slide 19 Results: E-mail Problem Anecdotes Breakdown of causes of service outages All ProblemsLost E-mail Configuration problems (25%) Upgrade- related (17%) Operator error (8%) User error (8%) External resource (8%) Software error (8%) Hardware/ Env’t (17%) Unknown (8%) (12 reports)(60 reports) Hardware/ Env’t (20%) External resource (15%) Software error (7%) Configuration problems (13%) Upgrade- related (8%) Operator error (7%) User error (<2%) Unknown (<2%) DoS/Resource Exhaustion (22%) Network (5%)

20. Slide 20 In Their Own Words… High (often unrealistic) user expectations. – “Users do not understand email; they expect it to be instantaneous, reliable, and single-hop.” – “Users have an intimate relationship with their email…when email is involved users get quite excitable.” – “Many of our users would be far less affected by losing their phone for a week than by losing email access for a day (or even hours).”

21. Slide 21 In Their Own Words… (2) Spam is the key problem for e-mail admins. – “Spam…is our largest problem. Our machines are well spec'd and very stable, until they are hit with too much email all at once!” – “I want to find the spammers and toss them off a 12 story building, then drive over them with a truck a few times.” – “Spammers should be charged for every piece of junk mail we have to accommodate in load planning, users assistance, and bounce tracing. I’d be able to buy Bill Gates at only $0.01 per junk message.” …but problems with viruses and worms, too.

22. Slide 22 In Their Own Words… (3) New technologies are useful – “Automation of all sorts has been improving performance and increasing reliability of our servers.” – “If you choose the right software, you don’t generally have any problems.” – “Email is old hat, actually.” …more are needed – “Testing correctness of configurations is one of the hardest things to do.” – “No matter how much work you do with it, you are never sure that it is secure and stable.” – “sendmail.cf is a bitch”

23. Slide 23 Discussion: Results Configuration issues dominate for e-mail. – Primarily SPAM- and virus-management, and platform deployment tasks. – Errors here can cause degraded service and lost mail. – Little system support to make these tasks easy » No undo capability for configuration changes or upgrades, difficult to understand impact of configuration changes. – Can take days to weeks for administrators to be confident.

24. Slide 24 Discussion: Results (2) Users have less tolerance for unscheduled than scheduled downtime. – …but not by much. Diagnosis takes more time than repair. – Another area with little system support. SPAM-related DoS a significant problem.

25. Slide 25 Discussion: Results (3) E-mail administration can be high stress. – As soon as any users start seeing problems, must fix them, and quickly. – Errors can be silent. – Undo capability could alleviate some of this.

26. Slide 26 Conclusions Surveys are a cheap and easy way to get insight into the world of practicing sysadmins. – But must be created with unique characteristics of sysadmin community in mind. E-mail administrators challenged by configuration and deployment issues. – Mostly related to SPAM control. Configuration management, diagnosis are key areas to tackle to improve sysadmin user experience.

27. Slide 27 What Next? Use survey results to develop realistic and representative scenarios. – Run experiments with Undo tool, placing participants into these scenarios. These experiments will be conducted soon! – Hopefully this summer in Berkeley.

28. What do E-mail System Administrators Do? Acknowledgements – Berkeley/Stanford ROC Research Group – Professor David Patterson For more info – http://roc.cs.berkeley.edu/projects/emailsurvey/ – abrown@cs.berkeley.edu

29. Backup Slides

30. Slide 30 Results: Number of Users

31. Slide 31 Results: E-mail Volume