WiscMail Problems and Plans

1
WiscMail Problems and Plans

• Pat Brennan
• Roger Hanson

2
WiscMail Outage

• What happened
• Technical Issues
• Communication Issues
• Other concerns

3
WiscMail What Happened

• 1215 a.m. 3/1 Outage begins
• 230 a.m. CPU offline with OS command
• 300 a.m. Mail store recovered - system operating
• 330 p.m. 3/1 Mail store crashes again.
  Hardware indicated
• 100 a.m. 3/2 Platform tech. Investigates
• 130 a.m. 3/2 System restarted again - crashes
  after 30 minutes

4
WiscMail What Happened

• 1000 a.m. 3/2 Platform and Mail teams meet, CPU
  is physically removed from system
• 3/2 Sunday 11 a.m. Multiple level recoveries -
  system up at 3p.m. System delivers 143K inbound
  messages.
• 3/3 715 a.m. Call to Sun on reconstruct
  problems. 815 addition errors found - Mail
  store closed for recovery.

5
WiscMail What Happened

• 3/3 900 a.m. conference call with Sun. Sun
  advises repair process. Remove index file from
  each account. Perform folder level re-index.
• 3/3 1130 a.m. Repair process started.
• 3/5 545 a.m. Reconstruction completes.
• 3/5 600 a.m. SMTP processes restarted to
  deliver 500,000 message backlog to Mail Store.
• 3/5 1200 noon. Logins enabled.

6
WiscMail Technical Issues

• Need new steps to diagnose failing hardware
• Speed up reconstruct processes
• Use database index snapshot to reduce re-index
  time.
• Automate account purge process add SPAM
  filtering
• Balance Mail store accounts

7
WiscMail - Communications

• Notify customers on Saturday and Sunday via
  Portal of problem
• Notify customers via portal login module of
  problems
• Provide pop-up alert on Portal front page
• Use alternate communication paths to notify
  customers

8
Other Concerns

• Email Backup
• ESS
• Records Retention
• Other?

9
Next Steps

• WiscMail Forum 3/13 1pm Union South
• Invitation from Annie and WiscMail Team
• Informal discussion

10
WiscMail Plansbefore 3/1

• Enhance Security
• Quota implementation
• SPAM filtering
• Departmental Mail Service

11
WiscMail Security

• SSL Login for POP IMAP - does not include
  HIPPA
• Deadline April 15th
• Client Reconfiguration
• Migration Utility
• Notification process
• http//www.doit.wisc.edu/WiscMail/configuring/inde
  x.asp

12
WiscMail Quota

• Increase quote to 100 MB
• Begin Quota enforcement 4/1
• Create warning messages for 80, 90, 95
• Coordinate with Help Desk
• Provide WebMail Alert at 95

13
WiscMail - SPAM

• Change in project direction
• ActiveStates PureMessage leading candidate
• Provide HotMail type interface
• Opt in system
• Pilot testing underway
• http//activestate.com
• Implementation in summer

14
WiscMail Dept. Mail option

• MadMail Alternative
• Features
• New Mail domains
• Aggregate quote management
• Backup
• Group and Role based accounts
• Non UW Accounts
• Local Administration

15
WiscMail Dept. Mail Option

• Also, provide anti-virus and SPAM filtering
• Mid-July Availability

16
WiscMail Cluster