Lessons Learned from disaster recovery

1
Lessons Learned from disaster recovery
6th APGridPMA in Taipei
Jinny Chien April 20, 2009
2
Motivation

• ASGCCA encountered the accident in February and
  how to avoid the same situation happening
• Do we have sufficient backup procedure
• Is our CA server at the safe place
• Do we have the standard policy or incident
  response procedure to introduce the recovery

3
ASGCCA Event

• Time 900 Feb-25-2009 UTC
• Event description the unexpected accident on
  the part of data center at ASGC, all on-line
  services were shut down including ASGCCA web
  server.
• Result ASGCCA certificate activities were
  down. CRL did not publish in time.

4
Process

• 900 Feb 25 UTC Sent an EGEE broadcast to all
  ROC managers, VO managers, WLCG users, APGridPMA
  members, ASGCCA users
• 200 Feb 26 UTC Sent an announcement to
  IGTF-RAT and IGTF-general lists. Try to recover
  ASGCCA web page
• 1200 Feb 26 UTC Moved ASGCCA web and CA server
  (offline) to the safe place and connected to the
  Internet.
• 1600 Feb 26 UTC ASGCCA web site was up. Sent
  the announcement to IGTF-RAT, IGTF-general and
  ASGCCA user lists

5
Review the process

• Feb-25-2009 UTC ASGCCA web was down and sent an
  announcement to APGridPMA, ASGCCA users,
  IGTF-RAT, IGTF-general
• Feb 26 UTC Recovered and ASGCCA web site was
  up. Sent the final announcement and checked all
  CA activities well.
• Total process is two days

6
Basic Recovery procedure

• Evaluate the scope of this disaster and how many
  days to recover
• Send the notification to IGTF-RAT, IGTF-general,
  APGridPMA members and your end entities. The
  matter should be described the disaster and
  schedule
• Recovery activities
• Check all CA activities well and CRL will be
  published regularly
• Re-work and send the final announcement to
  IGTF-RAT, APGridPMA member, IGTF and your end
  entities.

7
IGTF-RAT

• The International Grid Trust Federation (IGTF)
  Risk Assessment Team (RAT) is responsible for
  assessing risk and setting time and deadlines for
  response and action for concerns and
  vulnerabilities.
• Email address igtf-rat_at_eugridpma.org
• Members
• APGridPMA Yoshio Tanaka , Jinny Chien
• EUGridPMA Jens Jensen, Willy Weisz , David
  Groep, Sajjad Asghar
• TAGPMA Jim Basney , Vinod Rebello, Jim
  Marsteller
• Public webpage
• http//tagpma.es.net/wiki/bin/view/IGTF-RAT

8
Conclusion

• Please backup the CA server and web regularly
• The backup archive should be kept at the safe
  place
• Write down the recovery procedure for your CA
  activities

9
Discussion

• if the CA server and web destroy at the same
  time?
• To evaluate the disaster and plan a schedule
• Ask for help to IGTF-RAT
• Should we have the incident response procedure ?
• What is the time range if CA encounters any
  accident?

10
Thanks for your listening