VERMONT STATE COLLEGES WHO WE ARE

1
VERMONT STATE COLLEGESWHO WE ARE
2
VERMONT STATE COLLEGESSHARED SERVICES

• From the data center (located in a state-owned
  building)
• Wide-area network single Internet connection
• ERP
• Identity Access Management
• Portal
• IT Help Desk
• Data warehouse
• Conference events application
• Enterprise IT Staff
• 6 programmers 3 system admins 1 network 1
  security data center

3
VERMONT STATE COLLEGESDISASTER RECOVERY

• First plan in 2007
• Approved by Council of Presidents
• Reviewed twice a year - updated February, 2011
• Key feature assumed local and business process
  planning to accommodate the absence of critical
  systems for a period of up to two weeks
• In-scope and out-of-scope clearly articulated

4
VERMONT STATE COLLEGESBUSINESS CONTINUITY

• Plans for admissions, accounts payable, accounts
  receivable, finance, financial aid, payroll and
  registrars processes
• Reviewed and updated annually
• Some business areas could go for several weeks
  without extraordinary measures
• Some could only function for few hours or days

5
IRENE
6
VERMONT STATE COLLEGESWHAT HAPPENED

• On Aug 28, 2011, at 813 PM, "Pollak, Dianne M"
  ltdianne.pollak_at_vsc.edugt wrote
•  
• gt Just got a text from michael... First floor of
  the hospital is under water... They are
  evacuating randall st. We are on generator.
• gt
• gt Dianne Pollak
• gt Director of Administrative Information Systems
  Vermont State Colleges

7
VERMONT STATE COLLEGESWHAT HAPPENED
8
VERMONT STATE COLLEGES-WHAT HAPPENED

• By 6 am on 8/29 water had filled basement and
  up to 5 feet of the first floor
• Generator under water power gone
• Entire state office complex flooded
• Roads to town closed

9
(No Transcript)
10
VERMONT STATE COLLEGESBUSINESS IMPACT

• Two colleges had started classes
• Two colleges were about to start classes on
  Tuesday
• One college was still in marketing mode

11
VERMONT STATE COLLEGESCOMMUNICATIONS
12
VERMONT STATE COLLEGESSHORT-TERM SOLUTIONS

• Monday - Emergency web site up by Monday
• Monday - Alternate e-mail addresses collected
• Tuesday - Wide-area network and Internet restored
• Monday - Colleges using internal e-mail
• Monday-Thursday Some paper paychecks
  handwritten
• Tuesday-Friday Air-conditioning and server
  issues
• Friday morning ERP and other critical systems
  restored

13
(No Transcript)
14
VERMONT STATE COLLEGESEARLY LESSONS

• Even if you cant afford what you really need,
  have a plan!
• Business continuity plans had some issues
• Never underestimate the power of denial
• Good relationships sustain you bad ones are
  killer
• Stay on the lookout for good ideas and be
  flexible
• Its true never waste a good disaster
• Keep track of whos doing what
• Communications are critical difficult

15
Data center
16
DATA CENTER GENERATOR...
High Water Mark
17
VERMONT STATE COLLEGESEARLY LESSONS

• Take notes everywhere, in every conversation
• Have a conference call system set up
• Texting is critical (and set up those contacts!)
• Have an alternate e-mail address for all critical
  contacts IT and priority business areas
• Have a plan for alternate web site hosting
• Staff check-in food and sleep
• Staff in different places on the resilience scale
• As everyone gets tired, mistakes will be made
• Danger in false sense of stability/reliability

18
UP NEXT SNOWTOBER

• From The EDUCAUSE CIO Constituent Group Listserv
  mailtoCIO_at_LISTSERV.EDUCAUSE.EDU On Behalf Of
  Theresa RoweSent Thursday, November 03, 2011
  844 AMTo CIO_at_LISTSERV.EDUCAUSE.EDUSubject
  Re CIO Snowtober
• Thanks so much for taking the time to write out
  the details of your experiences.  We all learn
  from these posts.  Our experiences with
  electrical power failures and generators have
  been very difficult to manage.  Every situation
  has been different, so you really need thinking,
  problem-solving people on hand, rather than a
  scripted action.  You've given us more to think
  about.Theresa

19
Thinking, problem solving people
20
VERMONT STATE COLLEGES OPPORTUNITIES

• Reinforce the importance of both disaster
  recovery and business continuity planning and
  underscore the differences
• Make the case for disaster recovery
  resources-less difficult in the reality of
  business interruption
• Change things that were unchangeable
• Focus on whats important people and mission

21
VERMONT STATE COLLEGESTODAY

• Access to building restricted to 2-3 staff
• Off generator data center is only room with
  power
• Legislature to determine future use of the
  complex
• IT Staff distributed across the state
• Building two data centers looking at
  co-location
• Dealing with displaced employees
• Re-projecting the annual project list
• Getting back to normal - IT reporting cycle