The National Finance Center and Hurricane Katrina

1
The National Finance Center and Hurricane
Katrina

• John White, Deputy Director
• June 2006

2
USDA, Office of the Chief Financial Officer

• Chief Financial Officers View -
• USDA Characteristics
• 110,000 employees _at_ 14,000 offices and field
  locations
• 300 national programs
• 77B in annual spending
• 128B in assets
• 100B in loans
• Program Management Responsibility Includes
• Financial leadership across the enterprise
  roughly equivalent in size to 6th largest private
  sector firm in U.S.
• Financial policy and planning
• Financial systems and operations
• Government-wide service delivery through the
  National Finance Center (NFC)

3
Scope of USDA CFO Operations inNew Orleans
(co-located)

• National Finance Center (NFC)
• Government-wide payroll/personnel for 585,000
  employees
• Government-wide Human Resource services
• Government-wide Health Benefits Programs
• Disbursements and Collections
• Data Center Services
• At time of Katrina, over 1300 employees
• CFO Controller Operations Division
• Provides financial services to USDA agencies,
  including administrative payments (2.5
  million annually), interagency payments,
  reconciliations, and vendor file
• Assembles USDA financial reports, including FACTS
  I II, and annual consolidated financial
  statements
• At time of Katrina, almost 300 employees
• CFO Financial Systems Division
• Operates the corporate consolidated financial
  system for all USDA agencies
• Operates administrative systems, such as travel
  and property, for all USDA agencies
• Creates and reconciles data extracts for
  consolidated audit and financial statements
• At time of Katrina, over 70 employees

4
Prolog Accountability Guiding Principles

• Embrace that you are in charge and responsible
  for your organization, its finances, and people.
• Be prepared to go it alone, do what is necessary
  to get the job done and care for your staff. In
  a major situation, there may be no one else to
  consult about how to proceed.
• Never forget that during a crisis, people demand
  leadership, decisions must be made--often in a
  vacuum, stick to the plan but be flexible.

5
Preface Risk Management Guiding Actions

• Managing to deliver the expected business results
  by considering risks in doing what we do.
• Focusing on most significant risks associated
  with the nature of the business
• Economic controls testing based on what is at
  stake
• Operational environmental, internal, natural,
  compliance, technical, organizational, and human
  risks
• Keys to successful risk management
• Understand the risk profile of the business
• Leadership sets the tone
• Integrate risk management into day-to-day
  practices and decisions
• Evaluate on clarity, transparency, integrity,
  performance

6
Preface Business ContinuityApproach Prepare!

• Business Impact Assessment
• Disaster Recovery Plan
• Business Continuity Plan
• Uninterruptible Power Supply
• Emergency Power Supply
• Two annual NFC drills
• USDA and Government wide drills
• Three historical near misses

7
(No Transcript)
8
Hurricane Katrina
Initial Timeline

• Activated DR on Friday evening
• Deployed Advanced Teams on Saturday morning
• Completed payroll late Saturday evening
• Shutdown data center and facility early Sunday
• Made disaster declaration Sunday evening
• Began implementation of COOP on Monday

9
COOP Concept
Concept of operations

• Subscription service for data center,
  workstations, and bulk print and mail
• Philadelphia site used for data center and 101
  data center staff seats
• Grand Prairie site used for 150 business
  operations seats
• Plans all geared to loss of New Orleans
  facilities

10
(No Transcript)
11
Realities of Katrina
Initial reactions

• Implications were larger than anyone imagined
• This would be a long-term situation
• Immediate focus was
• Service to the customers
• Accountability and caring for employees
• Fiscal accountability
• Balance delivery and personal needs

12
Oversight Priorities
View from the top

• Incorporating local efforts into Departmental and
  Government-wide efforts
• Keeping long-term, big picture in focus
• Finding ways to add value while allowing those on
  the line to keep things moving
• Policies
• Resources
• Contracting
• Financial
• Liaison
• Monitoring and Reporting

13
Locating and Deploying Staff
First 14 days

• Local telecommunications infrastructure mostly
  inoperable
• Pre-planned out-of-town contact numbers
  functioning
• No single data base for locating people in
  shelters
• Travel impacted by roads and gas shortages
• Family issues had to be addressed

14
Expanding Capabilities to Meet Long-Term
Deployment
First 14 days

• Duration of COOP means more seats needed
• Trailers acquired in Atlanta
• USDA sites used in D.C., Kansas City, Roslyn, and
  Alexandria
• Customer sites in Birmingham and Fairfax
  leveraged
• Secure connectivity and postal mail were two key
  infrastructure challengesto achieve

15
First Job, First
First 14 days

• Critical data center resources are recovered
• Essential services are restored
• Payroll is made timely while migrating 60,000 new
  payees (565,000 total)
• Financial systems returned online

16
Settling In for the Long Haul
First 14 days

• Subscription service requires sharing of
  resources during 6-week recovery period
• Essential services require dedicated resources
• Impact of Katrina implies 6 months not 6 weeks
• Customer dedicated connectivity key issue
• Decision made to buildout in cold site

17
Focus on Service Delivery
First 14 days

• Resuming all services requires that all resources
  (human, technical, and other) be available
• Service delivery focus isfor Agency customers,
  payees, vendors, and employees
• Give employees the challenge and the toolsand
  stay out of their way

18
New Orleans Critical to a Timely Return to Full
Service
Next 60 Days

• Returning to full service required a full staff
• Deployed status had limited seat availability,
  some security concerns, was costly, and came at a
  personal sacrifice to many
• The New Orleans facility has the infrastructure
  to address business needs and support the
  additional population

19
Reconstitution in New Orleans
Next 60 Days

• Housing, family issues, telecommunications,postal
  services, and local infrastructure had to be
  addressed
• Security
• Logistics
• Habitation necessities
• Health care
• Family friends, FEMA trailers, and cruise ships
  address most housingneeds

20
Its Nice to Have Friends
Keys to success

• USDA Secretary Johanns and entire USDA family
  strongly supported COOP needs and the employees
• Tremendous outreach by Federal colleagues, local
  Government, local organizations, and peoplein
  deployed locations
• Food, clothing, education opportunities,
  friendship, and other assistance

21
Its Nice to Be Prepared
Keys to success

• Managers and involvedstaff built the plan
• Same managers and staff practiced the plan
• Live and desktop drills under varying scenarios
  honed the mental and technical readiness
• We know our customers rely on our mission
  delivery in order to sustain theirs

22
Lessons Learned Governmentwide
Learned

• Consolidated information source needed to support
  decision-making
• Central clearinghouse for needs and resources
• Designation of essential services and associated
  rights is needed
• Handbook to guide Federal Agencies through
  extraordinary authorities
• Single coordination point for Federal efforts
  andissues

23
Lessons Learned - NFC
Learned

• Communications are always the challenge planned
  communications channels are a real aid
• Having a well-drilled plan allows for the basics
  to be accomplished with little intervention and
  allows management to focus on exceptions and
  surprises

24
Lessons Learned - NFC
Learned

• Subscription service not an ideal model for
  essential service provider
• Administrative support a key aspect of support
  for long-term deployed staff
• For long-term deployment, coordination of
  outreach efforts targeting employees becomes an
  essential task

25
Lessons Learned - NFC
Learned

• Some organizations benefit from close proximity
  with one another
• Must revalidate plan assumptions, plan contents,
  and business requirements annually
• You cannot assume infrastructure items will be
  there for you

26
Working to Mitigate Future Risks

• Data Center equipment will remain at subscription
  service location until new primary facility is
  selected
• New AWS alternatives being explored
• All business operations are being reassessed
  based on experiences
• Lessons learned are being incorporated into
  revised DR and COOP plans

27
People, Planning, Practice

• People who understand believe in DR and COOP
  make it happen
• Plans have to account for your threats, the
  business requirements for COOP, and the people
  side of things
• Practice is the best teacher. Practice on a
  regular schedule (top priority) and against
  changing scenarios. Capture the valuable lessons
  from each exercise.

28
Epilog

• Facing the most devastating natural disaster to
  impact the United States, the employees of NFC
  set aside their personal concerns and focused on
  delivering for their customers.
• Given such an event, the best came out of people,
  far and wide, and we are all better because of
  it.

29
One-Year Review of Key Indicators

• Housing rehabilitation and demolition are well
  underway while the housing market tightens
• Pace of demolitions has accelerated in the last
  six months
• Number of permits issued for rehab has nearly
  doubled
• Rent prices have increased by 39 and home sale
  prices have spiked in suburban parishes
• Public services and infrastructure remain thin
  and slow to rebound
• Approximately half of the bus and streetcar
  routes running with only 17 of buses in use
• Gas service reaching 41 electricity reaching
  60 of pre-Katrina customer base

30
One-Year Review of Key Indicators

• Labor force in the region is 30 smaller than one
  year agowith slow growth over the last six
  months
• Unemployment rate remains higher than pre-Katrina
• The New Orleans metro area lost 190,000 workers
• In the past six months, 3.4 more jobs but much
  of that may reflect the rise in new job seekers
• Unemployment rate is now 7.2, higher than last
  August
• Over 100 billion in federal aid dedicated to
  serving familiesand communities impacted by
  hurricanes Katrina, Rita,and Wilma
• Number of displaced and unemployed workers
  remains high
• Federal government approved approximately 109
  billion in federal aid to the Gulf Coast states
• Half has been dedicated to emergency and
  longer-term housing
• Estimated 278,000 workers are still displaced
• 23 remain unemployed

31
(No Transcript)
32
(No Transcript)
33
(No Transcript)
34
(No Transcript)
35
(No Transcript)
36
(No Transcript)
37
(No Transcript)
38
(No Transcript)
39
(No Transcript)
40
Questions???

Please ask us questions important to you.

• Lagniappe is a New Orleans term often used to
  indicate getting something extra in a
  transaction. Our lagniappe for you today
  includes additional pictures of the post-Katrina
  New Orleans area.

41
LagniappeCanal Street in CBD

42
LagniappeLower 9th Ward

43
LagniappeMold

44
LagniappeThe lake is on the other side of the
building
LagniappeThe lake is on the other side of the
building

45
LagniappeBoat came 3 miles inland
LagniappeBoat came 3 miles inland

46
LagniappeBefore
LagniappeBefore

47
LagniappeAfter
LagniappeAfter

48
LagniappeAnyone missing a hot tub?
LagniappeAnyone missing a hot tub?

49
LagniappeA quite country road
LagniappeA quiet country road

50
LagniappeMain bridge into Eastern New Orleans
LagniappeMain bridge into Eastern New Orleans

51
LagniappeA view from the elevated expressway
LagniappeA view from the elevated expressway

52
LagniappeBourbon Street Never so clean!
LagniappeBourbon Street Never so clean!
53
NSAA IT Conference

Thanks!