Orphaned Servers and Broken Processes

1
Orphaned Servers and Broken Processes
2007 Security Professionals Conference April 12,
2007
2
Agenda

• Introductions
• Background
• Do You Have Orphaned Servers?
• Vigilance and Awareness
• Lessons Applied _at_ CSUN
• Reflections _at_ CSUN
• Lessons Relearned _at_ CSUN
• Summary
• Questions / Discussion

3
Background

• Moran Technology Consulting (MTC) conducted an
  Incident Audit for a large research university
• Audit focused on two (2) systems that were
  thought to have been compromised
• A server housed in central IT data center
  containing alumni donor information, including gt
  100,000 social security numbers
• A server outside of central IT data center
  containing campus health center data, including gt
  50,000 patient records

4
Background

• Health Center Server
• Incident Detection
• Vigilant scanning of system following prior
  incident
• Systems Management
• No monitoring or support from central IT(in
  spite of HIPPA related risks)
• Comprises detected
• A Rootkit (Hacker Defender Trojan)
• W32/pate.b virus
• Server had participated in attack on another
  university server

• Alumni Donor Server
• Incident Detection
• External complaints of brute force attack
  emanating from server
• Systems Management
• System was decommissioned and powered-off and
  removed from the network
• System was removed and returned to the network
  several times (as an orphaned server)
• Following compromises detected
• A Rootkit
• 4 GB of unauthorized music files
• The Hideout virus

The Compromises occurred 4 months and possibly 12
months prior to detection, respectively.
5
Background

• Why did these compromises occur?
• Insufficient Operational practices and procedures
  within central IT
• Lack of formalized standards for secure server
  configuration
• No clear roles and lines of responsibility
  resulted in lack of accountability for systems
• Failure of change management procedures to ensure
  proper system tracking and decommissioning
• While 75 of campus servers reside outside of
  central IT data center little or no effort had
  gone into securely supporting or monitoring these
  systems
• A silo culture existed within central IT
• Staff reductions left central IT with too few
  resources
• Previous attempts to establish campus-wide
  polices failed due to lack of executive support

6
Do You Have Orphaned Servers?

• Do you know who owns and is managing EVERY
  server
• In the central data center?
• Across the campus?
• How much do you know about the servers on your
  campus?
• What types of data do they house or connect to?
• Who is accountable for these systems and data?
• Who owns and maintains these systems
• Do you have servers on your network that are
  believed to have been powered OFF but are really
  powered ON?
• What are your current decommissioning policies
  and procedures
• Have these policies and procedures been
  formalized?
• Do they apply and are they in practice
  campus-wide?

7
Vigilance and Awareness

• What formal IT Policies and procedures guide
  security management roles and responsibilities?
• Do the they cover EVERY server on the campus,
  including both Central Data Center servers AND
  department servers?
• Executive Ownership and Participation is REQUIRED
  for Security Policies to be effective

8
Lessons Applied _at_ CSUN

• Decided to use the incident at another
  institution to repose the salient questions and
  to review our security stance!
• Have we identified EVERY server in the central
  data center?
• Have we identified EVERY server exposed to the
  Internet?
• Do we have sufficient information to protect
  these assets?
• What additional policies, procedures, and
  processes are needed?
• Developed an approach to review our environment.
• Reviewed firewall rules for all openings
• Reviewed DNS for named servers
• Reviewed ARP tables for IP assignments
• Performed network scans to identified live
  servers
• Established primary and secondary system owners
  for each server
• Formulated a plan to address deficiencies found.

9
Reflections _at_ CSUN

• We thought we were in a very secure position
  but we were wrong!
• We had spent a lot of time protecting the systems
  and networks everything you would expect of a
  professionally run IT organization to do.
• We were very surprised by the number of orphans
  we found on our network.
• What introduced our orphaned servers?
• Well meaning staff (Ill get to it.)
• Shifts of responsibilities and new staff. (I
  thought they/he owned it)
• Overloaded staff and lack of resources. (refer
  back to 1)
• This is a Presidents worst nightmare!
• The overall cost of an incident includes the
  damage done to the reputation and credibility of
  the institution.
• The question is not whether but when
• Hackers want two things
• Personal information they can sell and
• Access to powerful computing resources

10
Lessons Relearned _at_ CSUN

• Always presume that you are less secure than you
  are!
• Use external and internal security events to
• Increase security awareness at both the staff and
  executive levels
• Perform ad-hoc reviews of your environment
• Establish periodic reviews of your environment,
  which includes
• Scans of the network (dont assume your change
  management process are fail-proof)
• Review the ownership of servers and identify the
  administrators
• Review the associated Firewall and DNS rules for
  all servers
• Structure your IT organization/environment with
• Clearly defined role and responsibilities for
  every unit and individual
• Predefined separation of duties, with effective
  working teams.
• Established strong change-management procedures
  that include removing of services from
  production. (The cleanup process is the killer.)
• The one throat to choke principal does not
  apply! Everyone must participate to ensure
  success.

11
Summary

• Security Incidents happen to everyone.
• Incidents that occur at other institutions are
  excellent opportunities to improve your own
  security awareness and practices
• Preparedness and Vigilance are key to minimizing
  cost
• Presume that you are still vulnerable (You are!)
• Perform on-going reviews of your environment
• Develop a Incident Response Plan
• Institute policies and procedures to identify,
  manage and properly decommission every server on
  campus
• Orphan servers provide a window for hackers to
  exploit your entire IT environment

12
Questions / Discussion