SECURITY PERS Best Practices and an Assessment

1
SECURITYPERS Best Practices and an Assessment
PRISM 2006

• Tom Roark, Technical Services Manager
• Mississippi Public Employees Retirement System

2
The Session Game Plan.

• The Big Question about Security
• An Opportunity (Problem)
• An Undesirable Surprise
• Change/Upgrade Strategy
• Security Assessment
• Best Practices and Lessons Learned

3
Security - The Big Question

• Is there a way to totally secure your computing
  devices and data sources?

4
Security The Answer

• The only way to completely secure any computer
  device or data source is to disconnect it from
  the network and place it in a locked vault where
  no one has the key. In this case, the data would
  be completely secure but totally inaccessible.

5
Free Stuff

• A wise person learns from his/her own mistakes
  and experiences.
• An even wiser person learns from others mistakes
  and experiences.

6
An Opportunity (Problem)

• We were receiving 800 diskettes a month to
  collect Wage and Contribution Data
• Along with the diskettes we were also receiving a
  signed paper Form 8 as an official submission
  document
• Both the diskette and paper Form 8 were submitted
  to PERS via regular mail
• It was the year 2002 and PERS was still using a
  DOS based application. (The application was
  developed in Foxpro 2.0 (Foxpro 2.0/1990))

7
Our Assignment

• Upgrade the application from a DOS based platform
  to a Windows based platform
• Move from diskette submission via regular mail to
  secure electronic submission via the internet
• Eliminate the paper Form 8 required
• Distribute the new application via the internet

8
The Solution

• Upgraded to Visual Foxpro 5.0
• Decided to use FTP to transfer data files
• Since FTP was not a secure protocol, we
  researched and discovered a FREE Secure FTP
  client and server software solution
• Researched and integrated an FTP Activex control
  into our Visual Foxpro application. This provided
  for a seamless submission to PERS via FTP
• Eliminated the paper Form 8 by including a
  special data line in the transferred file
• Provided a download link on our web site to
  distribute the new software (unpublished, login
  required)

9
Distribution Configuration
10
Submission Configuration (Secure FTP)
11
Bomb (New State Security Policy)
NO MORE FTP ALLOWED
12
Go to Plan B (we had no plan B)

• Researched using an allowed protocol (HTTP) to
  transfer the data files and at the same time keep
  as much of our previously completed work as
  possible. We found an HTTP activex control for
  Foxpro.
• Since HTTP is not secure without SSL and PERS was
  not yet SSL enabled and savvy, we researched and
  found a file encryption activex control that
  could be used to secure the data files prior to
  transfer via HTTP (DES, 3DES, AES encryption)
• We also decided to transfer the data files to
  PERS web server via an unpublished URL which
  required a login for added security
• We researched and wrote PERL scripts to transfer
  the files from our web server to our inside FTP
  server and then immediately delete the files from
  the web server for another layer of security
• We also found a real plus to plan B in that it
  removed the secure FTP client and server software
  requirement

13
Submission Configuration (Plan B HTTP/FTP)
14
PERS Best Practices

• Always communicate with your ISP on projects that
  could be impacted by your dependence on them.
• Make sure any future plans you have will not be
  hindered by any future plans they have.

15
An Undesirable SurpriseMonday Morning 700 am
4/1/2002

• This really happened
• Dont let it happen to you!!!!!!

16
(No Transcript)
17
www.pers.state.ms.us was hacked

• Gained Administrator access to server
• Replaced the Root structure of PERS web site with
  hacked pages
• Changed Administrator passwords and locked us out
  of our own servers
• It was NOT an April Fools Joke!!!!

18
What did we do??????

• We panicked like most IT staff where
• The web server was deployed by a third party
• Web site was developed by another third party
• MIS staff had minimal web server experience
• Backups were done infrequently
• We started rebuilding our web server

19
Solving the Web server problem

• Rebuilt the server from the ground up
• Renamed administrator account
• Put Complex Password on administrator account
• Disabled the Guest account
• Used netscape best practices on web server
  configurations
• As an interim protective strategy we installed
  and configured a desktop firewall on the web
  server
• Implemented a new backup strategy for our web
  server

20
The Interim Configuration
21
PERS Best Practices

• Dont EVER leave an administrator password
  blank!!!! Due Diligence
• Take ownership in products delivered by 3rd party
  companies
• Play a vital role in all installations/deployments
  . Make sure your understand everything
• Have computing standards in place and make sure
  any installations/deployments done by 3rd parties
  meet your standards
• Make sure every computer asset you have has added
  security beyond that of an OS (Firewall, AV,
  etc)
• Make sure you have an adequate backup strategy
• Have documented server build procedures (Disaster
  Recovery for WHEN it happens)

22
The straw that broke the camels back
23
Time to Make Some Changes

• We did some serious evaluations and took an
  honest inventory at where we were
• 3 years ago an honest inventory indicated that
  PERS was
• Using a basically unprotected web site
• Old Version of Netscape Suitespot Web Server
• Using an outdated backup technology
• Very Slow Network
• Outdated Servers and Desktops
• Using Windows NT 4.0 on desktops and servers
• Using Novell 5.1 (File and Print sharing)
• Using Office 97
• Exchange 5.0
• SQL Server 6.5
• Norton Anti-Virus 6 or 7
• No web security
• No spy-ware/mal-ware security
• No email security
• No Desktop/Server/Backoffice software maintenance
  agreements
• Uncontrolled user environment

24
Time to Make Some Changes

• Did some research about available products and
  possible upgrade paths
• Set some goals, objectives and priorities
• Decided on a change strategy
• Got to work and started making changes

25
Change Strategy Pyramid
26
Hardware - Network and Firewall UpgradeWhat it
was like

• Outdated Slow Network (10mb half duplex hubs and
  an unsupported core LANPLEX (SPOF))
• Multiple protocol network (TCP/IP, IPX/SPX, etc)
• No DMZ or Service Network for exposed servers
  (www.pers.state.ms.us)
• Software firewall on Windows NT with 2 interface
  cards, inside and outside

27
Hardware - Network and Firewall UpgradeWhat we
did

• Purchased 10/100/1000 MB Ethernet
  state-of-the-art full duplex switches
• Purchased 10/100/1000 MB Ethernet nics for
  servers
• Researched and Purchased a firewall appliance
  with multiple functionalities including Intrusion
  Detection/Prevention, Content Filtering, AV, VPN,
  FW
• Made sure our firewall appliance had multiple
  interface capabilities, minimum of three (inside,
  outside, service)

28
The Current Configuration
29
Service Network Configuration
30
Hardware - Network and Firewall UpgradePERS Best
Practices

• Firewalls
• Use Appliances for firewalls
• Identify SPOFs and implement a fault tolerant
  strategy in case of failure (Cluster, spare, next
  business day maintenance contract)
• Implement a service network or DMZ zone for
  exposed servers
• Use a different network segment on each firewall
  interface
• Eliminate path from exposed/outside servers to
  inside servers/network. (i.e. Implement pulls
  from inside instead of pushes from
  outside/service network.)
• Restrict Administration access to firewall to
  internal specified machines and users, NO
  external configuration allowed
• Lock down firewall rules to interfaces and
  entities, make them as tight as you can get them
• Disable all services not being used on firewall
• Configure any alerts to go to a firewall
  administrators email group
• Lock down VPNs to pass traffic to proxies so
  that rules must be created to allow exact data,
  not just anything
• Lock down smtp to ISP relays only or equivalent,
  dont allow smtp from universe
• Network
• Only patch network drops actually being used to a
  switch port
• Require user/password security to administrater
  your switches
• Disable all services not being used on your
  switches
• Keep your network switches and firewall up to
  date with patches

31
Hardware Desktops/Servers/Backups/OtherWhat it
was like.

• Old outdated equipment
• Servers and desktops didnt even meet minimum
  requirements for new OSs
• Several different models adapters and monitor
  types
• 10 MB desktop nics/100 MB server nics
• Old Slow Scanners
• Very old backup solution

32
Hardware Desktops/Servers/Backups/OtherWhat we
did

• We upgraded our desktops, servers, backups, etc
• Standardized on equipment manufacture
• Standardized on server type across the board
• Standardized on desktop type across the board
• Standardized on monitor types
• Standardized on network access with 10/100/1000
  MB nics
• Standardized on scanner types
• Implemented a D2D2T high speed backup solution
  (Really Good Decision)

33
D2D2T Backup Solution

• Before D2D2T
• 4 Backup Servers
• 8 Tape drives with 5 tape auto-changers each
  (DLT)
• 4 Racks of space
• Slow 100 MB Network
• 12-16 Hour Backup window (incremental on some)
• Very Problematic
• After D2D2T
• 2 Backup Servers
• 2 High Speed Tape Cache Systems (1.5 TB each)
• 2 Tape drives and no auto-changers necessary
  (LTO-2)
• 1 Rack of space
• Fast Gigabit Network
• 5 Hour Backup window
• 1-2 Hour Offline tape copy

34
D2D2T Backup Solution
35
Hardware Desktops/Servers/Backups/OtherPERS
Best Practices

• Purchase similar equipment where possible in
  order to have standardization and swappable parts
  in case of emergencies
• Use hardware RAID disk configurations on all
  servers
• Place all servers on UPSs
• Implement UPS power failure graceful shutdowns
• Configure equipment with more resources than
  software minimum requirements
• Disable unnecessary devices on desktops (USB,
  diskette, etc)
• Implement a D2D2T backup strategy
• Standardize, Standardize, Standardize !!!!!!!!

36
Picture of server room
37
Desktop and Server OSWhat it was like

• Windows NT4 sp6
• Users could do almost anything, uncontrolled
  environment
• Users could run any .exe
• Users could make all types of desktop preference
  changes
• Had Control Panel access
• Minimal group policy usage
• Many different configurations
• Take 1 days to completely rebuild a pc from the
  ground up
• The bottom line is we had inconsistent, unstable,
  problematic and unsecured configurations

38
Desktop and Server OSWhat we did

• Implemented an enterprise agreement for desktop
• Implemented software assurance for servers
• Windows XP sp2
• Windows Server 2003 sp1
• Made good use of Active directory GPO.
• Minimize the number of different configurations
  as much as possible
• Implemented the use of Ghost images for builds.
  We can have a pc completely rebuilt in less than
  an hour

39
Desktop and Server OSPERS Best Practices

• Standardize configurations
• Minimize the number of different configurations
  as much as possible
• Implemented Disk quotas per user
• Segregated MIS and Business user home directories
• Rename administrator accounts and deleted
  descriptions
• Disable guest accounts
• Delete other windows help accounts
• Disable all unnecessary services
• Control drive mappings with group policy and/or
  login scripts
• Implement user time restrictions
• Documented desktop and server builds
• Used Microsoft Security Baseline Analyzers (MBSA)
  and followed recommended best practices
• Made copies of all software and build procedures
  and placed offsite for disaster recovery
• Control the desktop experience for backgrounds,
  screen savers, colors, etc
• Use AD GPO.

40
Desktop and Server OSPERS Best Practices AD GPO

• Configure Active Directory according to your
  organizations structure
• Implement global policies and then implement
  departmental policies
• No Control Panel Access
• Standard Background
• Standard screen savers with password protected
  Resume
• Standard Color Scheme
• Standard Start Menu View and settings
• Redirected Start Menus
• Redirected My Documents
• Run only allowed Windows Applications !!!!!!

41
Back Office ApplicationsWhat it was like

• No Web Security
• No Mail Security
• Netscape Suite Spot Server
• SQL Server 6.5
• Exchange Server 5.0
• Microsoft Office 97
• Adobe 4.0
• Flash (???)
• Java (1.???)
• Anti Virus 6.0 - 7.0
• Legato (4.???)
• No spyware or malware
• No Web IDE

42
Back Office ApplicationsWhat we did

• We upgraded to.
• Symantec Web Security
• Symantec Mail Security with Brightmail
• IIS 6.0
• SQL Server 2000/2005
• Exchange Server 2003
• Microsoft Office 2003
• Adobe (latest)
• Flash (latest)
• Java (latest)
• Antivirus 10.0 with spyware/malware
• Arcserve 11.5
• Dreamweaver and Coldfusion

43
Back Office ApplicationsPERS Best Practices

• Standardize on network file locations for all
  users for Word, Powerpoint, Excel, etc
• Standardize on other settings as much as possible
  (auto archive, empty deleted items, empty
  temporary internet files on exit, etc)
• Lets look at a couple of items in more detail
  (Web Security, Mail Security)

44
Back Office ApplicationsWeb Security - What it
was like...

• Had each user read and sign an internet security
  policy
• Lock in IP address with DHCP reservation
• Create a firewall user entity matching the DHCP
  IP address
• Add the firewall entity to the internet access
  group
• Users could browse anywhere in the world, NO
  restrictions
• Users could log into any computer on PERS network
• Basically our web security consisted of allowed
  or not allowed web access
• Whenever supervisors wanted an access listing for
  a particular user, I had to browse through TONS
  of firewall logs.
• Any sites that I wasnt sure of their content, I
  actually had to go to the site to determine its
  content
• I eventually wrote a program to parse the logs
  and give a report for a specific IP address

45
Web security Configuration
46
Guess what happened one weekend ???

• Someone was looking at things they were not
  supposed to be looking at ???
• From the Executive Directors assistants
  workstation

47
Back Office ApplicationsWeb Security What we
did/Best Practices

• Implemented a web security proxy server (Symantec
  Web Security) with automatic content filtering,
  reporting features and that was Active Directory
  aware
• Configure internet browsers connection settings
  via active directory group policy to use a proxy
  server
• Used AD GPO to prevent users from changing
  browser proxy settings
• Configure your proxy to require logins. This
  reminds the users they are being monitored
• Set up denied categories such as sex, games,
  gambling, etc
• Set up filtering on all allowed categories
• Set up some allowed sites lists for state
  government sites for non-internet users
• Implemented an autolock policy for repeated
  violations
• Do NOT allow temporary overrides for content
  filtering
• Make sure users can only login to the network on
  their workstation
• As a Result..
• We were able to create one firewall entity and
  rule for the web security proxy server only
• Eliminated individual firewall configuration by
  using a product that integrated with our windows
  active directory
• Eliminated searching through firewall logs for
  violations
• Eliminated DHCP IP address reservations
• Eliminated foxpro program to parse firewall logs
  for an individual sites visited because product
  had reporting features by user
• Stopped spyware that doesnt use IE proxy settings

48
Web security Configuration
49
Back Office ApplicationsEmail Security - What it
was like...

• File system AV on email server
• No spam detection, getting all types of garbage
• Exchange 5.0 which even allowed .exe file
  attachments
• Firewall allowed smtp traffic (email) from
  universe
• No access remotely via the web (secure or
  unsecured), directors and managers wanted it
• Basically, we had no email security

50
Email Security Configuration
51
Back Office ApplicationsEmail Security - What we
did...

• Upgraded to exchange 2003 (SP2), skipped 5.5 and
  2000
• Implemented Symantec Mail Security (SMS) for
  Exchange 2003 with spam and AV protection for the
  email database
• Started adding BAD words to SMS match lists
• Used real time black lists (RBL) of know spammers
• Created Blank subject/sender filters
• Reconfigured firewall to allow email from ISP
  relays only, not the universe (cut down on
  internal state attacks)
• After a year or so of fighting manual match list
  maintenance, we upgraded to SMS with Brightmail
  technology subscription (WOW!!!!)
• Configured suspect spam threshold and began
  routing suspect spam email to a spam catcher
  email account that we are monitoring
• Configured whitelist for bank clients, etc
• Used exchange baseline analyzers and followed
  best practices recommendations
• Implemented SSL webmail and only allow the
  firewall to connect

52
Email Security Configuration
53
Back Office ApplicationsEmail Security PERS
Best Practices...

• Implement a computer usage policy that has an
  email usage section.
• Require users to sign the computer usage policy
  agreement page and keep in personnel files
• Use an anti-spam, anti-virus, spyware/malware
  aware product on your email server
• Use Blank subject/sender filtering rules as well
  as any others deemed necessary
• Restrict attachment types
• Use whitelist for important customers that should
  not go through the email filtering process
• When implementing a webmail solution over the
  internet, use SSL (https)
• Only give secure webmail privileges to users that
  need it
• Implemented Mailbox quotas per user

54
Business ApplicationsWhat it was/is like

• Using Foxpro 2.0 to develop small miscellaneous
  applications
• Small applications were everywhere in our
  directory structure, no organization
• Using no longer supported forteg3 OOP language
  for LOB application
• No adhoc reporting for users
• Had 2 environments, Test and Production

55
Business ApplicationsWhat we did and Plan to
do

• Upgraded to Visual Foxpro for small apps and
  adhoc reporting
• Upgraded to Visual Studio for API small apps
• Plan to upgrade LOB application from forteg3 to a
  new development platform (java, .net, ???)
• Plan to implement some type of adhoc reporting
  for users (Data warehouse, BI)

56
Business Applications PERS Best Practices

• Have multiple environments, three if possible
  (Test, User Acceptance Test and Production)
• Have a designated directory structure for
  applications
• Implement good organization with a one to one
  correlation between the source code and user
  accessible application
• Use as few development platforms as possible
• Standardize and stick with the standard
• Have procedures and follow them as much as
  possible

57
UsersWhat it was like

• Users were changing colors and you could not see
  certain things
• Users would change display resolutions and font
  sizes
• All type of cursors, backgrounds, screen savers,
  gremlins, etc
• Browsers were being taken over by spyware
• Systems would respond slow and/or erratically
• Users were constantly complaining of computer
  problems (email garbage, etc..)

58
UsersWhat we did

• Implemented a new business policy and decided to
  upgrade the users every 3-5 years (Just Kidding)
• Implemented standards via AD GPO
• Took away the ability to do anything they wanted
• Provided training and education

59
UsersPERS Best Practices

• Involve the users in the upgrades
• Have them test and signoff that everything is
  working properly before the upgrades are
  implemented in production
• Educate your users on the new changes in security
• Train your users on the new aspects of the
  upgrades (OS, Office, etc..)

60
Quick Summary
61
Security Assessment

• We decided to wait until most of our upgrades
  were in place before we had our assessment
  performed
• We inquired about Homeland security money and
  were able to have the assessment done at no
  expense to PERS
• We got bids from three different companies and of
  course selected one
• We met with the selected vendor and agreed on an
  assessment strategy
• Our strategy was to perform an assessment on a
  subset of our network instead of the entire
  network. However, we made sure our subset had one
  of each type of machine/device configuration in
  the assessment (multiple environments scenario)
• We also decided to keep the assessment very quit
  to all personnel in order to try and get an
  accurate picture of where we really were. Very
  few people new the assessment was being
  performed.
• The assessment took about 3 months total

62
Security Assessment

• The vendor assessed 10 different categories and
  identified three levels of risk factor within
  each category
• High Risk, a severe security problem that could
  cause loss of service or immediate access to
  critical severs and file systems
• Medium Risk, less severe problem and by itself
  would not be an issue, but remediation would
  provide incremental improvements in security
• Low Risk, a vulnerability that is either very
  rare or would require significant skill to
  exploit or the potential exposure would be minimum

63
Security Assessment

• At first, several of the technical services staff
  posed questions after seeing unusual activity in
  logs and on administrator email notifications
• After a couple of days, the vendor had to ask for
  an administrator account to get access
• They had to ask how to get to our web server in
  the service network
• Initially, they said everything looks real
  good!!!!
• At the end of the assessment, the vendor said we
  have performed assessments on 10-15 state
  agencies and approximately 50 other entities in
  Mississippi and that PERS was one of the best
  they had seen.
• They also mentioned that the assessment will
  sometimes not be a fair indicator of an
  agencies overall security status because just a
  few missing patches will warrant a low score in a
  category and drive the overall rating down

64
Security Assessment

• PERS Overall score was a 2.5 out of a possible
  4.0
• However, PERS knew that because of legacy systems
  that not yet been upgraded, some of our internal
  hosts and database assessments would receive
  lower scores.
• The vendor asked PERS if we were sure we wanted
  older servers/OSs/etc.. scanned since we were
  planning to upgrade them
• What every security manager wants to hear
• Pileum was unable to compromise the PERS network
  through the available open services

65
Security Assessment Graph
66
Security Assessment Recommendations

• The only way to completely secure any computer
  device or data source is to disconnect it from
  the network and place it in a vault where no one
  has the key. In this case, the data would be
  completely secure but totally inaccessible.
  Therefore, there is a risk that must be assumed
  with any computing device or data source that is
  made accessible via network connections. It
  should be a security managers goal to minimize
  and be aware of the security risks, but not to
  assume that they can or will ever be eliminated
  completely.

67
Security Assessment Recommendations

• PERS investigate all High risk or critical
  vulnerabilities and their remedies
• Review our security policies and procedures and
  how they are enforced
• Have ongoing assessments quarterly
• Implement an effective patch management solution
• Implement a Windows event log management system

68
Going Forward PERS plans too

• Maintain software maintenance agreements on all
  software
• Finish our remaining planned upgrades
• Upgrade to SQL Server 2005 across the board
• Upgrade to Windows Server 2003 on legacy Systems
• Implement IE and other GPO settings
• Upgrade LOB application to current technologies
• Implement Self Service via the web
• Implement Security Assessment recommendations
• Implement a patch management solution

69
Final Thoughts Remember !!!!!!!

• A wise person learns from his/her own mistakes
  and experiences.
• An even wiser person learns from others mistakes
  and experiences.
• Visit the PRISM website at www.prism-assoc.org
  and download this presentation if you find any of
  this information helpful
• Take one of my business cards and shoot me an
  email or give me a call if you would like to
  discuss something in more detail