What Do You Do When You Have Been Hacked

1
What Do You Do When You Have Been Hacked?

• By Paul Wouters ltpaul_at_xtdnet.nlgt
• Xtended Internet, The Netherlands

2
What is Xtended Internet?Small ISP for
businesses

• Hosting (Linux, Windows NT, Solaris etc.)
• (Virtual) web hosting, DNS, mail etc.
• Leased lines
• Interpay services (credit card transactions)
• Internet intranet implementations
• Remote system administration
• Custom (mostly Linux-based) solutions

3
Personal Interests

• Linux (esp Linux Router Project,
  http//www.linuxrouter.org/)
• Computer security (not as full-time job!).
• Privacy, anonymity, and freedom of speech issues.
• European and Dutch law regarding computers and
  Internet issues.

4
You Have Been Hacked!Doing it by the book

• Disconnect network or systems
• Notify parties involved
• System administrators / management
• Clients and/or users
• The administrators of attacker.com.
• Local authorities (police)
• Internet authorities (Cert, FIRST)
• Analyse system(s)

5
You Have Been Hacked!Doing it by the book

• Obtain fix, restore system from tape, install fix
• Change credentials, revoke old credentials (PGP
  keys, SSL Certificates, SSH keys)
• Reconnect system

6
You Have Been Hacked!What most people do

• Leave compromised system running.
• Do not notify external parties.
• Have no time to analyse systems.
• Not sure what to fix, or there might not be a fix
  at all.
• Undesirable to change all credentials.

7
Risks operating a compromised system

• System might contain several back-doors
• More hackers might use the same vulnerability,
  possibly with more fatal consequences.
• Further penetration of network possible from
  compromised host.
• More information theft possible.
• Hackers might panic destroy.

8
Case 1 Qpopper Symptoms Entire network down

• Internet connection down. Firewall so busy
  generating syslog messages that normal routing
  functionality ceased.
• Pinpointed cause to combined pop-web server.
• Unknown, masked, root-uid binaries found running
  on system.
• What do you do?

9
Case 1 QpopperCollected runtime information

• Compared output ps versus pstree.
• Used pstree p to find the PIDs and collect
  information from the /proc/ltpidgt directories.
  (/kern and /proc for Net/Open/Free BSD)
• Kept the output of netstat an.
• Killed 9 ltPIDgts
• At this point the network connection was restored.

10
Case 1 QpopperA quick sweep

• Checked Frequently Manipulated Files
• Saved discovered alien binaries, prevented them
  from being started.
• Replaced binaries with clean binaries from
  neighbour server.
• Rebooted, trojans re-appeared at boot.
• Used strace v f and strings to find more
  trojans.
• Trojans didnt reappear on next reboot.

11
Case 1 QpopperA thorough check

• Restored tape of compromised machine on backup
  server, exported compromised file systems
  read-only over NFS to backup server
• Ran recursive comparison of entire machine using
  diff N q r on backup server, which yielded
  all the changed files.
• Saved rogue files and restored original files.
• At this point the original (vulnerable) server
  was back.

12
Case 1 QpopperFinding the cullprit

• A running daemon?
• A daemon started from (x)inetd?
• A webserver script?
• Check logfiles
• Compare local logfiles to remote logfiles

13
Case 1 QpopperWhat we found A Buffer Overflow

• Feb 26 015203 darling.xtdnet.nl
  qpopper-2.4beta112387 truncated
  _at_n122a97.XXXXXX.net -ERR Unknown
  command?????????
• Either syslogd or qpopper was overrun.

14
Case 1 QpopperCounter measures

• Installed gnu-pop3d
• Thoroughly analysed hacker tools
• strings
• strace -v -f on scrap system
• network sniffer (tcpdump, iptraf)
• gdb (GNU debugger)
• Blocked all listen ports used by tools
• Checked www.rootshell.com for exploits

15
Case 1 QpopperWe contacted the intended victim.

• Targeted machine was probed two hours prior to
  our compromise
• Targeted machine was hit by various hosts from
  the net, getting a sustained 150kbit/sec (20
  bandwidth) hit.
• The pseudo-random source IP and random ports were
  impossible to filter out.
• The customer had been threatened.
• We were the only ones that contacted the intended
  target.

16
Case 1 QpopperLessons learned

• Not all vulnerabilities make it to Bugtraq before
  they are used against you.
• Diversity of software is good.
• Blocking source IP spoofing was a good thing.
• Backups are Good.
• MD5 checksums can save a lot of time.

17
Case 2 Small Dutch ISPSymptoms nobody can log
in

• Login ceased to work, the system can only be
  entered in single user mode.
• strace -v -f /bin/login, then tried passwd
  which worked with /etc/shadow, then denied
  access.
• Tried pppd and popd, both denied access after
  using /etc/passwd.
• HUH? /etc/passwd or /etc/shadow ?

18
Case 2 Small Dutch ISPWhat we found

• Weird binaries running (.//eggbot)
• locate found /sbin/, which appeared to be a
  built directory of shadowsuite and linux-util.
• The only backup tape available (over six months
  old) was corrupted.
• Conclusion Hackers failed to install a rootkit
  and locked themselves out after rendering system
  unusable.

19
Case 2 Small Dutch ISPWhat we did

• Reconfigured, recompiled and installed rootkit.
• Wrote small perl script to convert back to
  regular system (shadow2passwd.pl)
• Removed all additional mess (mostly irc related)
• Upgrading system failed, it was too old to
  revive.
• Conclusion Dont run three year old systems,
  especially not without a backup.

20
Case 3 Big Dutch Cable ISPSymptoms We received
a lot of probes from a name server of large Dutch
cable ISP.

• We mailed the network administrators with a
  warning and received a reply which said
• We are not cracking, we are gathering
  information
• We also have an account at provider X, which
  might show up in your logfiles

21
Case 3 Big Dutch Cable ISPWe threatened to cut
them off. It then turned out that

• Their webserver had been hacked.
• When monitoring the hacker, the webserver (with
  all logfiles) was destroyed.
• The company was in the process of being taken
  over and had sensitive material on that server.
• They already knew Paul Wouters was the hacker
  and demanded a confirmation.
• They threatened (or used bad social engineering
  skills)
• I decided to stop communicating.

22
Case 3 Big Dutch Cable ISPThe evidence

• Two weeks later, a conversation followed. I chose
  to defuse the situation. Their evidence
• I was frequently logged in on the host used to
  attack their system
• I had an account at XS4ALL, another system used
  in probing their network
• A search on AltaVista confirmed that I fitted
  the profile of a hacker
• Based on this, they were trying to hack my
  systems

23
Case 3 Big Dutch Cable ISPTheir mistakes

• They panicked and became overly paranoid.
• Bad security -)
• Kept running a known compromised host without
  enabling remote syslog.
• Turned into hackers themselves.
• Put confidential materials on a public server.
• Threatened instead of communicating.
• Assumed that names are unique (I know of at least
  four Paul Wouters)

24
Contacting intrudersA good contact with
intruders can give valuable information

• Determine the type of intruder Cracker, Hacker
  or Script Kid.
• Cracker A professional thief, go to law
  enforcement authorities for help.
• Hacker Likely a reasonable person who made some
  mistakes. Engage in equal conversation.
• Script Kid Most likely an unreasonable person.
  No idea of Outside World. If you decide to talk,
  be careful. They might act irresponsibly.

25
Contacting intrudersHumor and the ScriptKid

• Scriptkid defaced customer website and left
  various hints as to his elite group.
• Sent false Moral Dilemma posting to a mailing
  list we were all subscribed to.
• Tremendous (serious) responses from people at the
  EFF, legal departments and press officers of
  large ISPs in the NL.
• Managed to educate the script kid somewhat (I
  hope)

26
When talking to sysadmins

• Dont be hostile, most likely they are victims
  just like you.
• Dont demand actions suggest reasonable actions.
• Dont demand (nor readily give out) user
  identities. Privacy issues are at stake here.

27
Information

• Bugtraq, Ntbugtraq, and Incidents mailing lists
  http//www.securityfocus.com/
• linux-security, linux-kernel, linux-net mailing
  lists on vger.rutgers.edu
• Cert http//www.cert.org/
• FIRST http//www.first.org/
• L0pht http//www.l0pht.org/
• Phrack http//www.phrack.com/
• HNN http//www.hackernews.com/
• RootShell http//www.rootshell.com/

28
Tools

• BigBrother http//MacLawran.ca/bb-dnld/
• MRTG http//ee-staff.ethz.ch/oetiker/webtools/mrt
  g/
• Mon http//www.kernel.org/software/mon/
• Network analysers (tcpdump, iptraf, ipgrab, ntop,
  etc), search for sniffer on http//freshmeat.net
  /
• Logcheck http//www.psionic.com/
• Strong Crypto, see ftp//ftp.replay.com/pub/crypt
  o
• SecureShell http//www.ssh.fi/
• International Linux kernel patches (IPsec, VPN
  software) ftp//ftp.kerneli.org/