Autonomix: Autonomic Defenses for Vulnerable Software

1
AutonomixAutonomic Defenses for Vulnerable
Software

• Crispin Cowan, Ph.D
• WireX Communications, Inc
• wirex.com

2
Talk Outline

• Progress LSM
• Experimentation Defcon
• Technology Transition
• Conclusions

3
LSM Linux Security Modules

• Linuxs open source broad popularity make it a
  great target for security research
• SubDomain, DTE, SELinux, etc.
• But one has to have a custom kernel to use these
  packages

• Solution security modules for Linux
• Standardized interface in the Linux kernel for
  security modules
• Get Linus et al to adopt LSM
• Expected result
• Can load advanced security into standard Linux
  kernels

4
LSM Linux Security Module

• Unfortunately, none are standard to Linux
• Maintained as kernel patches
• To deploy them, must acquire a custom kernel
• Linus would like to support advanced security
  policy, but not willing to endorse one project.
• Too political My security policy is better than
  yours.
• Linus is not a security expert, and doesnt want
  to be
• Linux is about choice anyway
• Solution enrich Linuxs module interface to
  support security policy modules

5
LSM Design

• syscall interposition, i.e. wrappers at the
  syscall interface
• not appropriate leads to module bloat
• already available by re-writing Linux syscall
  table
• Instead, we mediate access to internal kernel
  objects
• May subject X access object Y for operation Z?

6
LSM - Architecture
User-level process
Kernel
LSM Module

• Open syscall
• Std. error checks
• Std. Security checks
• LSM hook
• Complete request

• Policy engine
• examine context
• does request pass policy?
• grant or deny

7
LSM - Architecture
User-level process
Kernel
LSM Module

• Open syscall
• Std. error checks
• Std. Security checks
• LSM hook
• Complete request

• Policy engine
• examine context
• does request pass policy?
• grant or deny

8
LSM - Architecture
User-level process
Kernel
LSM Module

• Open syscall
• Std. error checks
• Std. Security checks
• LSM hook
• Complete request

ok with you?

• Policy engine
• examine context
• does request pass policy?
• grant or deny

9
LSM - Architecture
User-level process
Kernel
LSM Module

• Open syscall
• Std. error checks
• Std. Security checks
• LSM hook
• Complete request

ok with you?

• Policy engine
• examine context
• does request pass policy?
• grant or deny

Yes or no
10
Hook Style

• Restrictive module may only reject a request
  about to be granted
• Permissive module may only permit a request
  about to be rejected
• Authoritative module may totally over-rule
  standard kernel logic
• We chose restrictive hooks only, except for
  capabilities
• Simplifies LSM patch for maximum acceptability to
  Linux community

11
Module Stacking

• Strong desire to compose modules
• However, composition in general is intractable
• Solution stacking left to modules that want to
  stack
• Stackable module must export an LSM-like
  interface out the back
• Stackable module responsible for composing policy
  by taking down-chain modules results under
  advisement
• Module-stacking module (MUX) in development

12
Hook Location
13
LSM Linux Security Module Progress Since February

• Implemented working
• Running various WireX servers other LSM
  activists
• SELinux shipping exclusively LSM packages
• Heard all that last PI meeting ...

14
LSM Progress

• Paper presented at USENIX Security
• Plus IBM has a paper on LSM hook placement
  correctness
• Paper presented at Ottawa Linux Symposium
• Plus three other papers with LSM content
• We were invited to the Linux Kernel Summit
• Linus has accepted LSM
• Linux 2.5.27 has LSM in it

15
LSM ToDos

• Cut the LSM patch into bite-sized pieces
• Easier for the Linux maintainers to digest
• Work with Al Viro to get the VFS patch we need
• Hes working on it, but not quickly
• Address the network performance problem
• Leverage network hooks out of LSM
• Its Netfilters fault

16
LSM LessonHow to Get a Feature Into Linux

• Linux allows you to do it your way, but to be in
  Linus kernel, you have to do it in a way
  acceptable to Linus
• Do something that makes him happy
• Linus trusts his major subsystem maintainers, so
  work with them
• Keep someone interacting constructively with the
  LKML (Linux Kernel Mailing List), especially core
  developers
• Have to actually solve a problem
• LSM is effectively a direct response to Linus
  wishing out loud

17
LSM Lesson Hard Choices

• To keep Linus happy, we had to make some tough
  choices
• Security people largely would prefer
  authoritative hooks, and many more of them
• That would enable full POSIX audit logs
• But that also would have killed LSMs chances of
  getting into Linux
• Linus (and many others) do not like BSM
• They would have regarded LSM as unnecessary bloat

18
LSM Lesson Collaboration

• Before LSM
• Competing security projects all working
  independently
• With LSM
• Lots of collaboration among many projects
• Working to provide a common infrastructure
• Creating a common market for composable and
  competing security features
• The trick
• Finding the right layer to abstract
• Political engineering make sure there is
  something in it for everyone

19
LSM Collaboration Community

• 500 people on the mailing list
• Major contributions coming from
• 3 IBM sites (2 LTC sites T.J. Watson Lab)
• SELinux (NAI/NSA)
• SGI
• Assorted open source people

• Commercial LSM modules in the offing
• WireX
• SubDomain
• RaceGuard
• CryptoMark
• HP Secure Linux
• Ericsson Research

20
Experimentation ...

• Some real-world red teaming
• Play an Immunix server in the Defcon Capture the
  Flag (CtF) games
• Almost no holds barred
• No flooding
• No physical attacks
• New gaming rig designed by the Ghettohackers

21
Basic Defcon CtF Rules
Player Nodes
22
Basic Defcon CtF Rules
If all services found ...
Scorebot Polls player nodes, Looking for req.
services
Player Nodes
23
Basic Defcon CtF Rules
If all services found, Score one point for
the Flag currently on that node
Scorebot Polls player nodes, Looking for req.
services
Player Nodes
24
Basic Defcon CtF Rules
If all services found, Score one point for
the Flag currently on that node
Scorebot Polls player nodes, Looking for req.
services
while each team tries to replace others flags
Player Nodes
25
No Flooding

• DoS attacks are not interesting
• Explicit rule against flooding attacks
• Game masters will make you stop if you are caught
  at it
• Goal ensure that all teams are actually able to
  play
• Penalties
• Kicked out for overt DoS attacks
• Pay for bandwidth with a point penalty

26
Area View
27
Sporting Event
Immunix was white, hence Weiss Labs

• There was an official bookie -)
• Score broadcast on hotel cable

Score obfuscated
Teams named funky colors
28
The Catch

• The required services are secret
• Only a few clues
• They supply us with a VMWare/Linux image
  reference distribution that provides all required
  services
• It is also riddled with vulnerabilities
• The scorebot polls for the required services
• But the scorebot stops its poll if it finds
  something it doesnt like

29
The Reference Distribution

• Red Hat 6.2, unpatched
• nmap shows nearly everything open
• finger, POP, IMAP, SMTP, SNMP, Webmin ...
• Apache running as root
• CGIs for adduser and deleteuser
• Anonymous can create a user login on your node
• As any user number, including zero

30
Example Services the Scorebot Wanted

• Create a user
• Send that user mail
• Finger the user
• POP in to fetch the mail
• Delete the user
• Note no crypto protocols
• No proper authentication of the scorebot
• Must heuristically distinguish scorebot from
  attacks using behavior signatures

31
Interesting Challenge

• Not just survive severe attack, but also
• Protect bad code
• A lot of it
• Vague functional specification
• Rapid deployment
• Great new game infrastructure from Ghettohackers
• Interesting challenge
• Engaging scoreboard

32
Captains Meeting

• Explain the rules in detail
• Hand us the reference distribution

33
Setting Up
34
The Popular Strategy Human Intrusion Detection

• Launch the reference Linux distribution
• Ad hoc patch as stuff happens
• Defend
• look for logins, I.e. non-scorebot behavior
• kill them off ASAP
• very labor-intensive

35
The Immunix Strategy Protect Bad Code with
Immunix Tools

• Port all plausible services to Immunix 7
  distribution
• Use our own fingerd, httpd, etc., up-to-date and
  compiled with StackGuard and FormatGuard
• Run on an Immunix kernel with SubDomain and
  RaceGuard
• Wrap vulnerable services CGIs with SubDomain
  profiles to limit access to least privilege
  necessary
• Launch only when we were reasonably confident
  that the Immunix machine was configured securely

36
Dealing with Logins the SubDomain Shim

• Change adduser CGI to use a special default
  shell /bin/fubush
• /bin/fubush is just a hard link to /bin/bash
• Restrict /bin/fubush to only the operations
  needed by the scorebot
• Attackers can go ahead and create a login with
  uid 0 and it still wont do them any good
• They get a root shell, stuck in a tiny sandbox

37
Immunix Team
38
Immunix Team
Me
Seth Arnold
Steve Beattie
Chris Wright

• Plus 15 volunteers

39
From Our Corner
40
From Our Corner
John Viega
Me
Steve Beattie
Seth Arnold
Chris Wright
41
Mental Stress

• This is a tough game to play
• Head-to-head competition with a lot of very smart
  people
• Real-time
• Continuous
• The intensity of qualifying exams
• That go on for 22 hours in a 48 hour period
• set in the middle of a rave
• Hydrate or die -)

42
Rave

• Loud music
• Smoking
• Gawkers
• Social engineering
• Periodic news breaks

43
Our Strategic Error

• What We Did
• For first 4 hours
• No server at all
• Porting services to Immunix ASAP, based largely
  on nmap and source inspection
• Next 4 hours
• Launch Immunix server
• Its secure, but is not making the scorebot
  happy
• Cost us massive points
• Too focused on the science of can we defend
  Immunix? and not enough on the game rules

• What We Should Have Done
• Launch reference system immediately
• Defend ad hoc like everyone else
• Run network sniffer to determine what the
  scorebot wants
• Would have
• Put us over the top on points
• Learned what scorebot wants much faster
• We eventually did this

44
Immunix Server Not Up Yet
6th place
45
Once Immunix Server Up in the Scorebots
Opinion )

• Our score quickly rose

2nd place
46
Once Immunix Server Up in the Scorebots
Opinion )
Close 2nd place
47
Once Immunix Server Up in the Scorebots
Opinion )
1st place

• Stayed there most of Saturday

48
Late SaturdayNew Service Requirement

• With 4 hours of play to go, the scorebot
  changed now it wanted Webmin
• Open source web-GUI for Linux administration
• Competitor to WireXs commercial server appliance
  software
• Rather famously vulnerable )
• Took us 2 hours Sunday morning to make the
  scorebot happy again
• Lost our lead

49
Some of Our Creative Attacks

• Lock Out the Owner
• Once we root the machine, install a back door
• Also replace roots login shell with /sbin/halt
• Owner cant log in to their own machine
• But we can

• Spambot
• Add user to their server
• User sends spam mail to all the other teams
• Costs them penalty points
• Penalties are per connection
• Spambot sends 1-byte e-mails

50
Final Score 2nd Place
51
Lesson Symmetric Red Teaming Solves Rules Issues

• Everyone is both an attacker and defender
• Bad everyone needs to learn how to attack
• Good
• Everyone should learn how attacks are done -)
• Rule fussing about how hard or easy it is for the
  attacker apply to all parties -gt less fussing
• Ghettohackers have designed a great game
• Looking for technology transfer to Government

52
Lesson Mandatory Access Control is Not Enough

• telnetd was a required service
• WireX never bothered to patch a vulnerability in
  telnetd for Immunix
• Only idiots run telnetd -)
• Someone hacked our telnetd
• Didnt get out of the SubDomain sandbox
• Did make our telnetd stop working
• Cost us a point that round
• General case MAC protects your system, but not
  your individual services

53
Lesson Resource Management is a Security
Attribute

• SubDomain confined attacker logins to only run
  prescribed code
• Including PERL
• Attacker launched a PERL fork bomb
• Consumed all of real and virtual memory
• While our machine is thrashing, the scorebot
  passes us by
• Costs us a point that round

54
Lesson Redundancy Helps When You Are Vulnerable

• Penetration attacks take a long time to recover
• Must clean up state, find fix vulnerability
• DoS attacks take a long time to recover
• If machine crashes, must fsck file system can
  take 10 minutes
• Hot spare can be on-line in seconds
• Heterogeneous hot spare keeps attacker from
  immediately deploying the same attack

55
Lesson Redundancy is Resource-Constrained

• Must have humans on watch to clean up the
  compromised machine
• The hot spare will not protect you for long
• Presumption that hot spare prevents attacker from
  attacking again assumes resource limit at the
  attackers end
• If attacker has lots of exploits/resources, they
  will hack your heterogeneous server just as
  quickly

56
Lesson Immunix was Impenetrable, but not
Incorruptible

• No one ever flagged the Immunix server
• Others did plant enemy flags on our reference
  server (as expected)
• But they did hit the Immunix server hard enough
  to compromise availability
• Take out one required service, and the scorebot
  doesnt award a point
• We missed first place by less than 4 points out
  of approx. 55

57
Technology Transfer

• LSM Linux Security Module
• Linus said yes
• Immunix licensed to Compaq/HP
• Build a product family of security appliances
• Firewall, AV Mail server, NIDS, etc.
• Compaq hardware
• Immunix OS, user interface
• 3rd party security applications
• Press release this week

58
Summary

• LSM
• Technical work stable
• Political breakthrough
• Technology transfer heavy lifting coming next
  feed actual patches to Linus et al
• Other technologies working on them ...
• Experimentation Defcon fun )
• Technology Transfer going well
• LSM in Linux
• Immunix licensed to HP

59
Web Resources

• LSM http//lsm.immunix.org
• Active mailing lists
• Sign up if interested
• If you are hacking security into the Linux
  kernel, consider making it an LSM module
• Defcon
• Defcon convention http//www.defcon.org
• CtF game http//www.ghettohackers.net/ctf/