Operating System Security II

1
Operating System Security II

• Andy Wang
• COP 5611
• Advanced Operating Systems

2
Outline

• Single system security
• Memory, files, processes, devices
• Dealing with intruders
• Malicious programs
• Distributed system security
• Using encryption
• Secure distributed applications

3
Single System Security

• Only worrying about the security of a single
  machine (possibly a multiprocessor)
• One operating system is in control
• Threats comes from multiple users
• Or from external access

4
Protecting Memory

• Virtual memory offers strong protection tools
• Model prevents naming another users memory
• What about shared memory?
• Use access control mechanisms
• Backed up by hardware protection on pages

5
Protecting Files

• Unlike memory, files are in a shared namespace
• Requires more use of access controls
• Typically, access checked on open
• System assumes users has right to continue using
  open file

6
File Access Control in UNIX

• Every file has an owning user and group
• Access permissions settable for read, write, and
  execute
• For owning user, owning group, everyone else
• Processes belong to one user
• And possibly multiple groups
• Files opened for particular kinds of access

7
Protecting Processes

• Most of a processs state not addressable
  externally
• But IPC channels allow information to flow
• So security must be applied at IPC points

8
Protecting IPC

• Typically, IPC requires cooperation from both
  ends
• So a major question is authentication
• Does this channel connect where I think it does?
• OS guarantees identity, ownership of other process

9
Limiting IPC Access

• Each party to IPC has control over what is done
  on his side
• Some IPC mechanisms allow differing modes of
  access for different users
• So access control required for such cases

10
Protecting Devices

• Generally treated similarly to files
• But special care is necessary
• In some cases, a mistake allows an intruder
  unlimited access
• E.g., if you let him write any block on a disk
  drive

11
Controlling IPC Access in Windows NT

• General model related to file access control
• Processes try to access objects
• Objects include IPC entities
• On first access, request desired access rights
• Set of granted access rights returned
• System checks granted access rights on each
  attempted access

12
Covert Channel

• Two packets in quick succession ? 1
• Else 0
• CPU usage, memory allocation, HD access, white
  spaces

13
Other Covert channels

• Steganography
• Hiding secret message in graphics, movies, or
  sound
• Subliminal channels
• Names with different initials
• Different number of blank spaces at end of lines

14
Beware of Back Doors

• Many systems provide low-level ways to access
  various resources
• /dev/kmem
• raw devices
• pipes stored in the file system
• The lock on the back door must be as strong as
  the lock on the front door

15
Intruders

• Modern systems usually allow remote access
• From terminals
• From modems
• From the network
• Intruders can use all of these to break in

16
How Intruders Get In

• Usually by masquerading as a legitimate user
• Less frequently by inserting commands through
  insecure entry points
• finger daemons
• Holes in electronic mail
• Making use of interpreters that access data
  remotely

17
Detecting Intruders

• The sooner detected, the better
• Systems that detect and eject intruders quickly
  are less attractive targets
• Information gained from detecting intruders can
  be used to prevent further intrusions
• Detection presumes you can differentiate the
  behavior of authorized users and intruders

18
Some Approaches to Detecting Intruders

• Statistical anomaly detection
• Based on either
• Overall system activity
• Individual user profiles
• Rule-based detection
• Rules that detect anomalies
• Penetration expert systems

19
Audit Records

• Keep track of everything done on system
• Powerful tool for detecting intruders
• Used to build detection mechanisms
• Can use either general accounting info or
  specially gathered data
• Also invaluable if you decide to prosecute
• Must be carefully protected to be valuable

20
Malicious Programs

• Clever programmers can get software to do their
  dirty work for them
• Programs have several advantages for these
  purposes
• Speed
• Mutability
• Anonymity

21
Kinds of Malicious Programs

• Trojan horses
• Trapdoors
• Logic bombs
• Worms
• Viruses

22
Trojan Horses

• Seemingly useful program that contains code that
  does harmful things
• Unsuspecting users run the Trojan horse to get
  the advertised benefit
• At which time the Greeks spring out and slaughter
  your system
• Particularly dangerous in compilers

23
Trapdoors

• A secret entry point into an otherwise legitimate
  program
• Typically inserted by the writer of the program
• Most often found in login programs or programs
  that use the network
• But also found in system utilities

24
Logic Bombs

• Like trapdoors, typically in a legitimate program
• A piece of code that, under certain conditions,
  explodes
• Also like trapdoors, typically inserted by
  program authors

25
Worms

• Programs that seek to move from system to system
• Making use of various vulnerabilities
• Other malicious behavior can also be built in
• The Internet worm is the most famous example
• Can spread very, very rapidly

26
Viruses

• A program that can infect other programs
• Infected programs in turn infect others
• Along with mere infection, Trojan horses,
  trapdoors, or logic bombs can be included
• Like worms, viruses can spread very rapidly

27
How do viruses work?

• When a program is run, it typically has the full
  privileges of its running user
• Include write privileges for some other programs
• A virus can use those privileges to replace those
  programs with infected versions

28
Typical Virus Actions

• 1. Find uninfected writable programs
• 2. Modify those programs
• 3. Perform normal actions of infected program
• 4. Do whatever other damage is desired

29
Before the Infected Program Runs
Virus code
Infected program
Uninfected program
30
The Infected Program Runs
Virus code
Infected program
Uninfected program
31
Infecting the Other Program
Virus code
Virus code
Infected program
Infected program
32
How do viruses fit into programs?

• Prepended
• Postpended
• Copy program and replace
• Cleverly fit into the cracks
• Some viruses take other measures to hide
  modifications

33
Dealing with Viruses

• Prevention of infection
• Detection and eradication
• Containment

34
Preventing the Spread of Virus

• Dont import untrusted programs
• But who can you trust?
• Viruses have been found in commercial shrink-wrap
  software
• Trusting someone means not just trusting their
  honesty, but also their caution

35
Other Prevention Measures

• Scan incoming programs for viruses
• Some viruses are designed to hide
• Limit the targets viruses can reach
• Monitor updates to executables carefully
• Requires a broad definition of executable

36
Virus Detection

• Many viruses have detectable signatures
• But some work hard to hide them
• Smart scanners can examine programs for
  virus-like behavior
• Checksums attached to programs can detect
  modifications
• If virus smart enough to generate checksum
  itself, digitally sign it

37
Virus Eradication

• Tedious, because you must be thorough
• Restore clean versions of everything
• Take great care with future restoration of backups

38
Containment

• Run suspicious programs in an encapsulated
  environment
• Limiting their forms of access to prevent virus
  spread
• Requires versatile security model and strong
  protection guarantees

39
Security in Distributed Systems

• A substantially harder problem
• Many single-system mechanisms are based on
  trusting a central operating system
• Single-system mechanisms often assume secure
  communication channels
• Single-system mechanisms can (in principle) have
  access to all relevant data

40
Security Mechanism for Distributed Systems

• Encryption
• Authentication
• Firewalls
• Honeypots

41
Encryption for Distributed Systems

• Can protect secrecy of data while on insecure
  links
• Can also prevent modification and many forms of
  fabrication attacks
• But keys are a tricky issue

42
Encryption Keys and Distributed System Security

• To gain benefit from encryption, communicating
  entities must share a key
• Each separate set of entities need a different
  key
• How do you securely distribute keys?

43
Problems of Key Distribution

• Key must be kept secret
• Key must be generate by trusted authority
• Must be sure key matches intended use
• Must be sure keys arent reused
• Must be quick an automatic

44
Key Distribution Schemes

• Manual distribution by one party
• Use existing key to send new key
• Manual distribution by third party
• Key servers

45
Modulus Arithmetic Background

• 27 12 3, 27 3 (mod 12)
• 15 12 3, 15 3 (mod 12)
• All numbers land on the same point along a
  circles edge are the same

46
Modulus Arithmetic Background

• 11 12 11 (mod 12)
• 16 12 4 (mod 12)
• (11 16) 12
• (11 4) 12 3 (mod 12)
• (11 16) 12
• (11 4) 12 8 (mod 12)

47
Diffie-Hellman Key Exchange

• Need a prime number p
• Need a base integer g between 1 and p 1
• Site X picks x between 1 and p 2
• Site Y picks y between 1 and p 2

• p 13, g 7
• X x 3, Y y 5

48
Diffie-Hellman Key Exchange

• Site X computes
• gx mod p
• Site Y computes
• gy mod p
• Site X and Y exchange public values

• p 13, g 7
• X x 3, Y y 5
• X 73 mod 13
• Y 75 mod 13

49
Diffie-Hellman Key Exchange

• Site X computes
• (gy mod p)x mod p
• Site Y computes
• (gx mod p)y mod p
• Now X and Y have a shared secret
• Problem Prone to man-in-the-middle attacks

• X
• (75 mod 13)3 mod 13 5
• Y
• (73 mod 13)5 mod 13
• 5

50
Key Servers

• Trusted third party that can provide good keys on
  demand
• Typically on a separate machine
• Tremendous care must be taken to ensure secure
  communications with the key server

51
Authentication for Distributed Systems

• When a message comes in over the net, how do you
  tell who sent it?
• Generally with some form of digital signature
• Must be unique to signing user
• And also unique to the message

52
Digital Signatures

• A digital signature is a guarantee that an
  electronic document was created by a particular
  individual
• Basic mechanism for authentication
• Vital for electronic commerce, secure electronic
  mail, etc.
• S signature(M)

53
Desirable Properties of Digital Signatures

• Easy to generate and verify
• Nonforgeable
• Unique
• Nonrepudiable
• Storable

54
Providing Digital Signatures

• Encryption with a secret key has some of these
  properties
• Encrypt entire message
• Check signature by decrypting
• S E(M, Ke)
• But normal encryption has problems for digital
  signatures

55
Problems of Using Encryption for Digital
Signatures

• Both parties can create same message
• With same signature
• One key per pair of users required
• Signature is as large of message
• Poor storage properties
• Hard to handle multiple signatures per message

56
Public Key Encryption

• E(Kpublic, M) ? C
• D(Kprivate, C) ? M
• E(Kprivate, M) ? C
• D(Kpublic, C) ? M

57
Public Key Encryption

• Idea
• Public key is published
• Private key is the secret
• E(Kmy_public, Hi, Andy)
• Anyone can create it, but only I can read it
• E(Kmy_private, Im Andy)
• Everyone can read it, but only I can create it

58
Public Key Encryption

• E(Kyour_public, E(Kmy_private, I know your
  secret))
• Only you can read it, and only I can send it

59
Public Key Cryptography and Digital Signatures

• User X wants to sign a message M sent to user Y
• Calculate a characteristic Z of message M
  (checksum of something similar)
• S E(Z, Kx_private)
• Send both M and S to Y

60
Checking a Public Key Digital Signature

• Y calculates the characteristic ZM of M
• Then Y checks the signature
• Z D(S, Kx_public)
• If ZM Z, the signature is valid

61
Public Key Digital Signature Diagram
M
Sender X
Receiver Y
S
Z checksum(M) S E(Z, Kx_private)
62
Public Key Digital Signature Diagram
Sender X
Receiver Y
M
S
M S
63
Public Key Digital Signature Diagram

• If Z ZM, the signature is valid

Sender X
Receiver Y
M
S
ZM checksum(M)
Z D(S, Kx_public)
64
How does this scheme handle various attacks?

• What if an intruder changes the message?
• What if someone replays a message?
• What if the sender denies a message he sent?
• What if the receiver tries to alter the message?

65
Intruder Alteration Diagram
Sender X
Receiver Y
M
S
Intruder
Intruder
66
Discovering the Alternation

• Z does not equal ZM, so the signature is invalid

Sender X
Receiver Y
M
S
ZM checksum(M)
Z D(S, Kx_public)
67
Replay Diagram
Sender X
Receiver Y
M
S
Intruder
M
Intruder
S
68
Replay Occurs
Sender X
Receiver Y
M
Intruder
S
Intruder
69
How to handle this replay?

• Sequence numbers in messages
• Challenge/response to sender
• Timestamp messages and discard old ones
• Dont worry about it

70
Example Use of Public Key Encryption

• Privacy-Enhanced Electronic Mail (PEM)
• Goals
• Confidentiality
• Origin authentication
• Data integrity
• Non-repudiation of origin (whenever possible)

71
Basic Design Confidentiality

• M message
• KS session key
• KB Bobs public key
• Alice ? Bob
• E(M, KS), E(KS, KB)

72
Basic Design Integrity

• M message
• H(M) hash of message M
• KA Alices private key (non-repudiation)
• Alice ? Bob
• M, E(H(M), KA)

73
Basic Design Everything

• Confidentiality, integrity, authentication
• Alice ? Bob
• E(M, KS), E(KS, KB), E(H(M), KA)

74
Major Challenge in Public Key Cryptography

• How do I find out someones public key?
• If not done securely, the system is totally
  compromised
• Must also be efficient
• And how do I securely store and manage public
  keys?

75
Simple Protocol
Server
Alice wants to communicate securely with Bob
Bob
Alice
76
Simple Protocol
Server
E(Request for session key to Bob, KA)
Bob
Alice
77
Simple Protocol
Server
E(KS, KA), E(KS, KB)
Bob
Alice
78
Simple Protocol
Server
Bob
Alice
E(KS, KB)
79
Problems

• How does Bob know he is talking to Alice?
• Replay attack Eve records M from Alice to Bob,
  later replays it Bob may think hes talking to
  Alice, but he isnt
• Session key reuse Eve replays M from Alice to
  Bob, so Bob re-uses session key
• Protocols must provide authentication and defense
  against replay

80
Authentication Servers

• Like key servers, trusted third parties
• An authentication server can produce a ticket
  that guarantees the identity of a user
• Generally tickets expire
• Kerberos is the most popular authentication server

81
More on Kerberos

• Uses symmetric cryptography
• Servers are trusted by all parties
• Issues tickets that provide secure communications
  between clients and servers
• Tickets have a lifetime, then expire

82
Kerberos in Action
KDC
A client wants to communicate securely with a
server
Server
Client
83
The Client Asks Kerberos for a Ticket
KDC
C, S
Server
Client
84
The Client Asks Kerberos for a Ticket
KDC
E(KC,S, E(TC,S, KS), KC)
Server
Client
85
Whats going on here?

• Whats is in this message?
• TC,S is the ticket that allows the client to
  communicate with the server
• Its encrypted with KS (so only the server can
  read it)
• Message contains a new key KC,S
• Entire message encrypted in Cs key

86
Why the Extra Key?

• For authentication purposes
• Its also contained within the ticket
• Server can authenticate himself to client using
  that key

87
Client Sends Ticket to Server
KDC
Server
Client
E(AC, KC,S), E(TC,S, KS)
88
What does the client send?

• Sends encrypted ticket from Kerberos server
• Which only server can read
• Also sends authenticator AC in session key KC,S
• Server gets KC,S from ticket, sends back altered
  version encrypted with KC,S

89
Firewalls

• A program to allow selective access to the
  network
• In both directions
• Typically, firewalls protect entire networks
• They must examine everything that tries to pass
  into the protected domain
• Only authorized transmissions permitted

90
Firewall Example
Internet
Bastion host (gateway between an inside network
and an outside network)
91
What do firewalls do well?

• Prevent intruders from accessing machines on your
  network
• Prevent your users from inadvertently
  compromising security

92
What do firewalls do badly?

• Prevent many forms of legitimate access
• May get in the way of other forms of security
• Often, theres no further security behind the
  firewall
• So if it fails

93
Honey Pots

• Decoy machines with network accounts
• No legitimate users should access those systems
• If something happens, sound an alarm