Operating Systems

1
Chapter 6
2
Malicious code

• Trojan HorsesA Trojan Horse is an apparently
  useful program that has hidden functions, usually
  harmful. A Trojan Horse can violate integrity
  more easily than confidentiality.
• Viruses A virus is a program that attaches
  itself to another program, propagates, and
  usually causes some data destruction. General
  virus detection is an undecidable problem, but we
  can detect specific viruses.
• WormsA worm is a program that propagates itself
  without infecting the host.

3
Viruses and worms

• Self-propagating
• May destroy information and clog services
• A mix of vandalism and ego trip
• Take advantage of operating system and utilities
  flaws and uniformity of systems
• Examples of malicious software

4
Program Threats

• Trap door
• Code segment that misuses its environment and
  allows malicious code (trojan horse) to execute.
• Exploits mechanisms for allowing programs written
  by users to be executed by other users.
• Trap Door examples
• Specific user identifier or password that
  circumvents normal security procedures.
• Could be included in a compiler (Kernighans C
  compiler).
• Stack and Buffer Overflow - Exploits a bug in a
  program (overflow either the stack or memory
  buffers.)

5
Trap Doors

• (a) Normal code.
• (b) Code with a trapdoor inserted

6
Logic Bombs

• Company programmer writes program
• potential to do harm
• OK as long as he/she enters password daily
• ff programmer fired, no password and bomb explodes

7
Login Spoofing

• (a) Correct login screen
• (b) Phony login screen

8
Buffer Overflow

• (a) Situation when main program is running
• (b) After program A called
• (c) Buffer overflow shown in gray

9
Buffer Overflow example

• define BUFSIZE 4
• void myFunc(char someBuf)
• char bufBUFSIZE
• printf printf ("Copying ...")
• / Let's do a REALLY stupid thing here /
• strcpy(buf,someBuf) / NO Bounds check! /
• printf ("Returning... \n")
• The example above shows how overflows are
  performed, in practice. An unsafe C standard
  library function (in this case, strcpy(), which
  knows no bounds), copies a buffer, and exceeds
  the allocated space.

10
Operating System SecurityTrojan Horses

• Free program made available to unsuspecting user
• Actually contains code to do harm
• Place altered version of utility program on
  victim's computer
• trick user into running that program

11
The Internet worm of 1968

• Guessed and tried passwords with the use of a
  dictionary
• Exploited bugs in the finger and sendmail
  programs, including buffer overflow.
• Took advantage of the fact that some systems
  specify trusted computers from which remote
  logins are accepted without checking passwords.
• Used encryption and other ways to hide its
  existence (a polymorphic virus).

12
The Morris Internet Worm
13
Viruses - program segments

• Program segments attached to another executable
• When program is run, the virus proliferates
• actually a type of Trojan horse but not
  personal..
• Numerous possible actions
• erasing, modifying, encrypting files
• display extortion note send
• damage boot sector on hard disk (or ask for
  password..)
• Defenses
• look for known viruses in files
• perform checksum on files
• make directories of binary files unwritable for
  users...

14
How Viruses Work (1)

• Virus written in assembly language
• Inserted into another program
• use tool called a dropper
• Virus dormant until program executed
• then infects other programs
• eventually executes its payload

15
How Viruses Work (2)

• Recursive procedure that finds executable files
  on a UNIX system
• Virus could
• infect them all

16
How Viruses Work (3)

• An executable program
• With a virus at the front
• With the virus at the end
• With a virus spread over free space within
  program

17
How Viruses Work (4)

• After virus has captured interrupt, trap vectors
• After OS has retaken printer interrupt vector
• After virus has noticed loss of printer interrupt
  vector and recaptured it

18
Virus Damage Scenarios

• Blackmail(ask for money for decryption key)
• Denial of service as long as virus runs(fill out
  disk, clog CPU or memory)
• Permanently damage hardware (BIOS)
• Target a competitor's computer
• do harm
• Espionage
• Intra-corporate dirty tricks
• sabotage another corporate officer's files

19
Network Security

• External threat
• code transmitted to target machine
• code executed there, doing damage
• Goals of virus writer
• quickly spreading virus
• difficult to detect
• hard to get rid of
• Virus program can reproduce itself
• attach its code to another program
• additionally, do harm

20
How Viruses Spread

• Virus placed where likely to be copied
• When copied
• infects programs on hard drive, floppy
• may try to spread over LAN
• Attach to innocent looking email
• when it runs, use mailing list to
  replicate(News-group)
• Hide insite application macros (Ms-word, MS-excel)

21
Antivirus and Anti-Antivirus Techniques

• (a) A program
• (b) Infected program
• (c) Compressed infected program
• (d) Encrypted virus
• (e) Compressed virus with encrypted compression
  code

22
Antivirus and Anti-Antivirus Techniques

• Examples of a polymorphic virus
• All of these examples do the same thing

23
Antivirus and Anti-Antivirus Techniques

• Integrity checkers
• Behavioral checkers
• Virus avoidance
• good OS
• install only shrink-wrapped software
• use antivirus software
• do not click on attachments to email
• frequent backups
• Recovery from virus attack
• halt computer, reboot from safe disk, run
  antivirus

24
Information Flow Controls

• 1. Assignment b e
• 2. Compound begin S1 . . . Sn end
• 3. Alternation if e then S1 else S2
• 4. Iteration while e do S1
• 5. Call q(a1, . . . , am, b1, . . . bn)

25
Information Flow Controls, cont.

• Security Conditions for AssignmentExecution of
  an assignmentb eis secure if e lt b

26
Information Flow Controls, cont.

• Security Conditions for CompoundExecution of
  the statement begin S1 . . . Sn end
• if each of the Si is secure

27
Information Flow Controls, cont.

• Security Conditions for Alternation Execution
  of the statement if e then S1 else S2is
  secure if(i) Execution of S1 and S2 is secure,
  and(ii) e lt S, where S S1 ? S2 and S1
  ?b?b is a target of an assignment in S1), S2
  ?b?b is a target of an assignment in S2)

28
Information Flow Controls, cont.

• ExampleFor the following statement if x gt y
  then begin z w i k
  1 end,condition (ii) is given by x ? y lt z ? I

29
Execution-Based Mechanisms

• Secure execution of the if statement
• if x1 then y1
• Is described by
• if x1
• then if x?y then y1 else skip
• else skip.
• Suppose x is 0 or 1, y is initially 0, xHigh,
  and yLow thus, the flow x?y is not secure.
  Because the assignment to y is skipped both when
  x1 (because the security check fails) and when
  x0 (because the test x1 fails), y is always 0
  when the statement terminates, thereby giving no
  information about x. note that if an error flag E
  is set to 1 when the security check fails, then
  the value of x is encoded in the flag (E1
  implies x1, E0 implies x0).

30
Execution-Based Mechanisms, cont.

• Example
• Consider the statement
• if x1 then y1 else z1
• where xHigh. Suppose that when x1, yHigh and
  zLow, but when x?1, yLow and zHigh. If both
  relations x?y and x ?z are tested on both
  branches, the program will be rejected, even
  though it can be securely executed using Fentons
  approach. (Verification of this is left to the
  reader).

31
Execution Based Mechanism

• procedure copy1 (x integer
• var y integer)
• copy x to y
• var z integer
• begin
• y 0 Low ? z
• z 0 Low ? y
• if x 0 then z 1
• if z 0 then y 1
• end
• end copy1
• Either l(X)ltl(Z) is tested or l(Z)lt l(Y) is
  tested but not both!
• With Compiler both are tested!

32
Covert Channels (1)
Encapsulated server can still leak to
collaborator via covert channels
Client, server and collaborator processes
33
Security Policies
Control of Information Flow
if profit lt 0 then delete file T else
begin write file T, text close file T
end
U1
The existence of file T (regardless of its label)
gives information about the value of profit. Its
content even more
34
Covert Channels (2)

• A covert channel using file locking

35
Targeted Malicious Code
36
Covert Timing Channel
37
Covert Channels (3)

• Pictures appear the same
• Picture on right has text of 5 Shakespeare plays
• encrypted, inserted into low order bits of color
  values

Hamlet, Macbeth, Julius Caesar Merchant of
Venice, King Lear
Zebras
38
Fighting Covert Channels (after Detection)

• Deletion of the channel if feasible
• Add randop noise on the channel
• Reduce the channel bandwidth, useful for Timing
  channels
• Audit the channel
• Ignore, danger is not very serious

39
Language-Based Protection

• Specification of protection in a programming
  language allows the high-level description of
  policies for the allocation and use of
  resources.
• Language implementation can provide software for
  protection enforcement when automatic
  hardware-supported checking is unavailable.
• Interpret protection specifications to generate
  calls on whatever protection system is provided
  by the hardware and the operating system.

40
Security in Java

• Language features verified by the Bytecode
  verifier
• Sandbox model using the Security manager
• No bypass of the security manager using the
  Class-loader

41
Security in Java Language Features

• No typedef or define
• No automatic type conversion
• No casting
• Strongly typed language
• Indexes to arrays and strings range validated
• Automatic garbage collection
• Use of OO features like Public, Private

42
Java Security (1)

• A type safe language
• compiler rejects attempts to misuse variable
• Bytecode Verifier Checks include
• 1. Attempts to forge pointers
• 2. Violation of access restrictions on private
  class members
• 3. Misuse of variables by type
• 4. Generation of stack over/underflows
• 5. Illegal conversion of variables to another
  type AND
• 6. All system calls are done via the Security
  Manager!

43
The Java Sandbox
Server
Web page
Applet
Applet (Byte code)
44
????? ?? ???? ?????
45
Mobile Code (1) Sandboxing

• (a) Memory divided into 1-MB sandboxes
• (b) One way of checking an instruction for
  validity

46
Mobile Code (2)

• Applets can be interpreted by a Web browser

47
Mobile Code (3)

• How code signing works

48
Protection in Java 1.2

• Protection is handled by the Java Virtual Machine
  (JVM)
• A class is assigned a protection domain when it
  is loaded by the JVM.
• The protection domain indicates what operations
  the class can (and cannot) perform.
• If a library method is invoked that performs a
  privileged operation, the stack is inspected to
  ensure the operation can be performed by the
  library.

49
Java Security (2)

• Examples of specified protection with JDK 1.2