OMSE 510: Computing Foundations 9: Course Cleanup

1
OMSE 510 Computing Foundations9 Course Cleanup

• Chris Gilmore ltgrimjack_at_cs.pdx.edugt
• Portland State University/OMSE

Material Borrowed from Jon Walpoles lectures
2
Today

• Memory Management Finish
• Filesystems
• Final Sample questions

3
Memory protection

• How is protection checking implemented?
• compare page protection bits with process
  capabilities and operation types on every
  load/store
• sounds expensive!
• Requires hardware support!
• How can protection checking be done efficiently?
• Use the TLB as a protection look-aside buffer
• Use special segment registers

4
Protection lookaside buffer

• A TLB is often used for more than just
  translation
• Memory accesses need to be checked for validity
• Does the address refer to an allocated segment of
  the address space?
• If not segmentation fault!
• Is this process allowed to access this memory
  segment?
• If not segmentation/protection fault!
• Is the type of access valid for this segment?
• Read, write, execute ?
• If not protection fault!

5
Page-grain protection checking with a TLB
6
Page sharing

• In a large multiprogramming system...
• Some users run the same program at the same time
• Why have more than one copy of the same page in
  memory???
• Goal
• Share pages among processes (not just threads!)
• Cannot share writable pages
• If writable pages were shared processes would
  notice each others effects
• Text segment can be shared

7
Page sharing
Physical memory
Process 1 address space
Process 1 page table
Stack (rw)
Process 2 address space
Process 2 page table
Data (rw) Instructions (rx)
8
Page sharing

• Fork system call
• Copy the parents virtual address space
• ... and immediately do an Exec system call
• Exec overwrites the calling address space with
  the contents of an executable file (ie a new
  program)
• Desired Semantics
• pages are copied, not shared
• Observations
• Copying every page in an address space is
  expensive!
• processes cant notice the difference between
  copying and sharing unless pages are modified!

9
Page sharing

• Idea Copy-On-Write
• Initialize new page table, but point entries to
  existing page frames of parent
• Share pages
• Temporarily mark all pages read-only
• Share all pages until a protection fault occurs
• Protection fault (copy-on-write fault)
• Is this page really read only or is it writable
  but temporarily protected for copy-on-write?
• If it is writable
• copy the page
• mark both copies writable
• resume execution as if no fault occurred

10
On Page replacement..

• Paging performance
• Paging works best if there are plenty of free
  frames.
• If all pages are full of dirty pages...
• Must perform 2 disk operations for each page
  fault

11
Page replacement

• Assume a normal page table
• User-program is executing
• A PageInvalidFault occurs!
• The page needed is not in memory
• Select some frame and remove the page in it
• If it has been modified, it must be written back
  to disk
• the dirty bit in its page table entry tells us
  if this is necessary
• Figure out which page was needed from the
  faulting addr
• Read the needed page into this frame
• Restart the interrupted process by retrying the
  same instruction

12
Page replacement algorithms

• Which frame to replace?
• Algorithms
• The Optimal Algorithm
• First In First Out (FIFO)
• Not Recently Used (NRU)
• Second Chance / Clock
• Least Recently Used (LRU)
• Not Frequently Used (NFU)
• Working Set (WS)
• WSClock

13
The optimal page replacement algorithm

• Idea
• Select the page that will not be needed for the
  longest time

14
Optimal page replacement

• Replace the page that will not be needed for the
  longest
• Example

a a a a b b b b c c c c
d d d d X
15
Optimal page replacement

• Select the page that will not be needed for the
  longest time
• Example

a a a a a a a a a b b b b
b b b b b c c c c c c c c
c d d d d e e e e e
X X
16
The optimal page replacement algorithm

• Idea
• Select the page that will not be needed for the
  longest time
• Problem
• Cant know the future of a program
• Cant know when a given page will be needed next
• The optimal algorithm is unrealizable

17
The optimal page replacement algorithm

• However
• We can use it as a control case for simulation
  studies
• Run the program once
• Generate a log of all memory references
• Use the log to simulate various page replacement
  algorithms
• Can compare others to optimal algorithm

18
FIFO page replacement algorithm

• Always replace the oldest page
• Replace the page that has been in memory for the
  longest time.

19
FIFO page replacement algorithm

• Replace the page that was first brought into
  memory
• Example Memory system with 4 frames

a a a b c c c c
d d X
20
FIFO page replacement algorithm

• Replace the page that was first brought into
  memory
• Example Memory system with 4 frames

a a a a a b b b c c
c c e e d d d d
X
21
FIFO page replacement algorithm

• Replace the page that was first brought into
  memory
• Example Memory system with 4 frames

a a a a a a a c c
b b b b b b b c c c c e e
e e e e d d d d d d d
a X X X
22
FIFO page replacement algorithm

• Always replace the oldest page.
• Replace the page that has been in memory for the
  longest time.
• Implementation
• Maintain a linked list of all pages in memory
• Keep it in order of when they came into memory
• The page at the front of the list is oldest
• Add new page to end of list

23
FIFO page replacement algorithm

• Disadvantage
• The oldest page may be needed again soon
• Some page may be important throughout execution
• It will get old, but replacing it will cause an
  immediate page fault

24
Page table referenced and dirty bits

• Each page table entry (and TLB entry) has a
• Referenced bit - set by TLB when page read /
  written
• Dirty / modified bit - set when page is written
• If TLB entry for this page is valid, it has the
  most up to date version of these bits for the
  page
• OS must copy them into the page table entry
  during fault handling
• On Some Hardware...
• ReadOnly bit but no dirty bit

25
Page table referenced and dirty bits

• Idea
• Software sets the ReadOnly bit for all pages
• When program tries to update the page...
• A trap occurs
• Software sets the Dirty Bit and clears the
  ReadOnly bit
• Resumes execution of the program

26
Not recently used page replacement alg.

• Use the Referenced Bit and the Dirty Bit
• Initially, all pages have
• Referenced Bit 0
• Dirty Bit 0
• Periodically... (e.g. whenever a timer interrupt
  occurs)
• Clear the Referenced Bit

27
Not recently used page replacement alg.

• When a page fault occurs...
• Categorize each page...
• Class 1 Referenced 0 Dirty 0
• Class 2 Referenced 0 Dirty 1
• Class 3 Referenced 1 Dirty 0
• Class 4 Referenced 1 Dirty 1
• Choose a victim page from class 1 why?
• If none, choose a page from class 2 why?
• If none, choose a page from class 3 why?
• If none, choose a page from class 4 why?

28
Second chance page replacement alg.

• Modification to FIFO
• Pages kept in a linked list
• Oldest is at the front of the list
• Look at the oldest page
• If its referenced bit is 0...
• Select it for replacement
• Else
• It was used recently dont want to replace it
• Clear its referenced bit
• Move it to the end of the list
• Repeat
• What if every page was used in last clock tick?
• Select a page at random

29
Clock algorithm (same as second chance)

• Maintain a circular list of pages in memory
• Set a bit for the page when a page is referenced
• Clock sweeps over memory looking for a victim
  page that does not have the referenced bit set
• If the bit is set, clear it and move on to the
  next page
• Replaces pages that havent been referenced for
  one complete clock revolution

30
Least recently used algorithm (LRU)

• Keep track of when a page is used.
• Replace the page that has been used least
  recently.

31
LRU page replacement

• Replace the page that hasnt been referenced in
  the longest time

32
LRU page replacement

• Replace the page that hasnt been referenced in
  the longest time

a a a a a a a a a a b b b
b b b b b b b c c c c e e
e e e d d d d d d d d d c
c X X X
33
Least recently used algorithm (LRU)

• But how can we implement this?
• Implementation 1
• Keep a linked list of all pages
• On every memory reference,
• Move that page to the front of the list.
• The page at the tail of the list is replaced.
• on every memory reference...
• Not feasible in software

34
LRU implementation

• Take referenced and put at head of list

35
LRU implementation

• Take referenced and put at head of list

a a b b c c d d
36
LRU implementation

• Take referenced and put at head of list

a a a a b b b b c c c c d d
d d X
37
LRU implementation

• Take referenced and put at head of list

a a a a a a a a a a b b b
b b b b b b b c c c c e e
e e e d d d d d d d d d c
c X X X
38
Least recently used algorithm (LRU)

• But how can we implement this?
• without requiring every access to be recorded?
• Implementation 2
• MMU (hardware) maintains a counter
• Incremented on every clock cycle
• Every time a page table entry is used
• MMU writes the value to the entry
• timestamp / time-of-last-use
• When a page fault occurs
• Software looks through the page table
• Idenitifies the entry with the oldest timestamp

39
Not frequently used algorithm (NFU)

• Associate a counter with each page
• On every clock interrupt, the OS looks at each
  page.
• If the Reference Bit is set...
• Increment that pages counter clear the bit.
• The counter approximates how often the page is
  used.
• For replacement, choose the page with lowest
  counter.

40
Not frequently used algorithm (NFU)

• Problem
• Some page may be heavily used
• ---gt Its counter is large
• The programs behavior changes
• Now, this page is not used ever again (or only
  rarely)
• This algorithm never forgets!
• This page will never be chosen for replacement!

41
Modified NFU with aging

• Associate a counter with each page
• On every clock tick, the OS looks at each page.
• Shift the counter right 1 bit (divide its value
  by 2)
• If the Reference Bit is set...
• Set the most-significant bit
• Clear the Referenced Bit
• 100000 32
• 010000 16
• 001000 8
• 000100 4
• 100010 34
• 111111 63

42
Paged Memory Mangement

• Concepts.

43
Working set page replacement

• Demand paging
• Pages are only loaded when accessed
• When process begins, all pages marked INVALID
• Locality of Reference
• Processes tend to use only a small fraction of
  their pages
• Working Set
• The set of pages a process needs
• If working set is in memory, no page faults
• What if you cant get working set into memory?

44
Working set page replacement

• Thrashing
• If you cant get working set into memory pages
  fault every few instructions
• No work gets done

45
Working set page replacement

• Prepaging (prefetching)
• Load pages before they are needed
• Main idea
• Indentify the processs working set
• How big is the working set?
• Look at the last K memory references
• As K gets bigger, more pages needed.
• In the limit, all pages are needed.

46
WSClock page replacement algorithm

• All pages are kept in a circular list (ring)
• As pages are added, they go into the ring.
• The clock hand advances around the ring.
• Each entry contains time of last use.
• Upon a page fault...
• If Reference Bit 1...
• Page is in use now. Do not evict.
• Clear the Referenced Bit.
• Update the time of last use field.

47
WSClock page replacement algorithm

• If Reference Bit 0
• If the age of the page is less than T...
• This page is in the working set.
• Advance the hand and keep looking
• If the age of the page is greater than T...
• If page is clean
• Reclaim the frame and we are done!
• If page is dirty
• Schedule a write for the page
• Advance the hand and keep looking

48
Summary
49
Theory and practice

• Identifying victim frame on each page fault
  typically requires two disk accesses per page
  fault
• Alternative ? the O.S. can keep several pages
  free in anticipation of upcoming page faults.
• In Unix low and high water marks

50
Free pages and the clock algorithm

• The rate at which the clock sweeps through memory
  determines the number of pages that are kept
  free
• Too high a rate --gt Too many free pages marked
• Too low a rate --gt Not enough (or no) free pages
  marked
• Large memory system considerations
• As memory systems grow, it takes longer and
  longer for the hand to sweep through memory
• This washes out the effect of the clock somewhat
• Can use a two-handed clock to reduce the time
  between the passing of the hands

51
The UNIX memory model

• UNIX page replacement
• Two handed clock algorithm for page replacement
• If page has not been accessed move it to the free
  list for use as allocatable page
• If modified/dirty ? write to disk (still keep
  stuff in memory though)
• If unmodified ? just move to free list
• High and low water marks for free pages
• Pages on the free-list can be re-allocated if
  they are accessed again before being overwritten

52
Onto the Filesystem!
53
Why do we need a file system?

• Must store large amounts of data
• Data must survive the termination of the process
  that created it
• Called persistence
• Multiple processes must be able to access the
  information concurrently

54
What is a file?

• Files can be structured or unstructured
• Unstructured just a sequence of bytes
• Structured a sequence or tree of typed records
• In Unix-based operating systems a file is an
  unstructured sequence of bytes

55
File Structure
Sequence of bytes
Sequence of records
Tree of records
56
File extensions

• Even though files are just a sequence of bytes,
  programs can impose structure on them, by
  convention
• Files with a certain standard structure imposed
  can be identified using an extension to their name

57
Typical file extensions
58
Executable and archive file formats
An archive
An executable file
59
File attributes

• Various meta-data needs to be associated with
  files
• Owner
• Creation time
• Access permissions / protection
• Size etc
• This meta-data is called the file attributes
• Maintained in file system data structures for
  each file

60
Example file attributes
61
File access

• Sequential Access
• read all bytes/records from the beginning
• cannot jump around (but could rewind or back up)
  convenient when medium was magnetic tape
• Random Access
• can read bytes (or records) in any order
• essential for database systems
• option 1
• move position, then read
• option 2
• perform read, then update current position

62
Some file-related system calls

• Create a file
• Delete a file
• Open
• Close
• Read (n bytes from current position)
• Write (n bytes to current position)
• Append (n bytes to end of file)
• Seek (move to new position)
• Get attributes
• Set/modify attributes
• Rename file

63
File-related system calls

• fd open (name, mode)
• byte_count read (fd, buffer, buffer_size)
• byte_count write (fd, buffer, num_bytes)
• close (fd)

64
File storage on disk

• Sector 0 Master Boot Record (MBR)
• Contains the partition map
• Rest of disk divided into partitions
• Partition sequence of consecutive sectors.
• Each partition can hold its own file system
• Unix file system
• Window file system
• Apple file system
• Every partition starts with a boot block
• Contains a small program
• This boot program reads in an OS from the file
  system in that partition
• OS Startup
• Bios reads MBR , then reads execs a boot block

65
An example disk
66
An example disk
Unix File System
67
File bytes vs disk sectors

• Files are sequences of bytes
• Granularity of file I/O is bytes
• Disks are arrays of sectors (512 bytes)
• Granularity of disk I/O is sectors
• Files data must be stored in sectors
• File systems define a block size
• block size 2n sector size
• Contiguous sectors are allocated to a block
• File systems view the disk as an array of blocks
• Must allocate blocks to file
• Must manage free space on disk

68
Contiguous allocation

• Idea
• All blocks in a file are contiguous on the disk

After deleting D and F...
69
Contiguous allocation

• Idea
• All blocks in a file are contiguous on the disk.

After deleting D and F...
70
Contiguous allocation

• Advantages
• Simple to implement (Need only starting sector
  length of file)
• Performance is good (for sequential reading)

71
Contiguous allocation

• Advantages
• Simple to implement (Need only starting sector
  length of file)
• Performance is good (for sequential reading)
• Disadvantages
• After deletions, disk becomes fragmented
• Will need periodic compaction (time-consuming)
• Will need to manage free lists
• If new file put at end of disk...
• No problem
• If new file is put into a hole...
• Must know a files maximum possible size ... at
  the time it is created!

72
Contiguous allocation

• Good for CD-ROMs
• All file sizes are known in advance
• Files are never deleted

73
Linked list allocation

• Each file is a sequence of blocks
• First word in each block contains number of next
  block

74
Linked list allocation

• Each file is a sequence of blocks
• First word in each block contains number of next
  block

Random access into the file is slow!
75
I-nodes

• Each I-Node (index-node) is a structure /
  record
• Contains info about the file
• Attributes
• Location of the blocks containing the file

Blocks on disk
Other attributes
I-node
76
I-nodes

• Each I-Node (index-node) is a structure /
  record
• Contains info about the file
• Attributes
• Location of the blocks containing the file

Enough space for 10 pointers
Blocks on disk
Other attributes
I-node
77
I-nodes

• Each I-Node (index-node) is a structure /
  record
• Contains info about the file
• Attributes
• Location of the blocks containing the file

Enough space for 10 pointers
Blocks on disk
Other attributes
I-node
78
The UNIX I-node entries
Structure of an I-Node
79
The UNIX I-node
80
The UNIX File System
81
The UNIX file system

• The layout of the disk

82
Naming files

• How do we find a file given its name?
• How can we ensure that file names are unique?

83
Single level directories

• Folder
• Single-Level Directory Systems
• Early OSs
• Problem
• Sharing amongst users
• Appropriate for small, embedded systems

Root Directory
c
d
a
b
84
Two-level directory systems

• Letters indicate who owns the file / directory.
• Each user has a directory.
• /peter/g

Root Directory
harry
peter
todd
micah
c
a
b
c
d
e
d
g
a
e
b
85
Hierarchical directory systems

• A tree of directories
• Interior nodes Directories
• Leaves Files

/
C
B
A
E
D
F
i
j
k
l
m
n
G
H
o
p
q
86
Hierarchical directory systems
Root Directory

• A tree of directories
• Interior nodes Directories
• Leaves Files

/
C
B
A
Users Directories
E
D
F
i
j
k
l
m
n
G
H
Sub-directories
o
p
q
87
Path names

• MULTICS
• gtusrgtharrygtmailbox
• Windows
• \usr\harry\mailbox
• Unix
• /usr/harry/mailbox
• Absolute Path Name
• /usr/harry/mailbox
• Relative Path Name
• working directory (or current directory)
• mailbox

Each process has its own working directory
88
A Unix directory tree
. is the current directory .. is the parent
89
Sharing files

• One file appears in several directories.
• Tree ? DAG (Directed Acyclic Graph)

/
C
B
A
E
D
F
i
j
k
l
m
What if the file changes? New disk blocks are
used. Better not store this info in the
directories!!!
G
H
n
o
p
q
90
Sharing files

• One file appears in several directories.
• Tree ? DAG (Directed Acyclic Graph)

/
C
B
A
E
D
F
i
j
k
l
m
G
H
n
o
p
q
91
Hard links and symbolic links

• In Unix
• Hard links
• Both directories point to the same i-node
• Symbolic links
• One directory points to the files i-node
• Other directory contains the path

92
Hard links
/
C
B
A
E
D
F
i
j
k
l
m
G
H
n
o
p
q
93
Hard links

• Assume i-node number of n is 45.

/
C
B
A
E
D
F
i
j
k
l
m
G
H
n
o
p
q
94
Hard links

• Assume i-node number of n is 45.

/
Directory D
m
123
n
45
C
B
A


E
D
F
i
j
k
l
Directory G
n
45
m
G
H
o
87


n
o
p
q
95
Hard links

• Assume i-node number of n is 45.

The file may have a different name in each
directory
/B/D/n1 /C/F/G/n2
/
Directory D
m
123
n
45
C
B
A


E
D
F
i
j
k
l
Directory G
n
45
m
G
H
o
87


n
o
p
q
96
Symbolic links

• Assume i-node number of n is 45.

/
C
B
A
Hard Link
E
D
F
i
j
k
l
m
G
H
n
o
p
q
Symbolic Link
97
Symbolic links

• Assume i-node number of n is 45.

/
Directory D
m
123
n
45
C
B
A


Hard Link
E
D
F
i
j
k
l
m
G
H
n
o
p
q
Symbolic Link
98
Symbolic links

• Assume i-node number of n is 45.

/
Directory D
m
123
n
45
C
B
A


Hard Link
E
D
F
i
j
k
l
Directory G
n
/B/D/n
m
G
H
o
87


n
o
p
q
Symbolic Link
99
Symbolic links

• Assume i-node number of n is 45.

/
Directory D
m
123
n
45
C
B
A


E
D
F
i
j
k
l
Directory G
n
91
m
n
G
H
o
87
Separate file i-node 91


o
p
q
Symbolic Link
/B/D/n
100
Other Issues

• Managing the disk
• Organizing faster seeks
• Dealing with bad sectors
• Grouping data
• Dealing with fragmentation
• File System Management
• Dealing with inconsistencies
• Networked file systems
• Performance
• Caches
• Read/Write Buffers

101
Sample Final Questions

• What is the difference between threadsafe and
  reentrant? Why would you prefer one over the
  other? Examples?
• Why do modern operating systems use variations on
  round robin instead of shortest job first for
  scheduling?
• Why would do we use semaphores?
• Why would we choose to use a semaphore over a
  monitor?
• What is the difference between a condition
  variable and a semaphore?

102
Sample Final Questions

• Describe two page replacement algorithms
• Why do we need a virtual address and a physical
  address?
• Describe how modern hardware translates between
  the two.
• What are the advantages of page sharing?
• Describe/draw a diagram of all the steps that
  occur to manage a page fault.

103
Sample Final Questions

• Describe LRU, FIFO, second chance, NFU, WSClock
• Describe the steps in performing a context switch
• Describe the steps in performing a system call
• Draw a diagram and explain a multilevel page
  table, and an inverted page table
• Why do write multi-threaded applications? What
  are the drawbacks?
• What is meant by a race-condition? Critical
  Section? Atomic instruction? Give an example.

104
Sample Final Questions

• Describe and outline a solution for sleeping
  barber/dining philosophers/consumer-producer
• Whats the difference between Hoare and Mesa
  semantics for Monitors, provide an example to
  illustrate
• Whats the difference between internal and
  external fragmentation, provide an example
• Whats thrashing?
• What do we have to consider when choosing the
  difference between our page sizes?