Outline

1
Outline

• File Management
• Structured files
• Low-level file implementations

2
Operating System Components
3
Why Programmers Need Files
4
File system context
5
Fig 13-2 The External View of the File Manager
Application Program
mount()
write()
open()
close()
read()
lseek()
File Mgr
Device Mgr
Memory Mgr
File Mgr
Device Mgr
Memory Mgr
Process Mgr
Process Mgr
UNIX
Windows
Hardware
6
Levels in a file system
7
Information Structure
8
Logical structures in a file
9
Low-Level Files
10
File systems

• File system
• A data structure on a disk that holds files
• actually a file system is in a disk partition
• a technical term different from a file system
  as the part of the OS that implements files
• File systems in different OSs have different
  internal structures

11
A file system layout
12
File system descriptor

• The data structure that defines the file system
• Typical fields
• size of the file system (in blocks)
• size of the file descriptor area
• first block in the free block list
• location of the file descriptor of the root
  directory of the file system
• times the file system was created, last modified,
  and last used

13
File system layout variations

• MS/DOS uses a FAT (file allocation table) file
  system
• so does the Macintosh OS (although the MacOS
  layout is different)
• New UNIX file systems use cylinder groups
  (mini-file systems) to achieve better locality of
  file data

14
Locating file data

• The logical file is divided into logical blocks
• Each logical block is mapped to a physical disk
  block
• The file descriptor contains data on how to
  perform this mapping
• there are many methods for performing this
  mapping
• we will look at several of them

15
Dividing a file into blocks
16
Contiguous Allocation

• Each file occupies a set of contiguous blocks on
  the disk
• Simple only starting location and length are
  required
• Random access
• Wasteful of space (dynamic storage-allocation
  problem)
• Files cannot grow
• Mapping from logical to physical
• Block to be accessed Q starting address
• Displacement into block R

17
A contiguous file
18
A contiguous file cont.
19
Keeping a file in pieces

• We need a block pointer for each logical block,
  an array of block pointers
• block mapping indexes into this array
• Each file is a linked list of disk blocks
• But where do we keep this array?
• usually it is not kept as contiguous array
• the array of disk pointers is like a second
  related file (that is 1/1024 as big)

20
Block pointers in the file descriptor
21
Block pointers in contiguous disk blocks
22
Block pointers in the blocks
23
Block pointers in the blocks cont.
24
Block pointers in an index block
25
Block pointers in an index block cont.
26
Chained index blocks
27
Two-level index blocks
28
Two-level index blocks cont.
29
The UNIX hybrid method
30
The UNIX hybrid method cont.
31
Inverted disk block index (FAT)
32
DOS FAT Files
33
Free-Space Management

• Bit vector (n blocks)

0
1
2
n-1

1 ? blocki free 0 ? blocki occupied
biti
???

• First free block number

(number of bits per word) (number of 0-value
words) offset of first 1 bit
34
Free-Space Management - cont.

• Bit map requires extra space. Example
• block size 212 bytes
• disk size 230 bytes (1 gigabyte)
• n 230/212 218 bits (or 32K bytes)
• Easy to get contiguous files
• Linked list (free list)
• Cannot get contiguous space easily
• No waste of space

35
Free list organization
36
Free-Space Management - cont.

• Need to protect
• Pointer to free list
• Bit map
• Must be kept on disk
• Copy in memory and disk may differ.
• Cannot allow for blocki to have a situation
  where biti 0 in memory and biti 1 on
  disk.
• Solution
• Set biti 0 in disk.
• Allocate blocki
• Set biti 0 in memory

37
Implementing Low Level Files

• Secondary storage device contains
• Volume directory (sometimes a root directory for
  a file system)
• External file descriptor for each file
• The file contents
• Manages blocks
• Assigns blocks to files (descriptor keeps track)
• Keeps track of available blocks
• Maps to/from byte stream

38
Disk Organization
Boot Sector
Volume Directory
Blk0
Blk1
Blkk-1

Track 0, Cylinder 0

Blkk
Blkk1
Blk2k-1
Track 0, Cylinder 1


Blk
Blk
Blk
Track 1, Cylinder 0


Blk
Blk
Blk
Track N-1, Cylinder 0


Blk
Blk
Blk
Track N-1, Cylinder M-1
39
Low-level File System Architecture
Block 0


b0 b1 b2 b3
bn-1
.
.
.
Randomly Accessed Device
Sequential Device
40
File Descriptors

• External name
• Current state
• Sharable
• Owner
• User
• Locks
• Protection settings
• Length
• Time of creation
• Time of last modification
• Time of last access
• Reference count
• Storage device details

41
An open() Operation

• Locate the on-device (external) file descriptor
• Extract info needed to read/write file
• Authenticate that process can access the file
• Create an internal file descriptor in primary
  memory
• Create an entry in a per process open file
  status table
• Allocate resources, e.g., buffers, to support
  file usage

42
File Manager Data Structures
Keep the state of the process-file session
2
Copy info from external to the open file
descriptor
1
Open File Descriptor
Process-File Session
Return a reference to the data structure
3
External File Descriptor
43
Opening a UNIX File
44
Reading and Writing the Byte Stream

• Two stages
• Reading bytes into or writing bytes out of the
  memory copy of the block
• Reading the physical blocks into or writing them
  out of memory from/to storage devices
• Packing or unmarshalling procedure converts
  secondary storage blocks into a byte stream
• Unpacking or marshalling procedure converts a
  byte stream into blocks

45
Marshalling the Byte Stream

• Must read at least one buffer ahead on input
• Must write at least one buffer behind on output
• Seek ? flushing the current buffer and finding
  the correct one to load into memory
• Inserting/deleting bytes in the interior of the
  stream

46
Full Block Buffering

• Storage devices use block I/O
• Files place an explicit order on the bytes
• Therefore, it is possible to predict what is
  likely to be read after a byte
• When file is opened, manager reads as many blocks
  ahead as feasible
• After a block is logically written, it is queued
  for writing behind, whenever the disk is
  available
• Buffer pool usually variably sized, depending
  on virtual memory needs
• Interaction with the device manager and memory
  manager

47
Supporting Other Storage Abstractions

• Low-level file systems avoid encoding
  record-level functionality
• If applications use very large or very small
  records, a generic file manager may not be
  efficient
• Some operating systems provide a higher-layer
  file system to support applications with large or
  small files
• Database management systems and multimedia
  documents are examples

48
Structured Files
49
Record-Oriented Sequential Files
50
Electronic Mail Example
51
Indexed Sequential Files
52
Database Management Systems

• A database is a very highly structured set of
  information
• Stored across different files
• Optimized to minimize access time
• DBMSs implementation
• Some DBMSs use the normal files provided by the
  OS for generic use
• Some use their own storage device block

53
Disk compaction
54
Memory-mapped Files

• A files contents are mapped directly into the
  virtual address space
• Files can be read from or written to by
  referencing the corresponding virtual addresses
• Memory-mapped files are very useful when a file
  is shared or accessed repeatedly

55
Memory-mapped Files cont.
56
Directories

• A directory is a set of logically associated
  files and other directories of files
• Directories are the mechanism we use to organize
  files
• The file manager provides a set of commands to
  manage directories
• Traverse a directory
• Enumerate a list of all files and nested
  directories

57
Directory Structures

• How should files be organized within directory?
• Flat name space
• All files appear in a single directory
• Hierarchical name space
• Directory contains files and subdirectories
• Each file/directory appears as an entry in
  exactly one other directory -- a tree
• Popular variant All directories form a tree,
  but a file can have multiple parents.

58
Directory Structures
59
Directory Structures cont.
60
A directory tree
61
Directory Implementation

• Device Directory
• A device can contain a collection of files
• Easier to manage if there is a root for every
  file on the device -- the device root directory
• File Directory
• Typical implementations have directories
  implemented as a file with a special format
• Entries in a file directory are handles for other
  files (which can be files or subdirectories)

62
Directory Implementation

• Linear list of file names with pointer to the
  data blocks.
• simple to program
• time-consuming to execute
• Hash Table linear list with hash data
  structure.
• decreases directory search time
• collisions situations where two file names hash
  to the same location
• fixed size

63
Mounting file systems

• Each file system has a root directory
• We can combine file systems by mounting
• that is, link a directory in one file system to
  the root directory of another file system
• This allows us to build a single tree out of
  several file systems
• This can also be done across a network, mounting
  file systems on other machines

64
UNIX mount Command
65
Mounting a file system
66
VFS-based File Manager
67
NFS Architecture
68
Summary of File Storage Methods

• Contiguous files
• Interleaved files
• File pointers in the file descriptor
• Contiguous file pointers
• Chained data blocks
• Chained single index blocks
• Double index blocks
• Triple index blocks
• Hybrid solutions