File System

1
File System
File System (2) and Disk Storage Management
Recommended Reading Bacon, J. Concurrent
Systems (5) Nehmer, J. Systemsoftware Grlg.
mod. BS, (9) Silberschatz, A. Operating System
Concepts (10,11) Stallings, W. Operating
Systems (12) Tanenbaum, A. Modern Operating
Systems (4) Wettstein, H. Systemarchitektur
(18)
2
File System
Storage Media Directories (Folders) Pathnames
File System Storage Management Bu
ffer Management Examples
3
File System
Storage Media

• Magnetical Media
• Disk
• Floppy
• Streamer
• ...
• Optical Media
• CD-ROM
• CD-R or CD-RW
• DVD
• ...
• Magneto-optical Media

4
File System
File Directory

• Contains some information about files, e.g.
• symbolic name or FID (necessary)
• attributes
• location (necessary)
• ownership
• Directory itself is a structured file owned by
  the OS
• with some interface operations offered to
  users
• Provides a mapping between file names
• and the file objects

5
File System
Directory Operations

• Search
• Create file
• Ceate a new file name for an existing file
• Delete file (name)
• List all files of a directory

6
File System
Simple Structure for a Directory

• Array or list of entries, one entry for each
  file
• Sequential file with the name of the file
  serving as the key
• Provides no help in organizing the files
• Forces user to be careful not to
• use the same name for two different files

Conclusion A flat ordering concept is useful, as
long as only few objects have to be ordered.
7
File System
Two-level Scheme for a Directory

• One directory for each user and a master
  directory
• Master directory contains entry for each user
• provides address and access control information
• Each user directory is a simple list of files
  for that user
• still a name problem within the files of a user
• a user with many files may loose control
• Still provides no help in structuring
  collections of files

8
File System
Tree-Structured Directory

• Master directory (root) with user directories
• Each (user) directory may have subdirectories
• and files as entries
• To ease the use of directories
• directory entries are kept small
• Unix directories may grow, but never shrink,
  i.e.
• allocated disk blocks remain allocated until
  rmdir

Discuss now whether you would use fixed sized or
variable sized records for directories
9
File System
Tree-Structured Directory
Master Directory
System
Absolute path name /User B/Word/Unit A/ABC
10
File System
Tree-Structured Directory

• Files can be located by following a path from
  the root,
• or master directory down various branches
• this is the pathname of a file
• Can have several files with the same file name
• as long as they have different unique path
  names
• Current directory is the working directory, i.e.
  the user
• is updating her/his project files or reading
  documents
• Files may be referenced relatively to the
  working directory
• Instead of long pathnames some systems offer
  shortcuts

11
File System
Path Names

• Enabling users to work with symbolic names
  (supports clever users.)
• However, if users may choose their file names,
• name conflicts will arise for certain.
• Two way of pathname can be distinguished
• absolute path names and
• relative pathnames

Pathnames with hierarchical structured
directories reflect this ordering
scheme /etc/passwd is an absolute pathname in
an old Unix system, starting with the so called
root directory / containing subdirectory etc,
where the password file passwd can be found.
12
File System
Access Control

• Way to control access to a particular file
• Users or groups of users are granted
• certain access rights to a file

Example (The only access rights may be read r
or write w)
13
File System
More on Access Rights

• Execution
• the user can load and execute a program but
  cannot copy it
• Reading
• the user can read the file for any purpose,
• including copying and execution
• Appending
• the user can add data to the file but cannot
• modify or delete any of the files contents

14
File System
Access Rights

• Updating
• the user can modify, delete, and add to the
  files data.
• This includes creating the file, rewriting it,
  and removing
• all or part of the data
• Changing protection
• user can change access rights granted to other
  users
• Deletion
• user can delete the file

15
File System
Access Rights

• Owners
• (in most cases) has all rights previously listed
• may grant additionally rights to other users
  using
• the following classification
• specific user
• user groups
• all for public files

16
File System
Simultaneous Access

• User may lock entire file when it is to be
  updated
• User may lock the individual records during the
  update
• Mutual exclusion and deadlock are issues
• for shared access
• (especially for files with more than one
  entrance point,
• e.g. a B-tree structured index sequential
  file)

17
File System
Record Blocking Methods
Fixed blocking (widely used)
Track 2
R5
R6
R7
R8
Size of physical block
Waste due to block size constraint from fixed
record size
Analysis 1. Simplifies I/O and buffer allocation
in main memory 2. Simplifies memory management on
secondary storage (e.g. disk)
18
File System
Record Blocking Method
Variable Blocking Spanned
Track 1
R1
R2
R3
R4
R4
R5
R6
Track 2
R6
R7
R8
R9
R9
R10
R11
R12
R13
Analysis 1. No wasted space on disk (except at
the end of the file) 2. Additional linking
between parts of a record is necessary
random access may require up to 2 disk-I/Os 3. No
limit on the size of a record
19
File System
Record Blocking Method
Variable Blocking Unspanned
Analysis 1. Wasted space in most physical blocks
(unless next record fits exactly into the
reminder of the block) 2. Limits size of a record
size to the size of a physical block
20
File System
Secondary Storage Management

• Space must be allocated to files (done in
  portions)
• A portion may vary between a block or the entire
  file
• What kind of data structure is used to keep
• track of the portions assigned to a file
• (can be FAT file allocation table)
• How to organize management of free portions

21
File System
Preallocation

• Need to know the maximum size of file at
  creation time
• (in some cases this estimation can be done
  quite accurately,
• e.g. file copy etc.)
• In other cases its difficult to reliably
  estimate
• the maximum potential size of a file
• Tend to overestimate the file size
• (so as not to run out of space)

Conclusion In general there are advantages to
use dynamic allocation, i.e. the portions
will be allocated when needed.
22
File System
Portion Size

• Contiguity of space may increase the performance
• (especially for the file-type sequential)
• Large number of small portions increases
• the size of tables needed
• Fixed-size simplifies the reallocation of space
• Variable-size minimizes waste of unused storage

23
File System
Allocation of Files

• Three ways of allocating sequential files
• contiguous
• chained
• indexed
• fixed block portions
• variable block portions

Remark Dont mix up a sequential file (e.g. a
Unix file) with the file-type
index-sequential!
24
File System
Methods of File Allocation

• Contiguous allocation
• a single set of blocks allocated to a file (at
  time of creation)
• only a single entry in the file allocation table
• starting block and length of the file
• fragmentation of the storage system will occur
• hard to find contiguous blocks of sufficient
  length

25
File System
Characteristics of three File Allocation Methods
26
File System
Contiguous File Allocation
File Allocation Table
File Name
Start Block
Length
FileA
FileA
2
3
0
1
2
3
4
FileB
9
5
FileC
18
8
5
6
7
8
9
FileD
27
2
FileB
FileE
30
3
10
11
12
13
14
15
16
17
18
19
FileC
Remark To overcome external fragmentation
periodical compactions
20
21
22
23
24
FileD
25
26
27
28
29
FileE
30
31
32
33
34
27
File System
Contiguous File Allocation (After Compaction)
File Allocation Table
File Name
Start Block
Length
FileA
0
3
FileB
3
5
FileC
8
8
FileD
16
2
FileE
18
3
28
File System
Method of File Allocation

• Chained Allocation
• allocation on basis of individual block
• each block contains a pointer to the next block
  in the chain
• only one single entry in the file allocation
  table
• starting block and length of file
• No fragmentation
• Any free block can be added to the chain
• No accommodation of the principle of locality

29
File System
Chained File Allocation
File Allocation Table
File Name
Start Block
Length
...
...
...
FileB
5
1
...
...
...
Remark If you only access sequentially this
implementation is quite suited. However
requesting an individual record requires tracing
through the chained block, i.e. far too many disk
accesses in general.
30
File System
Method of File Allocation

• Indexed allocation
• file allocation table contains a separate
  one-level index for each file
• the index has one entry for each portion
  allocated to the file
• the file allocation table contains block number
  for the index

31
File System
Indexed Allocation with Block Portions
32
File System
Indexed Allocation Variable Length Portions
33
File System
Free Space Management

• Three common methods
• bit tables
• chained free portions
• indexing

Hint Find out pros and cons of each of the above
methods
34
File System
Bit Map or Bit Table
Bit map per physical disk or per partitions (e.g.
a cylinder group etc.) To accelerate access onto
the bit map you can buffer it entirely or
piecewise into main memory
35
File System
However, its overhead memory never be used for
applications
Bit Map or Bit Table
Stallings A bit map is small as possible and
can be kept in main memory. Its easy to find out
one free block respectively a contiguous group of
free blocks.
Example 4 disks à 16 GB each, a block size of
512 byte, and main memory of
128 MB (not a tiny one). Absolute bit map
amount/disk 16 GB 8 512 16 GB 4 K 4
MB Relative bit map amount/disk 4 16 000
0.25 Absolute amount for 4 completely loaded
bit maps 16 MB Relative amount 16 128
12.5 which is far too much.
Remark We may reduce the space amount if we do
not keep the entire bit map in main memory. In
any case a bit map is some special memory
portions, never available for client use. Are
there alternatives? Remember one of these
orthogonal design parameters!
36
File System
List of Free Portions
Free portions may be chained together, you need
only one pointer in main memory as an entry
point.
Can be applied to all file allocation methods.
If allocation is by variable-length portion a
first-fit method is appropriate.
Compare amount and efficiency of this list
chained portions with bit maps! Do we really
exploit the concept of integrated memory
management data structures?
37
File System
Indexing
Free portions are pointed to via a table similar
to a FAT, thus one pointer to any contiguous
group of free blocks, only (not to every block).
Provides efficient support for all of the file
allocation methods!
38
File System
Blocknumber
Blocknumber logical address of a block
Disk D
0
b bits for a blocknumber
Capacity(D) ?
Reasons for disk partitioning?
39
File System
Reliability
Example Suppose we have file allocation on
demand, and we have disk and file allocation
table, and the following scenario
1. User A requests an additional portion for its
file f1 2. Request is granted, disk and file
allocation tables (in main memory) are
updated. 3. User A stores a pointer to the new
portion within file f1. 4. System crashes and
reboots, restoring the old disk and file
allocation tables from the disk. 5. User B
requests an additional portion, which is granted,
however it overlaps with the above request
of user A. 6. User A accesses the overlapped
portion, too.
Such reliability problems may occur whenever the
images of control structures in main memory and
in secondary storage differ. To prevent this type
of error, you can use a protocol enabling equal
images whenever changes in the control
structures have to be made. However, this may
result in poorer performance.
40
File System
Buffer Management

• Because data often are accessed more than once
  (e.g. index blocks etc.)
• its useful to buffer frequently used data
  blocks of the file system in main meory
• Some OSes use the entire unused main memory as a
  disk cache
• (disk controllers may have an internal,
  transparent cache of their own)
• Before accessing any file data the buffer
  management looks
• if the desired block is already or still in one
  of the file buffers
• In case of buffer shortage a buffer replacement
  method (LRU, or LFU) is used
• If you delay the updating of a modified buffer
  until this buffer has to be
• replaced. You may loose its content in case of
  a system crash
• Important blocks for file system consistency
  like directory blocks or index
• blocks should be updated at once or at least
  more frequently.

41
sync
42
File System
Examples of existing File Systems

• Linux
• Unix
• MS-DOS
• NT

Hint Short description of Unix and MS-DOS in A.
Tanenbaum Modern OSes, pp. 315 Short
description of Unix and Mach in Silberschatz OS
Concepts, pp. 607 Short description of
Unix and NT in Stallings, pp. 529 For a more
detailed description see special books on these
systems, e.g. D.A. Solomon Inside Windows NT,
Microsoft Press, 1998 Beck et al Linux Kernel
Programming, U. Vahalia Unix Internals, pp. 220
43
File System
Unix File Management

• Files array of bytes, no record structures at
  system level
• Types of files
• ordinary contents entered by user or program
• directory contains a list of file names and
• pointers to inodes (index nodes)
• special used to access peripheral devices
• named for named pipes
• Inodes are file descriptors containing file
  attributes
• file mode
• link count
• owner and group id etc.

44
File System
Access Structure
Remark Depending on the block size (e.g. 512
Bytes, ...) and on the pointer length (e.g. 4
Bytes) at least the maximal file size is
greater than 2 MB. Small files are favored in
any case.
45
File System
Unix Directories

• A directory entry contains three elements
• length of directory entry (due to variable
  length file names)
• file name (up to 255 characters)
• inode number (index to a contiguous table of
  inodes)
• Each directory contains at least two entries
• .. is the link to the parent directory (forming
  the directory tree)
• . is a link to itself
• Thus a Unix file system offers a tree-like
  structure,
• supporting the human preference to arrange and
  order things
• in some hierarchical manner.

46
File System
Unix Directories
Multiple directory entries may point to the same
inode (hard link) Pathnames are used to identify
files /etc/passwd is an absolute pathname
../home/lief/exam is a relative pathname Path
names are resolved from left to right As long as
its not the last component of the pathname the
component name must be a directory With
symbolic links you can address files and
directories with different names, you can even
define a symbolic link to a file, currently not
mounted
47
File System
Symbolic Links
sys
testfile.h
With symlink(/usr/src/uts/sys,
/usr/indclude/sys/) you add a file link to sys
With symlink(/usr/include/realfile,
/usr/src/uts/sys/testfile) you add a file
link to realfile.h
The following 3 pathnames access same
file /usr/include/realfile /usr/include/sys/
testfile.h /usr/src/uts/sys/testfile.h You
can benefit by hard and soft links using relative
path names.
48
File System
Hard Links versus Symbolic Links
A hard link is just another file name, i.e.
there is another directory entry pointing to a
specific file. The inode-field is the same in all
hard links (to the same file)! Any new hard link
increases the link counter in the inode of the
file. As long as link counter ? 0, file remains
existing after a remove(). A remove only
decreases the link counter. A symbolic link is a
new file containing a pathname pointing to a
file or to a directory. Symbolic links are
evaluated per access. If this file or directory
is removed the symbolic link points to
nirwana. You may even specify a symbolic link to
a file or to a directory currently not present or
even currently not existent.
49
File System
Logical and Physical File System
50
File System
Logical and Physical File System

• A logical file system may consist of different
  physical file systems
• A file system can be mounted at any place within
  another file system
• Accessing the local root of a mounted file
  system, a bit in its inode
• identifies this directory as a so called mount
  point
• Via mount respectively umount the OS manages a
  so called mount table
• supporting the resolution of path names
  crossing file systems
• The only resident file system is the root file
  system
• (in general on a partition of a hard disk)

51
File System
Layout of a Logical Disk

• Any physical file system is placed within a
  logical disk partition, a physical disk
• may contain several logical partitions (or
  logical disks)
• Any partition contains a boot block, a super
  block, the inode-table and
• the data blocks

• Only the root partition contains a boot block
  
  

• Border between inodes and data blocks region can
  be set thus supporting
• file systems with either few and large files
  or with many and small files

52
File System
Buffering

• Disk blocks are buffered in main memory. Access
  to buffers is done via a hash table.
• Blocks with the same hash value are chained
  together
• Buffer replacement is done according to LRU
• Free buffer management is done via a double
  linked list.

head of free list
hash table
53
File System
Using a Unix File

• Opening a file creates a file descriptor fid

• Used as an index into a process specific table
  of opened files

• The corresponding table entry points to a system
  wide file table

• Via the buffered inode table you finally get the
  appropriate data blocks

fid open(...)
read(fid,)
opened files per process
file table (system wide)
inode table (in a buffer)
system address space
user address space
54
File System
MS-Dos File System
MS-Dos Version 1.0 supported only device A
representing a 5 1/4 diskette (160 KB
only),thats why Tim Paterson implemented a flat
directory, containing all files of the
system.Next version 1.x incorporated larger
diskettes.
For IBM PC/XT in version 2.0, a hard disk was
supported and MS-DOS was restructured including
many Unix features like a tree structure file
system on each drive A, B , C,
For IBM PC/AT (based on Intels 286) in version
3.0, hard disks larger than 10 MB are supported,
some space in memory was used as a file system
buffer.
Since version 4.0, up to 16 MB of RAM could be
used as file buffers. Version 5.0
55
File System
MS-Dos File System (Version 5.0) versus Unix
56
File System
MS-Dos File System (Version 5.0)
Hard disk layout
primary boot block
partition 1
partition 2
partition 3
partition 4
secondary boot block
root directory
FAT
Repl. FAT
...
(optional)
data blocks
57
File System
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 . . .
Conceptual Layout of FAT
EOF
13
2
data blocks of files A, B, and C
9
8
6
8
4
2
file A
free
4
12
5
9
12
3
file B
free
EOF
EOF
10
3
13
file C
free
bad
free
A FAT entry is 2 bytes long and contains a
pointer to next data block within a file. (Some
special codes are used for free and bad blocks,
and for EOF.) (The first 2 entries specify the
partition size and disk type etc.)
Problem 2 byte FAT-entries limit the partition
size. How can we support larger partitions?
58
File System
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 . . .
Larger Partitions with FAT
EOF
13
2
9
Use larger data blocks ? larger internal
fragmentation. Another drawback of the
FAT Analyze MS-DOS free storage management!
8
free
4
12
3
free
EOF
EOF
free
bad
free
59
File System
MS-Dos Directory Entry
A directory entry in MS-DOS contains a lot of
file attributes (other than in Unix), e.g. A
archive bit D directory bit V volume label
entry S system bit (i.e. undeletable files) H
hidden bit R read only bit
60
File System
MS-Dos Directory Entry and FAT
61
File System
Windows NT File System Overview

• NT supports several file systems, including the
  FAT file system of MS-DOS
• and Windows 98, and the HPFS (high performance
  file system) of OS/2
• Shortcomings of both FS
• Not very reliable
• Unable to handle large system configuration
• New NTFS to meet the requirements of
  workstations and servers, e.g.
• client/server applications such as
• file servers
• compute servers
• database servers
• resource-intensive engineering and scientific
  applications
• network applications for large corporate systems

62
File System
Key Features of NTFS

• Recoverability
• Security
• Data Redundancy and Fault Tolerance
• Large disks and large files
• Multiple data streams
• General indexing facility

63
File System
Recoverability

• To address requirements of reliable data storage
  and and data access,
• NTFS provides file system recovery based on
  transaction-processing
• An access operation modifying a file or the file
  system (e.g. extend a file,
• change the directory structure, allocate space
  for a new file etc.)
• is processed as a transaction (with the all or
  nothing property).
• A transaction being interrupted by a system
  crash must be undone.
• Furthermore, NTFS uses redundant disk storage
  for vital file system
• information (e.g. part of the master file
  table)

64
File System
Security

• Hospitals, banks etc. may have sensitive data
  that should not
• be accessible by unauthorized users.
• A user accessing an object (e.g. like a file)
  must have the permission
• to do so. An administrator may classify for
  each user its access potential.

65
File System
Data Redundancy and Fault Tolerance

• In addition to recoverability, some customers
  require that their data may
• not be endangered by a power outage or by a
  catastrophic system failure.
• Recoverability ensures that the entire file
  system is recoverable
• however NTFS cannot guarantee consistent user
  file contents.
• Data redundancy enables users of very critical
  files
• an extra level of integrity.
• NT layered I/O-driver model furthermore is used
  to provide fault tolerant
• disk support. (Another fault tolerant disk
  driver can mirror or duplicate data
• from one to another disk so that redundant
  copies can be retrieved.

66
File System
NTFS Volume and File Structure

• Sector smallest unit of storage on a disk,
• Cluster one or more contiguous sectors (on the
  same track)
• fundamental unit of allocation within NTFS
• Volume logical partition on a disk
• (consisting of 1 or more clusters)
• Complete disk or just a partition of a disk
• Can extend across multiple disks, too
• Raid 5 is supported
• Maximal Volume size 264 bytes
• Maximal file size 248 bytes (NT 5.0 264 bytes)

67
File System
NTFS Partitions and Cluster Sizes
68
File System
NT Volume Layout
The master file table is organized as a table of
variable-length rows. It corresponds to Unix
inode table containing all headers of files in
the volume.
Each row describes a file or folder including the
master file table being treated as a file. If the
appropriate file is small enough, its located in
a row of the master file table.
Otherwise, the row contains the file header
pointers to those clusters. Each row contains
some additional attributes describing the use of
that file (see below).
69
File System
NT Volume Layout

• The system files (a region of about 1 MB)
  contains the following files
• MFT2 a mirror of the first 3 rows of the MFT
  (enhancing
• Log file A list of transaction steps used for
  NTFS recoverability
• Cluster bit map a representation of the volume
  (which clusters are in use)
• Attribute definition table defines the
  attributes supported on the volume and
• indicates whether they can be indexed and
  whether they can be recovered
• Bad block file etc.

70
File System
NT Volume Layout
The file area contains all the clusters being
used for storing larger files or folders.
71
File System
File Attributes in NTFS
Remark Shaded attributes are necessary ones.
72
File System
Architecture of the NT I/O System
Environment Subsystem or DLL
user mode
kernel mode
NT System Services
Object Manager
Security Reference Monitor
Process Manager
Local Procedure Call Facility
Virtual Memory Manager
I/O Manager
Kernel
73
File System
NTFS and related Components
I/O Manager
Log the transaction
Read/write a mirrored or striped volume
NTFS Driver
Log File Service
Read/write the file
Fault Tolerant Driver
Flush the log file
Write the cache
Read/write the disk
Disk Driver
Load data from disk into memory
Cache Manager
Access the mapped file or flush the cache
Virtual Memory Manager
74
File System
NTFS and related Components
Log file service maintains a log of all disk
writes. This log file is used for
recovery. Cache manager provides system wide
caching for NTFS and other FS. All file systems
access cached files by mapping them into virtual
memory and then accessing the virtual memory.
When a thread tries to access a non cached file
(a file cache miss), the memory manager calls
NTFS to access the disk driver and obtain the
file contents from disk. The cache manager
supports lazy writing (i.e. delayed writing to
disk).
75
File System
NTFS Data Structures
Process
file object
file object
master file table
handle table
data attribute
NTFS data structures (use to manage the on-disk
structure)
user defined attribute
stream control blocks
file control block