Analysis and Evolution of Journaling File Systems

1
Analysis and Evolution of Journaling File Systems

• Paper By Vijayan Prabhakaran, Andrea C.
  Arpaci-Dusseau, and Remzi H. Arpaci-Dusseau

Presented By Sridhar Balijepalli and Brent
Everest
2
Presentation Outline

• Introduction
• Background
• SBA
• Synthetic Workloads
• Overhead of SBA
• Alternative Approaches
• STP
• Other Standard Approaches
• Testing Environment
• Ext3 File System
• Background
• Journaling Modes
• Behavioral Analysis
• Concurrency Analysis
• Journal Commit Policy
• Checkpoint Policy
• Evolving ext3 with STP

• ReiserFS
• IBM Journaled File System
• Windows NTFS
• Related Work
• Conclusions

3
Introduction

• The paper is written by the Computer Sciences
  Department at the University of Wisconsin,
  Madison. The university ranks consistently among
  the top 10 computer science departments in the
  U.S.
• The paper proposes to develop and apply two new
  methods for analyzing file system behavior and
  evaluation file system changes. (Semantic
  block-level analysis SBA and semantic trace
  playback STP).
• Through the analysis, design flaws, performance
  problems, and even correctness bugs are
  uncovered, and proposed changes are stated.

4
Background

• Most modern file systems implement journaling in
  one form or another.
• They investigated into the ext3, ReiserFS, JFS,
  and NTFS file systems. Detailed analysis is
  applied into Linux ext3 and ReiserFS and cursory
  analysis is given to Linux JFS and Windows NTFS.
• Journaling writes information about pending
  updates to a write-ahead log (Journal) before
  committing the updates to disk (fixed-location
  data structures).
• Journalings most easily seen advantage is for
  fast file system recovery after a crash.

5
Semantic Block-Level Analysis (SBA)

• Semantic block-level analysis is used here to
  introduce controlled workload patterns from above
  the file system operation, and analyze not only
  the time taken for completion of the assigned
  workloads, but to also see the resulting read and
  write requests within the file system calls.
• SBA does block-level tracing of disk traffic,
  which allows one to record the quantity of disk
  traffic, the block number of each block accessed,
  and the timing of each block. Each block is also
  able to be differentiated between read and write
  access requests.

6
SBA Continued

• To perform SBA, a pseudo-device driver is placed
  in the kernel, associated with an underlying
  disk, and mounts a file system of interest on the
  pseudo device.
• As disk traffic is passed through the driver, it
  records each request and response. This allows
  the SBA to distinguish between traffic sent to
  the journal and to the fixed-location data
  structures.
• SBA is also able to classify the different types
  of journal blocks (descriptor, journal data, and
  commit blocks.)

7
Synthetic Workloads

• Synthetic workloads are created to stress the
  file system for testing purposes. The workloads
  created in this paper vary these 4 parameters
• The amount of data written.
• Sequential versus random accesses
• The interval between calls to fsync
• The amount of concurrency.

8
Overhead of SBA

• Table 1 (below) shows the number of C statements
  required to implement the SBA driver.
• Processing and memory overheads of SBA are
  minimal for the created workloads, as they did
  not generate high I/O rates for these cases.

9
Alternative Approaches

• Directly instrumenting a file system to obtain
  timing information and disk traces.
• SBA is believed to be superior for 3 reasons
• SBA does not require file system source, plus
  much of the SBA driver can be reused across file
  systems and versions, increasing versatility.
• One may accidentally miss some of the conditions
  for which disk blocks are written, where the SBA
  driver is guaranteed to see all disk traffic, due
  to its placement in implementation.
• Instrumenting existing code may accidentally
  change the behavior of that code.

10
Semantic Trace Playback (STP)

• Semantic trace playback is a technique which is
  used here to rapidly suggest and evaluate file
  system modifications without a large
  implementation or simulation effort of
  rewriting/directly modifying the file system
  code.
• STP takes as input a trace, parses it, and issues
  I/O requests to the disk using the raw disk
  interface.

11
STP Continued

• STP needs to observe 2 high-level activities
• File-system level operations that create dirty
  buffers in memory. This is to be able to
  correlate why a block is flushed to disk, whether
  it be from reaching a threshold, or an interval
  timer expiring.
• Application-level calls to fsync to understand
  whether an I/O operation is due to an fsync call,
  or normal file system behavior.
• STP is limited to evaluating minor file system
  changes (nothing drastically radical).
• Does not provide the means for how to implement a
  given change, but instead helps to understand
  whether the modification helps to improve
  performance.

12
Other Standard Approaches

• Simply implement a new idea within the file
  system and measure the performance of the real
  system.
• Problem Time consuming
• Build an accurate simulation of the file system,
  and evaluate the new idea within the simulation.
• Problem Requires construction and maintenance of
  a detailed and accurate simulator.

13
Testing Environment

• All measurements are taken on a machine runinng
  Linux 2.4.18 with a 600 MHz Pentium III processor
  and 1 GB of main memory.
• The file system under test is created on an
  external disk, separate from the root disk.
• Where appropriate, each data point reports the
  average of 30 trials (all cases report low
  variance).

14
Ext3 File System - Background

• In ext3, data and metadata are eventually placed
  into fixed-location structures, where the disk is
  split into a number of block groups. (Depicted in
  Figure 1)
• Information concerning pending file system
  updates is written to the journal.

15
Journaling Modes

• There are 3 different journaling modes in ext3
• Writeback Mode
• Ordered Mode
• Data Journaling Mode
• Figure 2 shows the difference between the 3
  modes.
• For updates, ext3 groups many updates into a
  single compound transaction that is periodically
  committed to disk, instead of each individual
  update running separately.

16
Journaling Continued

• In ext3, all journaling modes log full blocks
  instead of only differences from old versions,
  meaning single bit changes result in the entire
  block being logged.
• Writing journaled metadata and data to their
  fixed-locations is called checkpointing. This
  occurs when any of a number of thresholds are
  crossed, such as
• File system buffer space is low
• Little free space left in the journal
• A timer expires

17
Basic Behavior of ext3

• The basic behavior of ext3 was tested by varying
• The amount of data written
• The sequentiality of the writes.
• The synchronization interval between writes
• The number of concurrent writers.
• The first workload writes to a single file
  sequentially and then performs an fsync to flush
  its data to disk. (Shown in Figure 3)
• The second workload issues 4 KB writes to random
  locations in a single file and calls fsync once
  every 256 writes. (Figure 4)
• The third workload issues 4 KB random writes
  also, but calls fsync for every write. (Figure 5)
• For each workload, the total amount of data it
  writes is increased, and the behavioral changes
  are observed.

18
Behavioral Responses
Sequential
Random w/fsync every 256
Random w/fsync every write
19
Concurrency

• The effect of concurrency upon journaling in the
  ext3 file system was tested by running workloads
  containing two diverse classes of traffic
• An asynchronous foreground process writing out a
  50 MB file without calling fsync.
• In competition with a background process
  periodically (frequency is the sync interval)
  writing a 4KB block to a random location,
  optionally calling fsync.

20
Concurrency Results

• As expected, when the foreground process runs
  with an asynchronous background process, the
  bandwidth is in competition with that of
  in-memory speed.
• But when the foreground process is in competition
  with the background process, the bandwidth drops
  to disk speed.
• As seen, the more frequently the background
  process calls fsync, the more traffic is sent to
  the journal.
• This shows a potential hazard of grouping
  unrelated updates into the same transaction, thus
  causing all updates to the disk to be committed
  at the same rate. (called tangled synchrony)

21
Journal Commit Policy

• Two factors are tested to influence the
  conditions for which ext3 commits transactions to
  its on-disk journal
• The size of the journal
• The settings of the commit timers

22
Impact of Journal Size

• As seen in the top graph, when the amount of data
  written reaches approximately ¼ the size of the
  journal, the bandwidth drops considerably.
• In the bottom graph, as can be extrapolated from
  the top, the metadata and data are forced to the
  journal when it is approximately ¼ of the journal
  size.

23
Timer Impact

• For Linux 2.4 ext3, three timers have some
  control over when data is written
• Metadata commit timer (5 sec default)
• Data commit timer (30 sec default)
• Commit Timer (5 sec default)
• The kupdate daemon is responsible for flushing
  dirty buffers to disk.
• The kjournal daemon is specialized for ext3 and
  is responsible for committing ext3 transactions.

?Both managed by ? the kupdate daemon
Managed by the kjournal daemon
24
Timer Transaction Impact

• Figure 8 shows the results of how the timers
  affect when transactions are committed to the
  journal.
• To make sure the timers influence the journal
  commits and not the journal size, the size was
  set to be sufficiently large and all other timers
  set to large values (60 sec)

25
Journal vs Fixed-Location Trafficin ext3

• The difference between writeback and ordered mode
  is
• Writeback mode does not enforce any ordering
  between writes to the journal and to the
  fixed-location data.
• Ordered mode ensures that the data is written to
  its fixed location before the commit block for
  that transaction is written to the journal.

26
Journal vs Fixed-Location Trafficin ext3
Continued

• As seen in Figure 9, wrtes to the journal and to
  fixed-location data do not overlap.
• For ext3, data writes to the fixed location are
  issued and waits for completion before issuing
  the journal writes, and again waits for
  completion before finally issuing the commit
  block and waiting for the last completion.
• This proves that ext3 has falsely limited
  parallelism.

27
Impact of Journal Size on Checkpointing

• It is shown that checkpointing in ext3 is a
  function of the journal size and the commit
  timers, as well as the synchronization interval
  in the workload. They test for a journal size of
  40 MB.

28
Impact of Timers on Checkpointing

• For this experiment, the kupdate data timer was
  varied while setting the other timers to 5
  seconds.
• Figure 11 will show how the kupdate data timer
  impacts when data is written to its fixed
  location.
• Ordered and data journaling modes force data to
  disk either before or at the time of metadata
  writes, therefore both data and metadata are
  flushed frequently to disk.
• Advantage By forcing data to disk in more timely
  manner, large disk queues can be avoided and
  overall performance improved.
• Disadvantage Temporary files may be written to
  disk before subsequent deletion, increasing the
  overall load on the I/O system.

29
Evolving ext3 with STP

• From the SBA analysis completed on the ext3 file
  system, 3 improvements were pointed out which
  were quantified with STP
• Changing the placement of the journal
• The value of using different journaling modes
  depending upon the workload
• Having separate transactions for each update
• Overlapping pre-commit journal writes with data
  updates in ordered mode.
• Using differential journaling, in which block
  differences are written to the journal instead of
  full blocks.

30
Changing Journal Location

• By default, ext3 creates the journal as a regular
  file at the beginning of the partition.
• Figure 12 shows the results of evaluation with
  the journal in the middle of the disk.
• By placing the journal in the middle of the disk,
  the longest seeks (end to end of the disk) are
  avoided during synchronous workloads.

31
Modifying Journaling Mode

• Obviously, different workloads will perform
  better under differing journaling modes.
• A new adaptive journaling mode was evaluated
  with STP that chooses the journaling mode for
  each transaction according to writes that are in
  the transaction. (sequential transactions use
  ordered journaling else use data journaling).
• Through testing, it was proven that adaptive
  journaling out performs any single-mode approach.

32
Altering Transaction Grouping

• ext3 groups all updates into system-wide compound
  transactions and commits them to disk
  periodically.
• Using STP, it was tested to only force the
  process that issues the fsync to commit its data
  to disk, and the results are shown in Figure 13.
• As seen, for smaller sync intervals, untangled
  transaction grouping is far superior, and for
  larger sync intervals it still outperforms the
  standard, but not as drastically.

33
Overlapping Journal Writes

• STP was used to modify the timing so that journal
  and fixed-location writes were all initiated
  simultaneously, and the commit transaction is
  written only after the previous writes complete
  still.
• As seen in Figure 14, STP predicts an improvement
  of about 18. (Also proven when changed directly)
• Thus, as would be expected, increasing
  concurrency also increases performance.

34
Differential Journaling

• Ext3 uses physical logging and writes new blocks
  in their entirety to the log.
• STP was used to test writing only the block
  differences to the journal instead of new blocks
  in their entirety.
• With data journaling mode, the amount of data
  written to the journal was reduced by a
  significant factor, where as for ordered and
  writeback modes the differences were negligible.
• The minimal differences were expected, because in
  ordered and writeback modes, only metadata is
  written to the log which, when differential
  journaling is applied, makes little difference in
  total I/O volume.

35
ReiserFS- Background

• Differs from Ext3 FS in three primary ways
• First, The two file system use different on-disk
  structures to track their fixed location data.
• Uses a B tree approach where in the data is
  stored on the leaves and metadata on the internal
  nodes
• The difference may be largely irrelevant as the
  paper doesnt deal with the impact on the
  fixed-location data structure.

36
ReiserFS Contd

• Second, It is different in the format of the
  journal in the way that in ext3 the journal may
  be a file which may be anywhere in the partition
  and may not be contiguous.
• The ReiserFS on the other hand is not a file and
  is a contiguous sequence of blocks at the
  beginning of the file system
• ReiserFS limits the journal to maximum of 32 MB

37
ReiserFS- Contd

• Third, ReiserFS and ext3 differ slightly in their
  journal contents.
• The fixed locations for the block are stored in
  both descriptor block and in commit block as
  well.
• It uses only one descriptor block in every
  compound transaction.
• This limits the number of blocks that can be
  grouped in a transaction.

38
Semantic Analysis of ReiserFS

• A similar experiment was performed on ReiserFS as
  done on ext3.
• Basic Behavior Modes and Workload
• The experiments showed that the performance was
  similar to that of ext3
• Random workloads with infrequent synchronization
  performed the best with data journaling.
• Sequential workloads generally performed better.
• It also showed that writeback mod and ordered
  mode generally performed better than data
  journaling

39
Semantic Analysis of ReiserFS- Contd...

• Basic Behavior Modes and Workload.
• The main difference between ext3 and ReiserFS was
  in sequential workloads with data journaling.
• The throughput of data journaling mode in
  ReiserFS does not follow saw tooth pattern
• This is further explained in SBA analysis.

40
Semantic Analysis of ReiserFS- Contd...

• Basic Behavior Modes and Workload.
• In ReiserFS all of the data all of the data is
  written to the journal and is also checkpointed
  to its in place location.
• For this reason, ReiserFS appears to checkpoint
  the data much more aggressively than ext3.

41
Journal Commit Policy

• This section talks about the factors that impact
  when ReiserFS commits transactions to the log.
• We have already seen that ext3 commits data to
  the log when approximately 1/4th of the log is
  filler or when the timer expires.
• But, ReiserFS uses a different threshold.
• It depends on whether the journal size is below
  or above 8MB.
• It commits data when about 450 blocks i.e., about
  1.7 MB or 900 blocks i.e., about 3.6 MB is
  written.
• ReiserFS has a falsely limited parallelism in
  ordered mode.

42
Checkpoint Policy

• The conditions which triggers ReiserFS to
  checkpoint data to its fixed-place locations is
  complicated than ext3.
• In ext3 data is checkpointed when the journal is
  about 1/4th to ½ full.
• But in ReiserFS, the point at which data is
  checkpointed depends not only on the free space
  in the journal but also on the number of
  concurrent transcations.
• This is shown by considering the workloads that
  periodically force data to the journal by calling
  fsync at different intervals.

43
Checkpoint Policy - Contd

• The results show that the data is checkpointed
  before 7/8th of the journal is filled.
• The above graph shows the amount of data
  checkpointed as a function of amount of data
  written.

44
Checkpoint Policy - Contd

• It shows that the data is checkpointed at least
  at intervals of 128 transactions.
• A similar workload experimented on ext3 revealed
  no relationship between the number of
  transactions and checkpointing.
• The above graph shows the amount of data
  checkpointed as a function of the number of
  transactions

45
Checkpoint Policy - Contd

• The experiment was done considering workloads
  where data is sequentially written and an fsync
  is issued after a specified amount of data.
• SBA reports the amount of fixed location traffic
  . In the experiment the amount of data and the
  number of transitions are varied, defined as the
  number of calls to fsync.
• In ext3, timers control when data is written to
  the journal and to the fixed locations.
• Thus, ReiserFS checkpoints data whenever either
  journal free space drops below 4 MB or when there
  are 128 transactions in the journal.
• Kreiserfs daemon is responsible for committing
  transactions for ResiserFS where kjournal does
  the job for ext3.

46
Checkpoint Policy - Contd

• A graph is plotted with data written to the
  journal and to the fixed locations as the
  kreiserfs timer is increased.
• It can be concluded that
• Log writes always occur within the first five
  seconds of the data written by the application.
• Fixed-Location writes occur only when the elapsed
  time is both greater than 30 seconds and is a
  multiple of the kreiserfs timer value.
• The timer policy of ReiserFS is simpler than that
  of ext3

47
Bugs Involved with Reiser FS

• Some of the bugs involved were
• Due to incorrect initialization, the first
  transaction after a mount, the fsync call returns
  before any data is written.
• The file block was overwritten in writeback mode
  and its stat information was not updated. This
  was due to the failure to update the inodes
  transaction information.
• Dirty data was not always flushed when commiting
  the old transactions.

48
The IBM Journaled File System

• This section talks about performing the SBA
  analysis on the Journaled file system (JFS).
• Journal by default is located at the end of the
  partition and is treated as a contiguous sequence
  of blocks.
• A trial and error approach was used to analyze
  this file system and in some cases new techniques
  were used.
• The traffic was filtered out and the system was
  rebooted to infer whether the filtered traffic
  was necessary for consistency.

49
The IBM Journaled File System

• Some of the Properties of JFS are
• JFS uses ordered journaling mode.
• It was due to the small amount of traffic to the
  journal, it was not employing data journaling.
• Ordering of writes matched that of ordered mode
• JFS orders the write such that data block is
  written before the metadata writes are issued.
• Logging Is done at the record level.
• Instead of the entire block only that structure
  i.e., inode tree, index tree or directory tree,
  is logged.
• JFS writes fewer journal blocks than ext3 and
  ReiserFS.
• JFS does not by default group concurrent updates
  into a single compound transaction.
• No commit timers in JFS and the fixed location
  writes happen whenever kupdate daemon's time
  expires.
• Joural writes are indefinitely postponed until
  there is another trigger such as memory pressure
  or an unmount operation.
• This infinite write delay limits reliability.

50
Windows NTFS

• NTFS is a journaling file system used on Windows
  OS.
• The experiment was performed by running windows
  XP operating system on top of VMware of Linux
  Machine.
• Every object in NTFS is a file. Metadata too is
  stores in terms of files.
• The journal for that matter is itself a file
  located at the center of the file system.
• Usually ntfsprogs tool is used to discover
  journal file boundries.
• The experimented showed that NTFS does not do
  data journaling which was verified by the amount
  of disk traffic observed by the SBA driver.

51
Windows NTFS - Contd

• NTFS does not do block-level journaling as well
• It journals only metdata that too in terms of
  records.
• NTFS performs ordered journaling.
• NTFS waits until the data block writes to the
  fixed-location complete before writing the
  metadata blocks to the journal.

52
Related Work

• Journaling Studies
• SBA was used as to reason why journaling
  performance drops in a deleted benchmark.
• It compares a range of existing Linux file
  systems including ext2,ext3, ReiserFS, XFS and
  JFS.
• File System Benchmarks
• Chen and Pattersons self-scaling benchmark was
  used.
• Benchmarks that exercise certain file system
  behavior in a controlled manner was used.
• File System Tracing
• By recording network-level protocol activity,
  network file system is analyzed which is similar
  to the SBA since both are positioned at the low
  level and needed to be reconstructed at
  higher-level to obtain a complete view.

53
Conclusion

• Semantic Block Level Analysis (SBA) was used to
  provide insight about the internal behavior of
  the file system.
• Two journaling file system namely ext3 and
  ReiserFS were analyzed.
• Preliminary analysis of Linux JFS and Windows
  NTFS were performed.
• STP provided benefits of numerous modifications
  to the current ext3 implementation for real
  workloads and traces.