Measuring and Characterizing System Behavior Using Kernel-Level Logging

1
Measuring and Characterizing System Behavior
Using Kernel-Level Logging

• Karin Yaghmour and Michael Dagenais
• EECS 800

2
Discussion

• Abstract
• Introduction
• Toolkit architecture
• Toolkit overhead
• Toolkit usage and comparison
• Conclusions and future directions

3
Introduction

• Conventional tools (ps, etc) allow analysis of
  performance of single processes or overall system
  performance
• Different approach required to analyze complex
  networked multi-process software systems, running
  on multi-processor systems

4
Introduction

• Linux Trace Toolkit (LTT) records all significant
  system events, and makes it possible to analyze
  any desired subset of running processes.
• Experimental results show that LTT time and
  memory overhead is minimal.
• LTT is easily extended in terms of system events
  gathered, and of later post-processing and
  graphical presentation.

5
Related Work

• Two broad categories of profiling tools detailed
  analysis of individual applications, and overview
  of system behavior.
• Tools like DCPI, Morph do not look at
  interactions between different processes
• Path Profiler has a large overhead of around 30
• These tools do not provide information on system
  dynamics

6
Related Work

• Specialized tools to track key system events
  while preserving their order of occurrence and
  some details, but they have been primarily used
  for debugging or security auditing, and not
  designed for measurement or characterization.
• Another approach SimOS simulates hardware on
  which an operating system runs in order to
  retrieve information regarding its behavior and
  how applications interact.

7
Toolkit architecture

• LTT composed of independent software modules
• Primary source of information instrumented
  kernel
• Arrows indicate flow of information through the
  different modules.

8
Toolkit architecture

• Events are forwarded to the trace module via
  kernel trace facility
• The trace module then logs the vents in its
  buffer
• Finally, the trace daemon reads from the trace
  module device and commits the recorded events
  into a user-provided file.

9
Kernel trace facility

• Extension to the core kernel facilities to
  provide a unique entry point to all the other
  kernel facilities that would like an event to be
  traced.
• It does not log the events itself it forwards
  the trace request to the trace module.
• Trace module registers itself with the trace
  facility and provides it with a call-back
  function that is to be called whenever an event
  occurs.
• Acts as a link between the trace module and the
  different kernel facilities.

10
Kernel instrumentation

• Consists of different events being traced in the
  kernel
• Each type of event has its own data set with
  varying degrees of detailed information.
• Set of macros used instead of a direct call to
  the services of the trace facility.
• Table 2 in the paper shows the kernel events
  traced and the event details recorded.

11
Trace module

• Key element of the architecture.
• Goal store the incoming event descriptions and
  deliver them efficiently to the trace daemon.
• Implementation is much more elaborate.
• The trace module must retrieve additional
  information for each event time of event and
  CPU identifier (4b, rdtsc/Pentium)

12
Trace module

• The trace module must be configurable options
  shown in section 3.3. Configuration is done
  through the ioctl system call.
• The trace module must be reentrant since an event
  can occur from two different levels of priority
  at the same time.
• The logging procedure holds a kernel lock during
  logging to prevent suspension by interrupts but
  this is only for logging events into a buffer.

13
Trace module

• Trace module provides a basic set of file
  operations to be accessible through the virtual
  file system as a device (open, ioctl, release,
  fsync).
• To efficiently deal with large quantity of data
  generated, the trace module uses double-buffering.

14
Daemon

• Primary function to retrieve and store
  information accumulated by the trace module and
  store it in a file.
• Provides user with number of options to control
  the trace process.
• Upon starting, it opens and configures the trace
  module, and sets a timer if a time duration is
  specified.
• In normal operations, the daemon sleeps awaiting
  SIGIO to read from the trace module, or
  timer/kill events to end tracing.

15
Daemon

• Uses double buffering like the trace module, but
  for a different reason to reduce impact on the
  system due to frequent read and write, and to
  reduce the pollution of the trace daemon
  utilizing system resources.
• To record the state before the trace starts, the
  daemon will record from the /proc directory PID,
  name and PPID.

16
Data analysis

• Data analysis and presentation software typically
  run offline, and uses initial process state and
  trace data files to recreate system behavior in
  the observed time interval.
• Trace file is used as the trace event database
• Accomplished by using mmap and a collection of
  primitives to enable extraction of information
  regarding the events from the trace.

17
Data analysis

• These functions (primitives) provide detailed
  information about each event
• Also, they enable forward or backward browsing of
  the events trace.
• They can determine if an event is a control event
  or not an event that results in the transition
  of control from or to the kernel.
• Trace analysis procedure reads the entire trace
  cumulating results about the behavior of the
  system during the trace. Core of trace processing.

18
Toolkit overhead

• Most important feature extent of information
  available with low overhead.
• Various experiments performed that showed
  individual contributions of each of the modules
  to the overall overhead.
• Experiments were also performed with CPU
  intensive processes, disk intensive processes,
  processes that are both CPU and disk intensive,
  and a set of applications that represented an
  extreme case of desktop usage.

19
Toolkit overhead

• Results of the experiments are shown in figure 3
  in the paper.
• Instrumenting the core kernel events yields an
  impact below 2.5, regardless of the type of job
  run.
• Largest traces are generated in the graphical
  environment job around 50 of the trace is
  composed of detailed non-core events.
• Some aspects of LTT use significant system
  resources disk accesses and disk space mmap
  and trace compression offer optimization approach

20
Conclusions

• Novel way of recording and analyzing system
  behavior.
• Results show that LTTs overhead is minimal, and
  it provides unique data sets, which can be
  successfully used to reconstruct the dynamic
  behavior of systems.
• LTT provides precise characterization when
  compared with conventional tools.
• LTT is modular, extensible, and openly available.