Performance Data Collection: Hybrid Approach

1
Performance Data Collection Hybrid Approach

• Edu Metz, Raimondas Lencevicius
• Software Performance Architecture Group
• Nokia Research Center
• 5 Wayside Road, Burlington, MA 01803, USA
• E-mail Edu.Metz_at_nokia.com, Raimondas.Lencevicius
  _at_nokia.com

2
Motivation

• Performance profiling of complex embedded
  software is important
• Profiling monitoring software execution
  obtained data analysis
• 2 ways to collect data
• Event tracing
• Sampling
• Both have drawbacks and advantages
• What to use?

3
Position
Data collection should be performed using hybrid
approach that combines completeness of event
tracing with low cost of sampling
4
Overview

• Event tracing
• Sampling
• Hybrid data collection with example
• Why it will fail
• Why it will succeed

5
Event Tracing

• Record events occurring during system execution
• Component entry and exit
• Function calls
• Execution states
• Message communication
• Resource usage
• Complete record of information
• If event was recorded, it did occur if it was
  not recorded, it did not occur
• Allows complete analysis of event relationships
  in time
• Event tracing comes at a cost
• Trace instrumentation takes time
• Changes behavior of software system
• Could violate real-time constraints and timing
  requirements in real-time systems
• Disproportional overhead for different software
  components
• Large data volume

6
Sampling

• Intermittent recording of software systems state
• Program counter
• Call stack
• Scheduled and blocked tasks
• Can be done strictly periodically or with certain
  randomness
• Overhead may be orders of magnitude less than
  event tracing
• Has its own downsides
• Granularity of information determined by sampling
  frequency
• Needs reasonable data collection interval to
  provide accurate data
• Loss of information completeness
• May complicate application-level information
  extraction

7
Hybrid Approach

• Often a way to minimize drawbacks of two
  techniques is to use them both!
• Trace a subset of events
• Sample the rest
• Provide low overhead with adequate information
  completeness
• But how not to inherit drawbacks of two
  techniques?
• Worst case high overhead and not enough
  information for causality determination
• Intelligent selection of what should be traced
  and what should be sampled is needed

8
Hybrid Approach Example Profiling with Tasks

• System contains a number of communicating tasks
• Performance engineer needs to understand what
  parts of software consume most of the processor
  time and how this influences task messaging and
  scheduling
• Using only event tracing would lead to large
  overhead
• Using sampling would not provide complete
  information about task messaging
• Hybrid approach solves the problem
• Low overhead due to sampling of inside task
  activities
• Complete information of task messaging through
  event tracing - low overhead due to infrequent
  tracing

9
Hybrid Approach Heuristics

• What should be traced?
• Infrequent events
• Task switches
• Peripheral activations and deactivations
• Events that have causality relationships
• Message sends and receives
• Interrupts (maybe)
• Events for which relative time to other event is
  necessary
• Video frame display
• Events for which the absolute time is necessary
• Events connected with real world alarm clock
• What should be sampled?
• Frequent events
• Function calls (maybe)
• Bytecode/CPU op execution
• Events for which causality is not important
• Heap size (maybe)

10
Related Work

• We do not know any hybrid profilers
• A lot of sampling profilers
• Intel Vtune either sample or trace
• Hollingsworth et al. collect all and then
  sample the collection buffers periodically

11
Why it will Fail

• Intelligent selection of what should be traced
  and what should be sampled is difficult
• Bad tracing/sampling selection may lead to large
  overhead and missing information
• Trace instrumentation is still required with all
  its drawbacks
• Yields two sets of profiling data that need to be
  reconciled

12
Why it will Succeed

• Solves the issue of obtaining complete causality
  information with low overhead
• Allows collecting infrequent events even in short
  program runs
• Limits profiling data volume

13
Questions ?