Measurement Techniques

1
Measurement Techniques

• Eileen Kraemer
• August 27, 2002

2
Some definitions

• State of system
• defined by values of storage elements (memory,
  registers, etc.)
• Relevant subset primary variables
• Others auxiliary
• Event
• Change in a relevant state variable

3
Classification of measurement techniques

• Type A -
• number of times a state is visited during a time
  interval
• Example initiation of disk I/O per unit time
• Type B
• Value of auxiliary variables when relevant state
  is entered
• Example number of processes in ready list when
  I/O initiates

4
Classification of measurement techniques

• Type C
• Fraction or amount of time for which system is in
  a given state

5
Questions to ask

• When to collect info
• How to collect info

6
How to know when to collect

• Sample the system check if system is in
  relevant state sampled monitoring
• Trace the system look for event that marks
  entry to/exit from relevant state trace
  monitoring

7
Tracing v. sampling

• Tracing can do A, B, C
• Sampling A, B may not be possible
• Miss some instances, multiply count some
  instances, if duration of state is shorter than
  inter-sample gap can estimate
• Type C, can derive estimates

8
Instrumentation

• Hardware monitoring
• Pro
• doesnt interfere w/normal function
• Can capture fast events
• Con
• Expensive
• Many low-level events, difficult to re-assemble
  to correlate with higher-level operations
• Useful for
• A and C type for fast-occurring events
• Examples device utilizations, cache hit rate,
  pipeline flush rate

9
Instrumentation

• Software monitoring
• Measurement code added to software or called from
  within software
• Pro
• Flexible, general
• Con
• Perturbation, difficulty with fast-occurring
  events
• Useful for
• Info about user program, OS
• Examples time in routine X, page-fault
  frequency, average number of processors in state X

10
Instrumentation

• Hybrid monitoring
• Signals collected under software control, sent to
  another machine for measurement and processing
• Pro
• Flexible,applicable to wide range
• Con
• Synchronization requirements
• Expensive, cumbersome

11
Tracing v. sampling

• Both tracing and sampling applicable to all
  3(hardware, software, hybrid monitoring)

12
Issues in selecting instrumentation/monitoring
strategy

• Accessibility
• Software cant get to HW functions
• HW cant easily relate low-level events to
  higher-level operations
• Event frequency
• SW cant track if too rapid
• HW or sampled SW
• Monitor artifact
• Measurement process may perturb workload,
  affecting accuracy of analysis

13
Issues in selecting instrumentation/monitoring
strategy

• Overhead
• Reduce useful work by too large a margin
• Flexibility
• How easy to modify, upgrade instrumentation or
  change info being collected
• SW easier than HW easier than hybrid

14
Obstacles to monitoring

• Signals or state variables off-limits or
  unavailable
• security, privacy, protection, lack of
  documentation, source code unavailable,
  inaccessible location (on a chip)
• Poor event resolution
• Can get events, but insufficient info to classify
  (Example can count I/O ops but cant tell
  whether theyre from batch or interactive jobs)
• Poor clock resolution
• Inaccurate timing of fast occurring events

15
Hardware Monitoring

• Based on a logic signal, S
• 0-gt1 state entered
• 1-gt0 state exited
• May be synthesized from single bit and multi-bit
  signals
• Boolean functions
• Comparison functions

16
HM

• May also need auxiliary signal, S, to indicate
  which of several relevant states is occurring
• Example
• S 1 a new instruction fetched into IR
• S identity of opcode

17
HM

• Type A measurements
• Increment counter on S0-gt1
• Array of counters, indexed by S
• Type B measurements
• On S0-gt1, transfer auxiliary state info from
  backplane to registers or monitor memory module
• Type C measurements
• Assume no S
• Tracing Time periods starting 0-gt1 and 1-gt0
  very hard to do for fast changing HW signals
• Sampling most likely

18
HM

• silent observer
• Should have own counters, timers, logic
  synthesizers, memory modules, etc. rather than
  sharing HW w/ system under study
• Typically dont contribute to monitor artifact

19
Instrumentation for sampled hardware monitoring

• Example

20
Choosing the interval

• Want to measure the fraction of time in
  condition?
• Choose interval 2N clock pulses, where N is
  bits in the counter then dont have to divide,
  result in counter IS fraction

21
Controlling the measurement process

• Machine instruction or call to start measurement
• Clear counter, interrupt generator and interrupt
  flag
• Load event def register with right code
• CPU should recognize interrupt posted by
  measurement circuit
• Then read value in counter

22
Controlling the measurement process

• HW monitor should have
• entry in interrupt vector
• event def register and counter should appear as
  command and data registers to the CPU
• Interaction proceeds via normal bus interface
• Can pick up other signals on data bus directly
  (instruction opcodes, operand addreses, operand
  values)
• Other signals
• Explicitly put them on the bus to allow monitor
  to pick them up (then really hybrid monitoring)
• Directly tap pins on chips(not on pin -gtnot
  avail)

23
Example

• Consider a system with one CPU and n channels.
  Show the setup for measuring the fraction of time
  the CPU and k channels (for a given k in 1..n)
  are busy simultaneously.

24
Figure
25
Controlling measurement

• Provide a system call, IO_OVERLAP(k,X) to spawn
  process
• Determine function code to load into event
  definition register using parameter k
• Init event def reg, clear both counters, reset
  interrupt latch (monitoring then starts
  automatically)
• Block until monitor posts interrupt, then read
  value from duration counter, convert to real
  number, put in location X, exit

26
Sources of error.

• Consider previous with k0 .. measures CPU
  utilization
• Number of samples fixed at 2N, info is
  statistical, N must be large enough to give
  significant info
• Sampling frequency (clock rate) f, interval T
  2N / f. For accurate results, S must make many
  transitions in T.
• Avoid synch between sampling and sampled signal.
  (If perfectly synchronized, then value would be
  100 or 0)..

27
Example

• Devise a sampled measurement technique for
  estimating the time spent by a program within a
  given loop.

28
Figure
29
Notes

• Address bus carries real addresses
• Wont work in virtual memory system or if
  addresses not guaranteed to be contiguous
• If memory management scheme permits programs to
  be dynamically relocated but contiguous, bounds
  registers must be reloaded every time the program
  is moved

30
Notes

• PC holds instruction addresses
• Why not use it instead of the address bus?
• PC is inside microprocessor chip, cant be
  accessed directly
• PC address is virtual, several programs could
  have same set of virtual addresses

31
Notes

• Because of sampling, time duration of experiment
  is only an approximation

32
Process-specific measurements

• Difficult to do in hardware
• Need to maintain identifying info in registers
  available to monitor

33
Software Monitoring

• Well-suited for program-level measurements
• Requires some support from OS and hardware
• Programmable timer
• Virtual clock
• Programmable virtual timer

34
Programmable timer

• load with desired time interval, count down,
  generate an interrupt at time zero
• Interrupt handling routine
• Read state variables, process collected data,
  close down experiment

35
Virtual clock

• Needed for measuring process-specific time
  durations
• Acts as a real-time clock that runs only while
  process n is executing
• Single physical clock
• Reserve slot in process control block (PCB) of
  each process for storing timer contents store
  out/in on context switch

36
Programmable virtual timer

• Runs down only when process n is executing
• And associate arbitrary routine with expiration
  of timer
• Similar to installing new device drivers,
  performed through system call interface, often in
  privileged mode

37
Trace Monitoring

• Add extra code to program to record info when
  interesting events occur
• Pro
• Flexibility
• Con
• Instrumentation process may require detailed
  understanding of program
• Added code may contain bugs
• Added code may perturb in unexpected ways
• Source code may be unavailable, undocumented,
  difficult to understand

38
gprof

• Trace monitoring facility available under
  Berkeley Unix (and descendants)
• To use
• Compile with pg option (inserts monitoring code
  for each procedure call)
• Computes average time take by each procedure,
  writes info to separate file
• Can use a option to obtain time take by major
  program blocks
• Well try out soon .

39
Interactive instrumentation environments

• Similar idea to interactive debugger
• Provide hooks in code, add instrumentation later
• Pathfinder/QBV is example for DS

40
(No Transcript)
41
Pathfinder/QBV
local snapshots
steering requests
Snapshot and Steering Manager
global snapshots
logical time
Membership and ordering information
and Consistency Detection

local steering requests
Snapshot and Steering Manager
Presentation Manager
Interaction Managers
42
Software Trace Monitoring

• Example
• New compiler slower than expected Found to be
  spending too much time manipulating the symbol
  table. Show how the fraction of time used for
  symbol table manipulations (frac) can be measured
  accurately.

43
Solution

• Time spent manipulating symbol table depends on
  program being compiled need set of randomly
  selected programs, or specifically selected set
  designed to be representative of actual workload
• Use statistical techniques to analyze data

44
Solution, continued

• Tot_time time needed for compilation
• ST-time time spent manipulating symbol table
• Frac ST-time / Tot_time

45
Computing Tot_time

• CPU time, usually available directly
• To compute explicitly
• At beginning of compilation initialize virtual
  clock
• At end, read clock
• Can modify compiler code, or by creating command
  line code (batch file) or calling program that
  does clock ops and calls compiler

46
Computing ST_time

• Modify compiler
• Add flag variable
• Identify procedures that comprise ST_handling
  function
• add measurement statements that depend on flag

47
Instrumentation of compiler routine
Procedure pn(parameter list) int start, end
if (measure_flag) start read_virtual_clock()
. if (measure_flag) finish
read_virtual_clock() ST_time (start
finish)
48
Notes

• Additional code affects (increases) both ST_time
  and Tot_time.
• Tot_time more affected, thus frac is likely
  underestimated.
• Use uninstrumented for Tot_time?
• Estimate instrumentation effect and subtract?

49
Notes

• Function call overhead can be significant we
  arent measuring it
• Solution
• Start read_virtual_clock()
• Pn(param_list)
• Finish read_virtual_clock()
• ST_time ST_time (finish start)
• Con
• Requires modification of entire prog, not just
  module of interest
• Alternate Solution
• Measure function invocation overhead separately,
  and keep track of number of calls, then adjust
  accordingly

50
Notes

• Other forms of perturbation may occur
• Change in workload, with effects on
• Process scheduling
• Page faults
• Etc.

51
Sampled monitoring in SW

• Use programmable timer to generate periodic
  interrupts
• Interrupt-handling routine reads state info
• Example how much time spent in procedure
• IH checks PC, increments counter if within range

52
Sampled monitoring in SW

• Pro no program instrumentation needed
• Dont need source code, cant add errors, etc.
• Easier to make corrections (estimate monitoring
  overhead)
• Con
• statistical, not exact, results
• IH overhead, maybe context switch overhead
• Some things hard to measure by sampling

53
Sampled monitoring in SW

• Example
• Use sampled software monitoring to estimate the
  distribution of seek distances for a moving head
  disk.

54
Solution seek distance distribution

• Assume
• File system variable CYL contains current head
  position
• Seek dist current CYL prev CYL
• Assume 10 40 20 60 30 50
• Actual_dist 30 20 40 30 20
• IH read current CYL
• Sample rate too low -- 10 20 30
• Sample_seek_dist 10 10
• Sample rate too high 10 10 10 40 40 40 20 20
• Sample_seek_dist 0 0 30 0 0 20 ..
• Tracing would be better approach for this problem

55
Goals of SW monitoring

• System tuning
• Alteration of some OS params so all users
  experience better response from system
• Accessibility is limited, may be limited to
  sampling
• Program tuning
• Adjustment of parameters in programs of interest
  to one or a few users
• Arbitrary source modification possible, tracing
  more likely possible

56
In either case

• 90-10 rule applies
• 90 of time spent in 10 of code
• find that 10 and work to improve it, worry less
  about remaining 90
• Strategies for improvement
• Better data structure /algorithm
• Pass by value -gt pass by ref
• Inline
• Compiler optimization flags

57
Hybrid Monitoring

• Transfer relevent info under SW control from
  system under measurement to set of interface
  registers
• Info then picked up by measuring system

Measured system
interface
Measuring system
58
Hybrid monitoring

• Goal
• Obtain complex measurements
• Keep workload perturbation w/in tolerable levels

59
Hybrid Monitoring example

• Measure relative frequency of instruction usage
• HW approach
• SW approach
• Hybrid approach

60
Instruction frequency HW approach

• Pick instruction opcode from data bus when
  instruction is fetched
• User counter array, initialized to zeros
• Increment countf(opcode)
• Relative freq countn / sum of counts
• Con
• requires substantial HW
• inflexible

61
Instruction frequency SW approach

• Counter array in main memory, initialized to zero
• trap after every instruction
• (Can be done on some machines)
• Turn off trapping
• Increment counteropcode
• Turn on trapping
• Return
• Con
• Huge overhead trap on every instruction

62
Instruction frequency hybrid approach

• M1 (under measurement) outputs opcode to
  interface register
• M2 (measuring system) maintains counter array,
  picks up opcode from interface register,
  increments counter
• Pro
• Dont need new HW if measurement objectives
  change .. Just reprogram M2
• M1 can place any relevant info on bus and into IR

63
Another possibility

• DMA Direct Memory Access
• Permits measuring system to xfer large chunks of
  info to/from main memory of measured system,
  w/out active cooperation of measured system
• Reduces effect of measurement on measured system
• Can capture fast hw-level signals

64
Hybrid Monitoring setup phase

• Specify/supply control program to be run on
  measured system
• Specify what info to caputre and when decide
  what info goes into which general purpose
  register or flag, when written, when read
• Specify how measuring system should retrieve info
  from the interface, what it should it.
• Specify how measured and measuring should
  synchronize. use flags, establish protocol

65
Example instruction frequency measurement
(assume M2 fast)

• Control program benchmark program for
  instruction frequency measurement
• Interface assignments
• R, register to hold opcode
• F1, flag to indicate opcode placed in R
• could do busy-wait on F1 or use F1 to post
  interrupt
• Measuring system reads R, increments appropriate
  counter element

66
Example instruction frequency measurement (M2
not fast )

• If M2 cant process opcode before another
  arrives, need two-way synch
• Classical single-slot producer-consumer
  synchronization problem
• Need F2 semaphore flag
• Init F10, F2 1

67
Two-way synch in hybrid measurement
M1 Loop process next instr busy-wait until
F2 1 reset F2 R opcode set
F1 Forever
M2 Loop wait until F1 1 reset F1 read
opcode from R increment countopc Set
F2 Forever
68
Notes on synch

• cause

69
Notes on HM

• 2-way synch causes monitor artifact, so try to
  avoid
• To get sw-level info, control program must
  interact w/ monitoring interface as a device
• Program must be instrumented to effect xfer of
  info through interface
• Limited amount of software info can be xferred,
  interface has limited storage, want to limit
  monitor interferece
• Info obtained from bus v. memory ,, need to be
  sure it is up to date

70
Notes on HM

• Less flexible than SW
• Slower than HW for fast-changing signals HW may
  be required

71
HMtracing v. sampling

• Tracing (as in example) gives exact results, but
  need proper synch
• Sampling
• M1 and M2 operate autonomously
• M1 puts info on interface register whenever
  necessary
• M2 reads periodically
• May use two sets of interface registers

72
Real-time control of performance

• Data collection, reduction (analysis), and usage
  all occur on-the-fly
• Measure parameter of interest (response time,
  throughput, memory usage) for each transaction
• Compute performance index (PI) (average,
  percentile)
• Compare PI to desired value
• Adjust control parameter accordingly

73
Real-time control of system performance
Desired PI level
Actual workload
compute PI
select sense
system
compare
converter
Control parameter updated
error
74
Issues in real-time control of system performance

• How to estimate PI
• How to convert error signal to change in CP
• How to apply control

75
Example

• Interactive computer system
• R average system response time
• T threshhold system response time, in seconds
• Goal maintain R below T

76
Estimating R

• Using earlier techiques (hybrid or software)
• However
• Must be done continuously under live workload,
  overhead a concern
• Cant use long measurement intervals
• Want to use measurement results in loop

77
Estimating R

• Measure response time experienced by n successive
  user commands
• Ri - response time of ith command

78
Estimating R

• Which n commands to use?
• Re-estimating on every new command causes jittery
  control
• So, recompute only on blocks of size n

79
Smoothing the controls

• Ri(k) ith response time during the kth block

80
Smoothing the controls

• PI(k) the performance index for the kth block
• ? - damping factor (0lt ? lt 1)
• Higher ? -- more importance to past behavior
• Exponential averaging

81
Note

• For any ? lt 1, effect of past behavior is damped
  exponentially, since

82
Choosing n

• Too small control is jittery
• Too large control is sluggish, lags

83
Error signal

• MaxPI(k) T,0

84
Choosing CP options

• Alter memory parameters
• Examples reduce degree of multiprogramming,
  alter working set size
• Suspend background processing
• Mail, accounting, logging, anything without
  timing constraints
• Suspend or terminate batch processes started by
  users, dont allow any more batch submissions

85
How are CP and error related?

• Approach successive approximation
• Guess an amount, observe an effect
• Example suspend batch processes one by one,
  until R lt T
• Approach binary search type
• Guess an amount, observe an effect
• Example suspend many batch processes, restart
  some, if possible
• Approaches may lead to jitter, lag, instability

86
Example

• Computer system with demand-paged memory
  management. Show how a real-time control
  mechanism can be used to keep system throughput
  near the optimum value.

87
Solution

• Adaptive control of the degree of
  multiprogramming (DMP)

88
Adaptive control of DMP

• Low end not enough jobs in the system lead to
  low throughput
• High end too many jobs in the system leads to
  thrashing
• Optimum DMP high enough to keep paging device
  busy, not too high to overload it

89
Adaptive control of DMP

• F average time between successive page faults
• S average time needed to swap in a page
• If F ? S , desirable condition

90
Adaptive control of DMP

• F depends on DMP, but also on locality, size, and
  other characteristics of the program so needs
  to be estimated on the fly
• F the performance index
• S target value

91
Adaptive control of DMP

• Measure time between successive page faults by
  trace monitoring in the page fault handling
  routine
• If requested page must be read from disk, note
  time, and compare to previous time
• Can also use damped averaging
• DMP too high swap out some processes
• DMP too low swap in some processes or bring in
  new ones

92
Measurability limitations

• Not all desired hardware signals are available on
  the system backplane
• In a chip (inaccessible), unless on a pin
• Solution multiplex many signals on a few pins
  (rarely done by manufacturers)
• Solution execute instruction that causes signal
  to be placed on the bus or result in some other
  observable effect

93
Measurability limitations

• Circuit considerations
• Fan-out limits
• Time skew from additional loading
• Problems in accessing signals on circuit board
  not brought out to connector pins

94
Measurability limitations

• Even if HW allows all reasonable measurements to
  be obtained, HW is unaware of software-level info
• Software-level measurements may require OS
  support retaining info in kernel DSs, making
  such info available for HW monitoring

95
Measurability limitations

• Privacy and security concerns
• Undesirable to allow user to acces info about
  processes, files, directories belonging to other
  users ..
• Solution
• Full measurability at machine level, only
  trusted users can access

96
Measurability limitations

• Lack of documentation,unavailablity of source
  code
• Poor resolution of clocks and timers provided by
  the OS

97
Measurability limitations

• Model calibration calibration of simulation and
  analytic models of performance
• Model must abstract out many details of the
  physical system
• Model calibration is difficult, usually biggest
  source of error
• Simple models usually better than complex (fewer
  parameters to go astray)

98
Monitoring Parallel Systems

• Assumptions of previous discussion
• HW monitor is able to track all relevant events
• Hybrid fast measuring system to avoid two-way
  synchronization between systems
• Problem high-end parallel machines probably
  already use fastest available logic, so may not
  be able to have faster monitor

99
Monitoring Parallel Systems

• If data produced rapidly, may not be able to
  reduce (filter, analyze) it online or store on
  secondary storage device
• May have to provide large buffer in monitor
  itself
• Need to correlate separate streams
• Monitors expensive, depend on parallelism of
  machine

100
Monitoring Distributed Systems

• Tracking messages between components is essential
  for cause-effect relationships between various
  activities
• Problems
• Cant track lower-layer messages from app layer
• Dont have all info about msg (routing, lost or
  corrupted packets)
• At lower levels, dont have app context info

101
Monitoring Distributed Systems

• Hierarchical Monitoring
• Monitor individual machines and connections
  between them
• Correlate gathered info based on app-specific or
  other higher-level info

102
Monitoring Distributed Systems

• Given set of measurement objectives, decide on
  LOD (level of detail) to monitor

103
Overall goals

• Record data
• Reduce data
• Compute performance measures
• Locate trouble spots
• Formulate remedial actions
• Apply remedial actions ..