TDC597 Linuxbased Network Management Lecture Seven Performance Management

1
TDC597Linux-based Network ManagementLecture
SevenPerformance Management

• James T. Yu, Ph.D.
• jyu_at_cs.depaul.edu
• School of CTI
• DePaul University

2
Lecture Outline

• MIBs for Performance Analysis
• Linux Cron Jobs
• Perl Script for Performance Analysis
• SNMP Configuration on Linux
• MRTG
• Hw06

3
Application Development

• Define the performance metrics
• Define the measurement scheme
• Design the data collection mechanism
• Design the data collection process
• Collect the data
• Develop the Perl program to compute the metrics
  and analyze the data
• Import a 3rd party application

4
Performance Metrics

• Round trip delay
• Jitter
• Packet loss
• Throughput
• Utilization
• Link
• Processor
• Memory

5
Measurement Process (server)
Switch-02
Server (Linux04)
ifconfig gt find its MAC address eth0 Link
encapEthernet HWaddr 0019B9B7AC9D
Use the MAC Address table to find the port on the
Ethernet switch. 0019.b9b7.ac9d DYNAMIC
gi0/2 Monitor the traffic to and from this port.
(gi0/2)
ifIndex 26
6
Measurement Process (backbone)
Router Backbone Traffic
Switch-01
Use the MAC Address table to find the port on the
Ethernet switch. 0.20.106.209.59.144 fa0/2
00-14-6A-D1-3B-90 Monitor the traffic to and from
this port. (fa0/2)
ifIndex 2
jyu_at_linux05 jyu arp -a labRouter
(140.192.40.1) at 00146AD13B90 ether on
eth0
7
MIBs for Performance Monitoring

• I_MIB"1.3.6.1.2.1.2.2.1.10
• ifInOctets bytes coming into the witch
• O_MIB"1.3.6.1.2.1.2.2.1.16"
• ifOutOctets bytes going out of the switch
• For Interface gi0/2, the MIBs are
• I_MIB"1.3.6.1.2.1.2.2.1.10.26
• Bytes going to the server (Linux04)
• O_MIB"1.3.6.1.2.1.2.2.1.16.26
• Bytes coming from the server (Linux04)

8
Data Collection Process

• A Shell script to collect the data from network
  devices
• Covered in Lecture Six
• A scheduling job that collect the data regularly
• Linux cron job

9
Linux Cron Jobs

• Cron is a Linux daemon process that enables you
  to execute a Shell script at a specified time or
  at a specific time of day.
• Users can also set the time interval for the
  execution of the script.
• The script that you want cron to run is defined
  in a file called crontab, and every user has
  their own independent crontab file.
• Cron is similar to the Windows scheduler which
  allows you to run commands/programs at predefined
  times and intervals.

10
Create a crontab file
The following script will be executed every 30
minutes.
0,30
/home/jyu/tdc597/cron/xrun

• Minute 00-59 exact minute the cron executes
• Hour 00-23 hour of the day the cron executes(0
  means midnight)
• day 01-31 day of the month the cron executes
• month 01-12 month of the year the cron executes\
• weekday 0-6 day of the week the cron executes
• (Sunday 0, Monday 1, Tuesday 2, and so
  forth)
• 6. Command complete PATH of the script to
  execute

11
Cronfile Time Examples
12
Linux command crontab

• crontab l show the list of cron jobs
• crontab r remove the cron jobs
• crontab ltfilegt Start the cron jobs with ltfilegt

For more information man crontab
13
Shell Script for Data Collection
14
Cron Job
15
Performance Analysis

• Understand the raw data (inputs)
• Specify the outputs
• Application development
• Testing and Validation

16
Perl Reading Multiple Files
set inut
files for data analysis and report
PREFIX/home/jyu/tdc597/
crom _at_SNMPDATA ( ) for(my ddstartDay
ddltendDay dd) _at_SNMPDATA (_at_SNMPDATA,
PREFIX/snmpdata.12-dd") foreach snmpfile
(_at_SNMPDATA) print "snmpfile
intervalinterval\n" open(PERFDATA, "lt
snmpfile") while( linebuf
ltPERFDATAgt) Perl code to compute performance
metrics
17
Performance Analysis Raw Data
18
Performance Analysis Raw Data
19
Output Specificationswhat could get from this
data?

• Metrics
• Throughput (average)
• Throughput (peak)
• Link utilization
• Time series analysis
• hourly
• daily
• weekly
• monthly

20
Script Syntax (Usage)
Usage perl perf.pl ltinterfacegt -from ltmm-ddgt
-to ltmm-ddgt -hourly -daily
-monthly Output Format 1. count of data
sessions 2. date/time 3. inOctets
(total) 4. outOctets (total) 5. in
throughput 6. out throughput 7. in peak
throughput (in 10 min interval) 8. out peak
throughput (in 10 min interval)
Q why there is no data report on link
utilization? A no single data point is greater
than 1. - not worth reporting it.
21
Performance Analysis counting bytes
my data _0 bytes my flag
_1 if( flag 0 ) ifInOctets
if( baseInOctets 0 )
dataInOctets 0
baseInOctets data else
dataInOctets data -
baseInOctets
baseInOctets data elsif(
flag 1 ) ifOutOctets if(
baseOutOctets 0 )
dataOutOctets 0
baseOutOctets data else
dataOutOctets data -
baseOutOctets
baseOutOctets data
22
Data Aggregation (I)
tt timer from the source file time
format 12-05-000001 my _at_a split(/\-/,
tt) if( int eq "-hourly")
my _at_b split(//, a2)
timeIndex "a0-a1-b0" elsif( int
eq "-daily" ) timeIndex
"a0-a1" elsif( int eq "-monthly" )
timeIndex a0
23
Data Aggregation (II)
if( timeIndex eq oldTimeIndex )
if( peakInOctets lt dataInOctets )
peakInOctets dataInOctets
if( peakOutOctets lt
dataOutOctets )
peakOutOctets dataOutOctets
dataInTotalOctets
dataInOctets dataOutTotalOctets
dataOutOctets else if(
oldTimeIndex ne "" )
printPerfReport(oldTimeIndex)
peakInOctets
dataInOctets peakOutOctets
dataOutOctets
dataInTotalOctets dataInOctets
dataOutTotalOctets dataOutOctets
oldTimeIndex timeIndex
24
Programmer Notes

• There are no hash tables in this application.
• The correlation of the data is only between
  datat and datat1 . Therefore, we need
  only two variables to capture this relation
• TimeIndex (current time)
• oldTimeIndex
• This is an important tip for time series analysis
  where the data correlation is well preserved in
  the data itself.
• Hash table is a useful and powerful programming
  technique however, we do not need it when there
  is a simpler solution.

25
Performance Report
my inTP 8 dataInTotalOctets / (nData
dataInterval) my outTP 8
dataOutTotalOctets / (nData
dataInterval) inTP int(inTP) outTP
int(outTP) my inUtil 100 inTP /
100000000 in percentage my outUtil 100
outTP / 100000000 in percentage my inPeakTP
(8 peakInOctets / dataInterval) my
outPeakTP (8 peakOutOctets /
dataInterval) inPeakTP int(inPeakTP) outP
eakTP int(outPeakTP) printf("d s 9d 9d\t
7d 7d\t 8d 8d\n", nData,
idx, dataInTotalOctets,
dataOutTotalOctets,
inTP,outTP, inPeakTP,outPeakTP)
26
Performance Analysis on Linux

• netstat -i
• top
• ping
• traceroute
• snmp

27
netstat -i
28
Linux Monitoring via SNMP

• Script to start SNMP
• /etc/init.d/snmpd start
• /etc/init.d/snmpd stop
• SNMP configuration file
• /etc/snmp/snmpd.conf
• Save a backup copy before you modify this file

29
Default SNMP setting on Linux(only system MIBs)
30
snmpd configuration fileadding more MIBs
31
SNMP Data Query to Linux
BEFORE (w/o MIBs)
AFTER (w/ MIBs)
32
Linux MIBs for CPU
CPU Statistics Load1 minute Load
.1.3.6.1.4.1.2021.10.1.3.15 minute Load
.1.3.6.1.4.1.2021.10.1.3.215 minute Load
.1.3.6.1.4.1.2021.10.1.3.3 CPUpercentage of user
CPU time .1.3.6.1.4.1.2021.11.9.0raw user cpu
time .1.3.6.1.4.1.2021.11.50.0percentages of
system CPU time .1.3.6.1.4.1.2021.11.10.0raw
system cpu time .1.3.6.1.4.1.2021.11.52.0percent
ages of idle CPU time .1.3.6.1.4.1.2021.11.11.0r
aw idle cpu time .1.3.6.1.4.1.2021.11.53.0raw
nice cpu time .1.3.6.1.4.1.2021.11.51.0
33
Linux MIBs for Memory
Memory Statistics Total Swap Size
.1.3.6.1.4.1.2021.4.3.0Available Swap Space
.1.3.6.1.4.1.2021.4.4.0Total RAM in machine
.1.3.6.1.4.1.2021.4.5.0Total RAM used
.1.3.6.1.4.1.2021.4.6.0Total RAM Free
.1.3.6.1.4.1.2021.4.11.0Total RAM Shared
.1.3.6.1.4.1.2021.4.13.0Total RAM Buffered
.1.3.6.1.4.1.2021.4.14.0Total Cached Memory
.1.3.6.1.4.1.2021.4.15.0
Ref http//www.debianadmin.com/linux-snmp-oids-fo
r-cpumemory-and-disk-statistics.html
34
MRTG

• The Multi Router Traffic Grapher (MRTG) is a tool
  to monitor the traffic load on network links.
  MRTG generates HTML pages containing PNG images
  which provide a LIVE visual representation of
  this traffic.
• Official web site http//www.mrtg.org
• Documentation http//oss.oetiker.ch/mrtg/doc/mrtg
  .en.html

35
MRTG Highlights

• Portable - MRTG works on most UNIX/Linux
  platforms and Windows.
• Perl - MRTG is written in Perl and comes with
  full source.
• Portable SNMP - MRTG Uses a highly portable
  SNMP implementation written entirely in Perl
  (thanks to Simon Leinen). There is no need to
  install any external SNMP package.
• Reliable Interface Identification
• Router interfaces can be identified by IP
  address, description and ethernet address in
  addition to the normal interface number.
• Constant size Logfiles - MRTG's logfiles do NOT
  grow thanks to the use of a unique data
  consolidation algorithm.
• Automatic Configuration - MRTG comes with a set
  of configuration tools which make configuration
  and setup very simple.
• Performance - Time critical routines are written
  in C (thanks to the initiative of Dave Rand my
  Co-Author).

36
MRTG Installation

• You do not need root privilege to install it.
• Create the file structure for mrtg
• Download mrtg-2_9_21_tar.gz to the above
  directory

cd mkdir MRTG cd MRTG
gunzip mrtg-2_9_21_tar.gz tar xf mrtg-2_9_21_tar
37
MRTG Installation (cont.)

• Read the installation guide
• http//oss.oetiker.ch/mrtg/doc/mrtg-unix-guide.en.
  html

configure make make install only if you are
root
TDC597 MRTG is already installed on Linux04 at
/usr/local/mrtg-2/bin
38
MRTG User Process

• Run cfgmaker to create the MRTG configuration
  file (mrtg.cfg)
• Run mrtg with mrtg.cfg as the parameter to
  collect the data and generate graphical files
• Use the HTML browser to view the data
• If the manual process works, use cronjob to run
  it at a regular interval.

39
cfgmaker

• Purpose creates mrtg.cfg files
• Syntax cfgmaker options community_at_router
  options community_at_router ...
• Options
• -global "x a" add global config entries
• --outputfile output filename default is
  STDOUT

40
MRTG Configuration File
Create a configuration script (confMRTG.sh)
BIN/usr/local/mrtg-2/bin WORKDIR/home/jyu/public
_html/mrtg OUTPUT/home/jyu/public_html/mrtg/mrtg.
cfg DEVICE"public_at_192.168.1.1" BIN/cfgmaker
--global "WorkDir WORKDIR" --output OUTPUT
DEVICE
Run the Configuration script
Note each MRTG configuration file (mrtg.cfg) is
for ONE network device. If you need to collect
performance data from multiple devices, you need
to create multiple CFG files (with different
names).
41
mrtg (Linux Command)

• mrtg is the name of the script you have to run to
  poll data and generate the graphs. Most
  configuration is set through the configuration
  file.
• mrtg is located at /usr/local/mrtg-2/bin on
  Linux04.
• Syntax mrtg options mrtg.cfg

42
Data Collection create a script and run it
The outputs are HTML files.
43
MRTG Screen Shots
44
indexmaker

• Purpose creates index files for mrtg web sites
• Syntax indexmaker
• --outputHOME/mrtg/index.html
• CFG File
• The index file has a summary graph for each
  interface of the device.

45
(No Transcript)
46
HW06Developing a Web Site forPerformance
Analysisfrom CGI to GUI
47
Summary

• Performance Analysis is primarily based on two
  MIBs
• ifInOcters 1.3.6.1.2.1.2.2.1.10
• ifOutOctets 1.3.6.1.2.1.2.2.1.16
• Software Development
• Home-grown tool
• Import 3rd party tools
• You need both.