Title: TDC597 Linuxbased Network Management Lecture Seven Performance Management
1TDC597Linux-based Network ManagementLecture
SevenPerformance Management
- James T. Yu, Ph.D.
- jyu_at_cs.depaul.edu
- School of CTI
- DePaul University
2Lecture Outline
- MIBs for Performance Analysis
- Linux Cron Jobs
- Perl Script for Performance Analysis
- SNMP Configuration on Linux
- MRTG
- Hw06
3Application Development
- Define the performance metrics
- Define the measurement scheme
- Design the data collection mechanism
- Design the data collection process
- Collect the data
- Develop the Perl program to compute the metrics
and analyze the data - Import a 3rd party application
4Performance Metrics
- Round trip delay
- Jitter
- Packet loss
- Throughput
- Utilization
- Link
- Processor
- Memory
5Measurement Process (server)
Switch-02
Server (Linux04)
ifconfig gt find its MAC address eth0 Link
encapEthernet HWaddr 0019B9B7AC9D
Use the MAC Address table to find the port on the
Ethernet switch. 0019.b9b7.ac9d DYNAMIC
gi0/2 Monitor the traffic to and from this port.
(gi0/2)
ifIndex 26
6Measurement Process (backbone)
Router Backbone Traffic
Switch-01
Use the MAC Address table to find the port on the
Ethernet switch. 0.20.106.209.59.144 fa0/2
00-14-6A-D1-3B-90 Monitor the traffic to and from
this port. (fa0/2)
ifIndex 2
jyu_at_linux05 jyu arp -a labRouter
(140.192.40.1) at 00146AD13B90 ether on
eth0
7MIBs for Performance Monitoring
- I_MIB"1.3.6.1.2.1.2.2.1.10
- ifInOctets bytes coming into the witch
- O_MIB"1.3.6.1.2.1.2.2.1.16"
- ifOutOctets bytes going out of the switch
- For Interface gi0/2, the MIBs are
- I_MIB"1.3.6.1.2.1.2.2.1.10.26
- Bytes going to the server (Linux04)
- O_MIB"1.3.6.1.2.1.2.2.1.16.26
- Bytes coming from the server (Linux04)
8Data Collection Process
- A Shell script to collect the data from network
devices - Covered in Lecture Six
- A scheduling job that collect the data regularly
- Linux cron job
9Linux Cron Jobs
- Cron is a Linux daemon process that enables you
to execute a Shell script at a specified time or
at a specific time of day. - Users can also set the time interval for the
execution of the script. - The script that you want cron to run is defined
in a file called crontab, and every user has
their own independent crontab file. - Cron is similar to the Windows scheduler which
allows you to run commands/programs at predefined
times and intervals.
10Create a crontab file
The following script will be executed every 30
minutes.
0,30
/home/jyu/tdc597/cron/xrun
- Minute 00-59 exact minute the cron executes
- Hour 00-23 hour of the day the cron executes(0
means midnight) - day 01-31 day of the month the cron executes
- month 01-12 month of the year the cron executes\
- weekday 0-6 day of the week the cron executes
- (Sunday 0, Monday 1, Tuesday 2, and so
forth) - 6. Command complete PATH of the script to
execute
11Cronfile Time Examples
12Linux command crontab
- crontab l show the list of cron jobs
- crontab r remove the cron jobs
- crontab ltfilegt Start the cron jobs with ltfilegt
For more information man crontab
13Shell Script for Data Collection
14Cron Job
15Performance Analysis
- Understand the raw data (inputs)
- Specify the outputs
- Application development
- Testing and Validation
16Perl Reading Multiple Files
set inut
files for data analysis and report
PREFIX/home/jyu/tdc597/
crom _at_SNMPDATA ( ) for(my ddstartDay
ddltendDay dd) _at_SNMPDATA (_at_SNMPDATA,
PREFIX/snmpdata.12-dd") foreach snmpfile
(_at_SNMPDATA) print "snmpfile
intervalinterval\n" open(PERFDATA, "lt
snmpfile") while( linebuf
ltPERFDATAgt) Perl code to compute performance
metrics
17Performance Analysis Raw Data
18Performance Analysis Raw Data
19Output Specificationswhat could get from this
data?
- Metrics
- Throughput (average)
- Throughput (peak)
- Link utilization
- Time series analysis
- hourly
- daily
- weekly
- monthly
20Script Syntax (Usage)
Usage perl perf.pl ltinterfacegt -from ltmm-ddgt
-to ltmm-ddgt -hourly -daily
-monthly Output Format 1. count of data
sessions 2. date/time 3. inOctets
(total) 4. outOctets (total) 5. in
throughput 6. out throughput 7. in peak
throughput (in 10 min interval) 8. out peak
throughput (in 10 min interval)
Q why there is no data report on link
utilization? A no single data point is greater
than 1. - not worth reporting it.
21Performance Analysis counting bytes
my data _0 bytes my flag
_1 if( flag 0 ) ifInOctets
if( baseInOctets 0 )
dataInOctets 0
baseInOctets data else
dataInOctets data -
baseInOctets
baseInOctets data elsif(
flag 1 ) ifOutOctets if(
baseOutOctets 0 )
dataOutOctets 0
baseOutOctets data else
dataOutOctets data -
baseOutOctets
baseOutOctets data
22Data Aggregation (I)
tt timer from the source file time
format 12-05-000001 my _at_a split(/\-/,
tt) if( int eq "-hourly")
my _at_b split(//, a2)
timeIndex "a0-a1-b0" elsif( int
eq "-daily" ) timeIndex
"a0-a1" elsif( int eq "-monthly" )
timeIndex a0
23Data Aggregation (II)
if( timeIndex eq oldTimeIndex )
if( peakInOctets lt dataInOctets )
peakInOctets dataInOctets
if( peakOutOctets lt
dataOutOctets )
peakOutOctets dataOutOctets
dataInTotalOctets
dataInOctets dataOutTotalOctets
dataOutOctets else if(
oldTimeIndex ne "" )
printPerfReport(oldTimeIndex)
peakInOctets
dataInOctets peakOutOctets
dataOutOctets
dataInTotalOctets dataInOctets
dataOutTotalOctets dataOutOctets
oldTimeIndex timeIndex
24Programmer Notes
- There are no hash tables in this application.
- The correlation of the data is only between
datat and datat1 . Therefore, we need
only two variables to capture this relation - TimeIndex (current time)
- oldTimeIndex
- This is an important tip for time series analysis
where the data correlation is well preserved in
the data itself. - Hash table is a useful and powerful programming
technique however, we do not need it when there
is a simpler solution.
25Performance Report
my inTP 8 dataInTotalOctets / (nData
dataInterval) my outTP 8
dataOutTotalOctets / (nData
dataInterval) inTP int(inTP) outTP
int(outTP) my inUtil 100 inTP /
100000000 in percentage my outUtil 100
outTP / 100000000 in percentage my inPeakTP
(8 peakInOctets / dataInterval) my
outPeakTP (8 peakOutOctets /
dataInterval) inPeakTP int(inPeakTP) outP
eakTP int(outPeakTP) printf("d s 9d 9d\t
7d 7d\t 8d 8d\n", nData,
idx, dataInTotalOctets,
dataOutTotalOctets,
inTP,outTP, inPeakTP,outPeakTP)
26Performance Analysis on Linux
- netstat -i
- top
- ping
- traceroute
- snmp
27netstat -i
28Linux Monitoring via SNMP
- Script to start SNMP
- /etc/init.d/snmpd start
- /etc/init.d/snmpd stop
- SNMP configuration file
- /etc/snmp/snmpd.conf
- Save a backup copy before you modify this file
29Default SNMP setting on Linux(only system MIBs)
30snmpd configuration fileadding more MIBs
31SNMP Data Query to Linux
BEFORE (w/o MIBs)
AFTER (w/ MIBs)
32Linux MIBs for CPU
CPU Statistics Load1 minute Load
.1.3.6.1.4.1.2021.10.1.3.15 minute Load
.1.3.6.1.4.1.2021.10.1.3.215 minute Load
.1.3.6.1.4.1.2021.10.1.3.3 CPUpercentage of user
CPU time .1.3.6.1.4.1.2021.11.9.0raw user cpu
time .1.3.6.1.4.1.2021.11.50.0percentages of
system CPU time .1.3.6.1.4.1.2021.11.10.0raw
system cpu time .1.3.6.1.4.1.2021.11.52.0percent
ages of idle CPU time .1.3.6.1.4.1.2021.11.11.0r
aw idle cpu time .1.3.6.1.4.1.2021.11.53.0raw
nice cpu time .1.3.6.1.4.1.2021.11.51.0
33Linux MIBs for Memory
Memory Statistics Total Swap Size
.1.3.6.1.4.1.2021.4.3.0Available Swap Space
.1.3.6.1.4.1.2021.4.4.0Total RAM in machine
.1.3.6.1.4.1.2021.4.5.0Total RAM used
.1.3.6.1.4.1.2021.4.6.0Total RAM Free
.1.3.6.1.4.1.2021.4.11.0Total RAM Shared
.1.3.6.1.4.1.2021.4.13.0Total RAM Buffered
.1.3.6.1.4.1.2021.4.14.0Total Cached Memory
.1.3.6.1.4.1.2021.4.15.0
Ref http//www.debianadmin.com/linux-snmp-oids-fo
r-cpumemory-and-disk-statistics.html
34MRTG
- The Multi Router Traffic Grapher (MRTG) is a tool
to monitor the traffic load on network links.
MRTG generates HTML pages containing PNG images
which provide a LIVE visual representation of
this traffic. - Official web site http//www.mrtg.org
- Documentation http//oss.oetiker.ch/mrtg/doc/mrtg
.en.html
35MRTG Highlights
- Portable - MRTG works on most UNIX/Linux
platforms and Windows. - Perl - MRTG is written in Perl and comes with
full source. - Portable SNMP - MRTG Uses a highly portable
SNMP implementation written entirely in Perl
(thanks to Simon Leinen). There is no need to
install any external SNMP package. - Reliable Interface Identification
- Router interfaces can be identified by IP
address, description and ethernet address in
addition to the normal interface number. - Constant size Logfiles - MRTG's logfiles do NOT
grow thanks to the use of a unique data
consolidation algorithm. - Automatic Configuration - MRTG comes with a set
of configuration tools which make configuration
and setup very simple. - Performance - Time critical routines are written
in C (thanks to the initiative of Dave Rand my
Co-Author).
36MRTG Installation
- You do not need root privilege to install it.
- Create the file structure for mrtg
- Download mrtg-2_9_21_tar.gz to the above
directory
cd mkdir MRTG cd MRTG
gunzip mrtg-2_9_21_tar.gz tar xf mrtg-2_9_21_tar
37MRTG Installation (cont.)
- Read the installation guide
- http//oss.oetiker.ch/mrtg/doc/mrtg-unix-guide.en.
html
configure make make install only if you are
root
TDC597 MRTG is already installed on Linux04 at
/usr/local/mrtg-2/bin
38MRTG User Process
- Run cfgmaker to create the MRTG configuration
file (mrtg.cfg) - Run mrtg with mrtg.cfg as the parameter to
collect the data and generate graphical files - Use the HTML browser to view the data
- If the manual process works, use cronjob to run
it at a regular interval.
39cfgmaker
- Purpose creates mrtg.cfg files
- Syntax cfgmaker options community_at_router
options community_at_router ... - Options
- -global "x a" add global config entries
- --outputfile output filename default is
STDOUT
40MRTG Configuration File
Create a configuration script (confMRTG.sh)
BIN/usr/local/mrtg-2/bin WORKDIR/home/jyu/public
_html/mrtg OUTPUT/home/jyu/public_html/mrtg/mrtg.
cfg DEVICE"public_at_192.168.1.1" BIN/cfgmaker
--global "WorkDir WORKDIR" --output OUTPUT
DEVICE
Run the Configuration script
Note each MRTG configuration file (mrtg.cfg) is
for ONE network device. If you need to collect
performance data from multiple devices, you need
to create multiple CFG files (with different
names).
41mrtg (Linux Command)
- mrtg is the name of the script you have to run to
poll data and generate the graphs. Most
configuration is set through the configuration
file. - mrtg is located at /usr/local/mrtg-2/bin on
Linux04. - Syntax mrtg options mrtg.cfg
42Data Collection create a script and run it
The outputs are HTML files.
43MRTG Screen Shots
44indexmaker
- Purpose creates index files for mrtg web sites
- Syntax indexmaker
- --outputHOME/mrtg/index.html
- CFG File
- The index file has a summary graph for each
interface of the device.
45(No Transcript)
46HW06Developing a Web Site forPerformance
Analysisfrom CGI to GUI
47Summary
- Performance Analysis is primarily based on two
MIBs - ifInOcters 1.3.6.1.2.1.2.2.1.10
- ifOutOctets 1.3.6.1.2.1.2.2.1.16
- Software Development
- Home-grown tool
- Import 3rd party tools
- You need both.