IPFIX/NetFlow Mediator Implementation and Test Results

1
IPFIX/NetFlow Mediator Implementationand Test
Results
2007/3/22Daisuke Matsubara (Hitachi), Atsushi
Kobayashi (NTT)
2
Overview

• Background
• IPFIX Mediator concept and draft was introduced
  in 65th and 67th IETF meeting. (draft-kobayashi-ip
  fix-mediator-01.txt)
• Mediator allows us to monitor the overview
  traffic such as traffic matrix, and retrieve
  specific flow records anytime.
• routers are able to simply export flows without
  aggregation even in large scale network, with
  minimum sampling rate.
• Actual prototype implementation of IPFIX/NetFlow
  Mediator was done by NTT/Hitachi.
• Testing of the prototype was conducted using MAWI
  traffic data.
• Objective of this presentation
• Introduce implementation of IPFIX/NetFlow
  mediator to show feasibility of the concept and
  clarify its importance.
• prototype system of IPFIX/NetFlow mediator.
• test results of aggregation and storing process.

3
Network monitoring without mediator
To monitor the routers traffic matrix, we should
collect the entire flow information to one
server. - Total traffic 440Gbps - 200 routers
in a network - 220f/s per router (1/1000
sampling) - Total flow rate 43kf/s
Monitoring Server
1 Monitoring Server (Maximum of 10kf/s)
43kf/s
Router
Router
100Gbps Routers 200
4
Network monitoring with mediator
Mediator stores and aggregates flow information
from 20 routers.
Monitoring Server
1 Monitoring Server (Maximum of 10kf/s)
43 -gt 8.17kf/s (aggregated flows)
Mediator
Mediator
10 NW domains 10 Mediators
Router
Router
20 edge routers per domain
100Gbps Routers 200
5
Aggregation Ratio Dependency

• Compare aggregation ratio
• 3 different traffic samples
• Aggregation Timer 5s - 180s
• Sampling Rate 1/1 - 1/1024

To utilize the flexibility of aggregation, we
need IPFIX mediator.
6
Mediator Architecture
7
Mediator Prototype Overview

• Features
• NetFlow ver. 5, ver.9 (IPv4/v6)
• Stores flow information in NetFlow format.
• Aggregates flow information
• Any-port
• DstHost
• BGPnexthop
• MPLS
• System Specification
• Implemented in C, Linux OS
• NetFlow ver. 5, ver.9 (IPv4/v6)

sum
IN_BYTES
IN_BYTES
sum
IN_PKTS
IN_PKTS
key
PROTOCOL
PROTOCOL
discard
INPUT_SNMP
IPV4_DST_ADDR
key
IPV4_DST_ADDR
key
SRC_AS
SRC_AS
EXP_IPV4_ADDR
append
append
AVE_ACTIVE_TIME
8
Performance Test Result
traffic data MAWI(200602231400.dump)
9
Conclusion

• IPFIX/NetFlow mediator is an essential component
  for realizing scalable real-time monitoring
  system in a large-scale network.
• Aggregation ratio varies depending on flow
  numbers and aggregation methods.
• We will proceed to study actual deployment of
  mediators in an operating network environment.
• We invite discussions regarding key
  standardization issues such as exporter
  information for IPFIX Mediators.
• Next step, we will try to refine the IPFIX
  Mediator draft and draw up this experimental
  approach.

10
Additional Function?

• Modify and create new information elements.
• For MPLS NW, append VPN id instead of label
  value.
• For simple 5-tuple flows, append BGP next-hop or
  AS number.
• Handle the exporter information.
• To notify the exporter information, we already
  introduced the new templates in IETF67th.
• In some case of exchange the traffic information
  between the different domain, it intentionally
  dont notify exporter information to hide the
  topology.
• In particular, a proxy needs to hide the related
  exporter information, such as next-hop and
  ifindex in the flow.
• Anonymize private parts of the flow.
• For example, DST address or SRC address should be
  anonymized in some case of situation.
• To monitor the traffic trend, it can be
  anonymized it. It prevent from security violation
  accident.