Operational Multicast Network Management

1
Operational Multicast Network Management

• Radhika Malpani
• Hewlett Packard Laboratories
• radhika_at_hpl.hp.com

2
Agenda

• Why deploy multicast?
• Operational multicast management
• Mmon
• Management scenarios using mmon
• Implementation details
• Deployment experiences

3
Why deploy Multicast Service?
Unicast
Load on network Load on server increases per
user
Not Scalable
4
Why deploy Multicast Service?
Multicast
Load on network Load on server may have
little to no increase per user
Scalable
5
Why deploy Multicast Service?

• Emerging applications best served by multicast
• Audio/Video streaming webcasting,
• Electronic software/data distribution
• Corporate announcements, communications across
  multiple locations
• Real world deployments ToysRUs, Promus Hotels,
  FedEx, Ford, GM,Cisco, Intel, ...

6
Factors Inhibiting Multicast Deployment

• Chicken and egg problem
• Fear of effects on networks
• absence of good mgmt tools
• Absence of ROI and billing models
• Security and access control
• Absence of well defined peering policies
• Success of pseudo multicast solutions

7
Management Concerns

• Fear effects of multicast on existing network
• router mis-configurations
• potential defects, instability of implementations
• increased CPU load
• increased traffic
• Control and security

Ongoing monitoring would alleviate these concerns
8
But ...

• No multicast-aware network management systems
• Public domain tools require protocol knowledge
• Few people understand multicast technology
• especially in NOCs (Network Operations Center)

9
Network Management Roles
Operations
Help desk

• Customer support

• First Level Operators
• Routine status and traffic monitoring
• Fault detection and isolation

Network Engineering

• Initial deployment and configuration
• capacity planning

• Subject Matter Experts
• Fault diagnosis

10
Existing Multicast Tools

• What is missing
• Tools for non-multicast savvy operators
• easy to use
• proactive monitoring of multicast networks
• multicast fault detection and isolation

• Deployment and config tools
• mgen, chariot, mconfig,
• Specialized monitoring tools
• rtpmon, mhealth, mlisten,
• Diagnostic tools
• mtrace, mrinfo, mstat, mview, ...
• require multicast expertise

11
Mmon

• Solution for NOC operators
• minimizes multicast expertise needed
• Integrated with HP Openview
• Automated discovery
• Ongoing status and availability monitoring
• Event-driven exception reporting
• On demand and continuous traffic monitoring
• tables,graphs,historical data
• group forwarding trees, receiver distributions
• Fault isolation, first level diagnostics

12
Brief IP Multicast Primer

• IGMP
• used by host to inform local router of groups it
  wants to receive
• Router sends host membership queries
• Hosts respond with host membership reports
• Routers use multicast routing protocols to
  construct delivery trees
• Dense Mode protocols (e.g. PIM-DM, MOSPF,DVMRP)
• Sparse Mode protocols (e.g. CBT, PIM-SM)

13
PIM-DM

• Flood and prune protocol
• data from source flooded on all links
• router does RPF check, and forwards out all other
  interfaces
• routers with no receivers send prunes upstream
• graft messages for reattaching pruned branches
• state at every router for every (S,G)
• periodic flooding of data

14
PIM-SM

• Explicit join protocol
• routers with receivers explicitly join shared
  delivery trees
• joins sent towards RP to create shared tree
• sources send data to RP, RP forwards down shared
  tree
• less state at routers
• less optimal path
• use src-specific shortest path trees

15
Agenda

• Why deploy multicast?
• Operational multicast management
• Mmon
• Management scenarios using mmon
• Implementation details
• Deployment experiences

16
Scenarios

• Browsing How are things?
• Monitoring a special event
• "Mike can't see the training class"
• "Shut it down!"

17
Browsing How are things?

• Automatic discovery of multicast routers
• includes workstation routers, tunnels
• Ongoing status monitoring
• multicast status separate from unicast
• Event generation, Alerts
• Overall traffic statistics
• Is anyone multicasting?
• What multicast traffic flows here?
• Impact on routers

18
Browsing Multicast Topology
19
Browsing Multicast Topology
20
Browsing Overall Traffic Statistics

• Per interface
• kbits per second
• capacity utilization
• incoming and outgoing
• Sorted by incoming utilization

21
Browsing Impact on Router
22
Monitor special event

• Pre-event check facility
• active testing
• select source and receiver routers
• test reachability
• Monitor group traffic flow and distribution of
  receivers

23
Monitor special event
Forwarding tree
24
Scenario "Mike Can't See the Training Class"

• Customer Support Mike is not receiving "Desktop
  Training Channel 26"
• View the forwarding tree for the group "Channel
  26"
• If forwarding tree in place continue diagnosis
• If forwarding tree incomplete check for group
  join
• Check group traffic near Mike
• If traffic flowing client problem, dispatch to
  Mike's local support
• If no traffic dispatch to appropriate support

25
Mike Can't See the Training Class
Im Mike, and I cant see my _at_!! Training
Class on Channel 26!
Forwarding Tree for group "Training Channel 26"
Customer Support, may I help you?
26
Mike Can't See the Training Class
I can tell that the training class is currently
transmitting in your local area. Ill dispatch
Gita, your local support person, to help you at
once!
Group traffic statistics from router of Mike's
subnet
27
Scenario "Shut it Down!"

• Event Alert Multicast traffic at limit on
  interface 10.128.132.41 - Building 4 Floor 2
• View overall traffic statistics
• selection from stats automatically highlights
  associated router on map
• Query router detailed traffic statistics to see
  source of load a single offending group,source
• Dispatch to router control to block the traffic

28
Shut it Down!
Multicast traffic on interface 10.128.132.41 is
above 9 of link capacity
29
Shut it Down!

• Multicast traffic is high on a router interface
• Operator noticed via ongoing traffic monitoring
  or...
• Threshold was set and triggered an event/alert
• Operator requests traffic stats detail for that
  router.

30
Shut it Down!
Offending multicast source,group producing 4,000
kbps vs. others at only 64 or 8 kbps. Wrong
compression in use! Shut down the offending MPEG2
stream!
31
Agenda

• Why deploy multicast?
• Operational multicast management
• Mmon
• Management scenarios using mmon
• Implementation details
• Deployment experiences

32
Technologies

• IGMP protocol
• dvmrp_ask_neighbors, dvmrp_neighbors
• SNMP
• MIB-II, IGMP MIB, IPMROUTE MIB, PIM MIB
• RTCP
• not always available (RTCP blocked, non-rtp apps
• rtp MIB
• MRM (multicast reachability monitor)

33
Multicast Reachability Monitor

• Agent based architecture
• test agents
• mrm manager
• Manager configures agents as srcs/rcvrs
• Src group, pkt size, pkt interval, duration,
  source
• Rcvr group, src, expected interval, loss
  threshold, min/max reporting delay

34
MRM Uses

• As-needed tests of infrastructure
• e.g. PA Check before transmission of important
  events (synthetic traffic)
• Ongoing infrastructure availability monitoring
• agents listen and report on low-bit rate
  transmissions all the time (synthetic traffic)
• What-if network engineering, capacity planning
• turn on sources with different loads, from
  different locations (synthetic traffic)

35
Experience

• Implementation experience
• mibs still evolving, hence unstable
• problem with igmp
• Deployment experience
• installed at several customer test sites
• Not just operational tool but also
• routing protocol test tool
• deployment configuration test tool

36
More work to be done...

• More work on automatic fault detection and
  isolation
• Integration of application level information
• Integrate end-user perceived QOS
• Support for inter-domain protocols
• Support for multicast aware switches

37
Operational Multicast Management Summary

• Huge demand for multicast network management
• requests for mmon from over 30 enterprises, ISPs,
  and research institutions
• Mmon currently installed at several customer
  sites
• Deployed on HP Intranet, Cisco Engineering net,
  and one other mission critical network
• 2 more deployments scheduled
• In process of collecting feedback and usage data

38
Operational Multicast Management Summary

• Huge demand for multicast network management
• requests for mmon from over 30 enterprises, ISPs,
  and research institutions
• Mmon currently installed at several customer
  sites
• Deployed on HP Intranet, Cisco Engineering net,
  and one other mission critical network
• 2 more deployments scheduled
• In process of collecting feedback and usage data

39
The End
ABC LABS