Ubiquitous Instrumentation

1
Ubiquitous Instrumentation

• Richard Mortier
• MSR Cambridge
• mort_at_microsoft.com

2
Measuring networks

• Active
• traceroute/ping/etc (ICMP)
• SNMP MIBs
• Non-critical functionality ? buggy
• Passive
• NetFlow, IPFIX, sFlow
• Port spanning, VLAN spanning
• Router/switch only ? poor visibility, scalability

3
Claims

• Routers are just specialized hosts
• Hardware, protocols, configuration
• Hosts are part of the network as well
• Transmit, receive, forward data

4
Claims

• Routers are just specialized hosts
• Hardware, protocols, configuration
• Hosts are part of the network as well
• Transmit, receive, forward data
• The strong distinction that has grown between
  them makes understanding network behaviour
  difficult
• Traffic is opaque to routers (and becoming more
  so)
• Network is black box service to hosts (and
  becoming more so)
• Hosts provide inputs
• we should use them to understand demands
• Routers provide resource
• their configuration implements constraints

5
Networking measurements

• Host-based measurement Anemone
• Instrumented stack, kernel structures
• Per-EXE per-packet live network stats
• Unify flows with routing topology
• Distributed query system Seaweed
• Access to Anemone flow/packet tables
• Built over Pastry, highly scalable (106 nodes)

6
Measure for measure

• Ubiquitous network measurement
• Undercarriage of the Knowledge Plane
• Infrastructure for autonomic-foo?
• Concretely
• Combining host network data SeaStar
• Single trust domain how to expand?
• User-visible diagnosis Constellation
• What do users want to know about the network?

7
Measure for measure

• Data gathering and management
• Existing tools primitive tend toward manual
  analysis
• Need performant, type-safe, distributed
  processing
• Theoretical aspects of temporal, spatial sampling
• Unifying many different datasets
• Routing and configuration data
• Pcap and netflow and SNMP and data
• Mining structure, relationships from data
• Machine learning techniques seem a very rich vein
• Robust automated processing techniques
• Distributed trust, provenance, privacy