Inter-Peer NOC Communication

1
Inter-Peer NOC Communication

• Mike Hughes
• mike_at_linx.net

2
Scene Setting Straw Poll

• Who here in this room does peering?

3
Scene Setting Straw Poll

• Who here in this room does peering?
• Have you ever had issues resolving problems with
  your peerings?
• Difficulties contacting peers, finding the right
  contact, communication problems?

4
Scene Setting Straw Poll

• Who here in this room does peering?
• Have you ever had issues resolving problems with
  your peerings?
• Do you maintain a local db of contacts?
• Why? Issues with freshness of data?

5
Scene Setting Straw Poll

• Who here in this room does peering?
• Have you ever had issues resolving problems with
  your peerings?
• Do you maintain a local db of contacts?
• When a peer needs to talk to you, where does
  their call/email arrive?
• Main NOC contact? Dedicated peering contact?
  Customer Care?

6
Scene Setting Straw Poll

• Who here in this room does peering?
• Have you ever had issues resolving problems with
  your peerings?
• Do you maintain a local db of contacts?
• When a peer needs to talk to you, where does
  their call/email arrive?
• Some names have been changed to protect the
  innocent and guilty

7
Why do you go peering?

• Long term money savings
• Less Transit
• Lower latency, better performance
• Traffic Control
• Diversity, Reliability
• Presence
• and so on

8
Wheres the problem?

• Poor inter-peer communication seems to be common
• Friendly IX operator called in to mediate
• Communication hitting the wrong place
• Customer NOCs
• IX Operator
• IP address maintainer (e.g. whois contact)

9
Identifying the right contact

• Sources of information
• Whois queries to databases
• IXP-maintained NOC and Peering contact db
• Internal databases
• Third-party voluntary databases
• http//puck.nether.net/netops list
• peeringdb.com
• All above are vulnerable to information rot

10
How to drive RIPEdb/RA, etc

• Some really subtle differences in the
  implementations
• RIPE expects AS before an AS number!
• Which contacts are useful
• Which objects to look up
• Like the Peer ASN, not the Peer IP address!
• Why cant ASN be logged in adjacency changes on
  routers?
• This seems to drive IP-based lookups

11
Drive the Data Sources Properly!

• Example using WHOIS queries
• Oh, I have an outage on WAIX, Ill look up the
  IP address

whois -h whois.arin.net 198.32.212.11less Org
Name Exchange Point Blocks RTechHandle
WM110-ARIN RTechName Manning, Bill
RTechPhone 1-310-322-8102 RTechEmail
bmanning_at_karoshi.com
12
Bad Data Enters the System

• Okay, Ill phone Bill Manning
• But all Bill did was give WAIX some v4 space
• Bill doesnt run WAIX, and isnt an operational
  contact for WAIX
• So, Bill either ignores your voicemail, or tells
  you to call someone else
• Whatever its added delay, increased
  frustration its how not to do it

13
Driving Whois Properly

• Always lookup the PEER ASN
• Not the IP address!
• Its a BGP problem, we use ASNs in BGP

whois -h whois.ra.net AS3856less aut-num
AS3856 as-name UNSPECIFIED descr Packet
Clearing House www.pch.net admin-c
Bill Woodcock tech-c Bill Woodcock remarks
peering_at_pch.net, 1 866 BGP PEER
14
Driving Whois Properly

• Always lookup the PEER ASN
• Not the IP address!
• Its a BGP problem, we use ASNs in BGP

15
So youve found the contact

• How do they respond to you?
• Confusing recursive call trees?
• Recalcitrant ticketing systems?
• First-line NOC Is it switched on?
• Youre not a customer, go away
• Once negotiated, peering is an engineering
  relationship
• So backbone ops, not customer care

16
Expectations of Peer Contacts

• Choose your points of contact carefully
• Big problems with
• Whats peering/BGP/WAIX?
• Are you a customer?
• Whats your circuit ID?
• Go away, you arent a customer
• All serious no-nos be nice to your peers!

17
PCH INOC-DBA Phones

• PCH operate a dial by ASN NOC hotline system
• They run the SIP registry/proxy
• Bring your own SIP compliant phone
• The idea is that it should get through to someone
  clueful
• No call-trees, no music-on-hold
• http//www.pch.net/inoc-dba/

18
Suggested Role Contacts

• Peering_at_
• For setting up new peerings, changing existing
  ones, no 24x7 expectation
• Shouldnt go to exclusively to sales_at_ -)
• NOC_at_
• Reaches your 24x7 NOC, which is either BGP
  friendly and has enable, or knows when, how and
  where to escalate
• Support_at_
• Is generally your customer-care/call center

19
Getting the message across

• Okay, so youve made contact
• Now, make your point
• Provide the peer with useful information
• Start with the subject line
• Be informative, who, when, what
• Messages like Help and Peering down arent
  helpful

20
How not to do it
-----Original Message----- From Joe Schmoe
ltschmoe_at_noc.foo.comgt Sent Wednesday, January
25, 2006 541 PM Subject Maintenance
Notification Dear Peers,

• Where? How does it affect me?
• All detail buried in wordy message body
• When? No TZ stamp!
• Help me handle my huge NOC inbox!

21
Example Useful Subject Headers

• AS7132s preferred subject line format
• ltIX locationgt -
• ltpeer writing to/ASNgt -
• ltpeer writing from/ASNgt -
• ltwhat is the issuegt -
• ltdate of initial correspondencegt -
• lttime of initial messagegt
• Example subject line  
• Equinix-Ashburn - RCN/6079 - SBC/7132 - new
  session turn-up - 29- Mar-06 - 945 am EST
• Thanks to Ren Provo

22
Look clueful
Subject Traffic Drop Dear Peer, We suddenly
noticed a 300Mb drop in traffic on our connection
to the PIE-IX. Can you investigate, and help us
find where the traffic has gone? Regards,

• What does this say about your peer?
• Dont you think they look silly?
• Run tools to help you answer these questions
  yourself
• Netflow, MAC accounting, etc.

23
How to escalate

• Check your equipment first
• Ask your peer - Whats up?
• Often you can resolve a problem bi-laterally
• Go to the IX only if you need to
• Not all IX operators can provide a 24x7 contact
• When to escalate a customer fault
• Dont stonewall customer reports
• Dont point them to the IX operator
• Co-ordinate directly with your peers

24
How the IXP Op can help

• Provide an up-to-date list of IX participants and
  their NOC/Peering contact information
• Usually password protected
• Help break comms deadlock
• Help fix dead ends
• Otherwise, they can only help with physical
  problems
• link down, packet loss, broken cables, packet
  corruption to all destinations connected to the
  IXP

25
In Summary

• Keep your own information up to date
• Whois db objects, third party dbs
• Make sure your peering and NOC contacts are
  appropriate
• No-one likes call-trees and holding
• Find the right contacts at your peers
• Be nice to your peers!

26
Thanks

• mike_at_linx.net