Protocol Analysis in a Complex Enterprise - PowerPoint PPT Presentation

1 / 10
About This Presentation
Title:

Protocol Analysis in a Complex Enterprise

Description:

Assess the strengths and weaknesses of the financial system ... 5. Map them against local market ... and where sheer scale makes effective management difficult. ... – PowerPoint PPT presentation

Number of Views:75
Avg rating:3.0/5.0
Slides: 11
Provided by: itse122
Category:

less

Transcript and Presenter's Notes

Title: Protocol Analysis in a Complex Enterprise


1
Protocol Analysis in a Complex Enterprise April
2nd, 2008 Hansang Bae Senior VP
Citigroup SHARKFEST '08 Foothill College March
31 - April 2, 2008
2
Challenges
  • As it turns out, size does matter!
  • Citis branch network spans 5,000 locations in
    the US
  • Citis network infrastructure includes 30,000
    devices
  • 300,000 users located in over 100 countries.
  • Compliance/Security Quagmire
  • Its for your own protection, or so Im told!
  • Doing a full packet capture is difficult
  • Wireshark is the only approved protocol analyzer
    at Citi. It dislodged past market leaders.

3
Challenges (cont)
  • Capturing and Analyzing Two pieces to the same
    puzzle
  • Enormous amounts PCAP data are involved.
  • In most cases, header analysis is adequate.
  • Wireshark/WinPCAP is not well suited for this
    much volume
  • Citi uses a commercial product for packet
    capturing. Working with the vendor, it took over
    three years of development before it was deemed
    Citi-ready

4
Example One Path MTU
  • Infrastructure size makes it interesting.
  • Very difficult problem without a proper protocol
    analyzer

5
Example One (Cont)
  • In depth understanding of routers and protocols
    were required.
  • Usenet to the rescue!
  • ICMP and IP.ADDR filters were key!
  • So which side am I on in the religious debate
    about whether ICMP messages should be included in
    the ip.addr display filter?
  • ..\..\..\Traces\Consumer\CBNA\ICMPRateLimit.pcap
  • In retrospect, it was an easy problem to solve.
    Yet the sheer size made it difficult to spot.

6
Example Two Clock Drift
  • MarketData driven business complains of extreme
    delays from UK to US.
  • At first glance, application logs seem to confirm
    delays in the 200ms delays. RTT is 70ms.
  • Because its easy, lets blame the firewall and
    the network!
  • SLA tracking and further investigation of
    routers/switches gets us nowhere with problem
    resolution.
  • Our analysis shows that something is not right!

7
Example Two (Cont)
  • Due to mis-matched traffic flow, pcap data itself
    yield unreliable data.
  • For example, we would see and an ACK for a packet
    that was not yet delivered. This was traced to
    the output buffer of the SPAN on the switch.
  • The SPAN issue forced us to look a the packets
    in detail, including the data timestamp

8
Example Two (Cont)
  • Charting the pcap timestamp with the data
    timestamp showed a peculiar pattern.
  • By spotting the pattern above, we were able to
    show the vendor that their clock was drifting!

9
Lessons Learned/Feature Request
  • Picture really is worth a thousand words.
  • The two pictures above show the same event!
  • Bounce diagrams can quickly pinpoint issues.

10
Lessons Learned (Cont)
  • Allow zoom in feature from the bounce diagram for
    even easier troubleshooting.
  • The above shows the slow start in action. Its
    immediately obvious whats going on with one look
    at the chart!
  • Increase performance for TCP/IP dissection.
    Although Wiresharks support for protocols is
    impressive, most folks in the enterprise deal
    with TCP/IP problems.
Write a Comment
User Comments (0)
About PowerShow.com