The Performance Enhancement Response Team: Origins and Evolution

1
The Performance Enhancement Response Team
Origins and Evolution

• Ann Harding, HEAnet ann.harding_at_heanet.ie
• Toby Rodwell, DANTE toby.rodwell_at_dante.org.uk
• Michal Przybylski, PSNC, michalp_at_man.poznan.pl

2
Overview

• Origins
• Evolution from trial to pilot
• GÉANT the PERT in action
• Organisation
• Tools
• PERT case
• FermiLab-Renater throughput

3
Origins

• Development of high capacity WAN, NREN and campus
  backbones
• Still seeing end-to-end performance problems
• Need to look beyond the network
• Internet 2 End to End Performance Initiative
  (E2EPI)
• Initial proposal of a PERT in 2001
• A group of specialists who would be to network
  performance what CERT is to network security
• Link user and expert to solve performance
  problems
• Internet2 concept remained theoretical

4
Evolution European Trial PERT

• 2002 TF-NGN meeting discussion
• Trial PERT for GÉANT Y4
• Principle participants
• GARR (IT), TERENA, DANTE, SWITCH (CH), CESNET
  (CZ), HEAnet (IE) and UKERNA (UK)
• Entirely dependent on volunteer effort
• Format
• Mailing list hosted by SWITCH (Dec 2002)
• Free, open source issue tracker (Roundup) hosted
  by SWITCH (Mar 2004)

5
Whats in a name?

• First mail to pert-discuss stated PERT stood
  for Performance Emergency Response Team
• Second mail said the E should be Enhancement!
• It was agreed that PERT should properly be
  Performance Enhancement Response Team
• Connotations of the CERT
• Removed the misleading Emergency element from
  the title

6
Evolution GÉANT2 Pilot PERT

• November 2004 GÉANT2 Pilot PERT
• Service Activity 3 - Performance and Allocated
  Capacity for End-users (PACE)
• New e-mail address for reporting PERT cases
• Roster of duty Case Managers
• Duty Case Manager spending to 2/3 hours per day
  on open issues
• PERT Wiki
• Diary, to track successes and failures of the
  pilot
• Preliminary knowledgebase

7
GÉANT2 Production PERT
8
GÉANT2 Production PERT

• Who?
• PERT customers
• PERT staff
• What?
• Any academic networked system performance problem
  
• Guaranteed investigation of problems
• Consultancy service
• When?
• March 2005 GÉANT production PERT

9
Production PERT - Organisation

• PERT Participants
• PERT managers
• Full Time Case Managers
• Subject Matter Experts
• PERT customer
• PERT forum moderators

10
Production PERT - Tools

• PERT Ticketing System (PTS)
• Ticket management notification
• E-mail and Jabber integration
• PERT Diary
• PERT Knowledgebase (KB)
• Wiki-based public knowledgebase
• Organised, categorised PERT knowledge
• Updatable by any PERT member
• PERT Public Forum

11
Production PERT PTS
12
PERT Case - Fermilab to RenaterThe Problem

• Case example from Pilot PERT
• Problem observed transferring files from fnal.gov
  (FermiLab) to a machine in Strasbourg
• The data consisted of many 15MByte files,
  totalling a few hundred gigabytes
• The transferring application was "rsync"
• The bottleneck links were 100Mbps, but the
  achieved transfer rate was typically 5Mbps

13
PERT Case - FermiLab to Renater The Tests

• Test machines in similar locations
• Use web100 tools
• Memory-memory routinely achieved 90Mbps, using
  nttcp
• limited system disk i/o capability on the
  receiving machine
• Alternative receiving test machine
• long path, fast machines on both ends
• data via ssh via TCP slower than accountable by
  crypto overhead
• highlighted the ssh/ssl buffer limitations

14
PERT Case - FermiLab to Renater Conclusions

• The FermiLab sender-side rsync server had small
  TCP buffers
• The Renater receive-side TCP buffers were too
  small
• On Linux, use auto-buffer-tuning on send and
  receive
• You've got to have at least 8MBytes of buffer
  space available for 1xGigE across an ocean
• Final throughput
• memory-memory 429Mbits/sec
• disk-disk 20Mbytes/sec (160Mbits/sec)

15
Conclusions

• Wide problem domain
• SMEs crucial to success
• Areas of networking, applications, protocols,
  systems
• Only service of this kind in the world!
• Each new case enhances the service
• Here to help

16
Acknowledgements

• The authors would like to acknowledge the
  pioneering work done in the PERT by Simon Leinen
  (SWITCH), Victor Reijs (HEAnet) and Sven Ubik
  (CESNET), and Larry Dunn (Cisco) for his analysis
  of the FermiLab-Renater case.