Benchmarking Results from the Tycho Performance Tests

1
Benchmarking Results from the Tycho Performance
Tests
Matthew Grove DSG Seminar 27rd January 2006
2
Outline

• Introduction.
• Changes made to the Tycho design from last time
  (June 2005).
• Example Tycho setup.
• Tycho benchmark motivations and methodology.
• Some performance results.
• Summary.

3
Introduction

• Tycho is a reference implementation of a combined
  extensible wide-area messaging framework with a
  built in distributed registry.
• Tycho is deigned to reuse existing infrastructure
  where ever possible.
• Tycho aims to simplify and speed application
  development by freeing developers from the need
  to use a combination of middleware to provide
  service discovery and messaging.

4
Whats new for Tycho

• We have changed the name from jGMA to Tycho, this
  lets us focus on the presenting the strengths of
  Tycho, removing the dependency to explain our
  motivation in terms of GMA and the Grid.
• The architecture has remained the same but we
  have completed the initial implementation.

5
Tychos Architecture
6
Example Tycho Setup
7
Implementation Changes

• There are 24 items in the change log, the API has
  remained the same, highlights are
• The IRC VR-connector has been completed.
• Extended the HTTP handler to provide VR boot
  strapping using a known web site.
• Support for JDBC data-stores (like HSQLDB).
• Added support for SQL keywords LIKE and
  LIMIT.
• Added caching for inter-VR responses.

8
Tycho Benchmarks

• We performed two rounds of benchmarking
• Message Passing These tests measured the
  performance of inter-client and inter-mediator
  communications for Tycho and NaradaBrokering.
• Virtual Registry These tests measured the
  performance of the VR (query routing and
  processing) in a range of different
  configurations
• These results were compared to other tests which
  measured performance of Hawkeye, R-GMA and MDS2
  and 4.

9
Message Passing Tests (A)

• We measured the latency of communication for LAN
  and simulated WAN messaging between two clients
  varying message size (ping-pong tests).
• Then we measured the affect on scalability of
  message latency using many producers publishing
  to a single consumer and single consumer
  consuming from multiple producers.
• We used Starbug to run the tests.

10
Test A - Configuration
11
Ping-Pong - Latency
Latency vs Message Length
12
Ping-Pong - Bandwidth
Bandwidth vs Message Length
13
Scalability Results - Producers
Bandwidth vs Message Length
14
Scalability Results - Consumers
Bandwidth vs Message Length
15
Message Passing Tests - Summary

• Tycho achieves comparable or lower latency than
  NaradaBrokering in all of these tests.
• The performance of both systems does not drop off
  after a producer or consumer has become
  saturated.
• NaradaBrokering needs the JVM heap size to be
  increased as the number of clients increases (due
  to internal buffers)
• Tycho used the default heap for all of the tests.

16
Virtual Registry Tests (B)

• We used 4 different configurations to measure the
  performance of the Tycho VR.
• For each test we varied one or more aspect of the
  configuration and measured the effect on client
  query latency
• Number of records in a mediator.
• Number of simultaneous client queries.
• Numbers of mediators in the VR.
• Different VR-interconnects between mediators.
• In order to make some comparisons between Tycho
  and some other systems we attempted to reproduce
  a test used in a related paper.

17
Configuration
 
18
Variables
 
Test Parameter Matrix the components that are
varied in each test are highlighted. S1 queries
select a single record, S2 queries select every
record.

• The comparison tests with the other systems used
  the test 2 configuration.

19
Number of Records
 
Response Time vs Number of Records
20
Simultaneous Client Queries
 
Response Time vs Number of Clients
21
Number of Mediators
 
Response Time vs Number of Mediators
22
VR-Interconnects
 
Response Time vs Number of Mediators
23
Virtual Registry Tests - Summary

• The Simple store performs best for small numbers
  of records, but HSQLDB and MySQL scale better
  with response size.
• The response time increases linearly with number
  of clients.
• For routing queries in the VR, IRC has better
  performance than HTTP.
• As response size increases HTTP performs much
  better than IRC, this is due to the IRC code
  having to send multiple messages.

24
Comparison Tests

• The tests measure the impact of increasing the
  number of clients simultaneously querying the
  registry.
• We had to use 5 compute nodes instead of 8 to
  make the test as close a match as possible.
• Each system has different sized responses which
  we matched by adding padding to Tycho registry
  information.
• The main difference is CPU (2.8Ghz vs 1.1Ghz).

25
Comparison Tests (2K)
Response Time vs Number of Clients (2Kbyte
Responses)
26
Comparison Tests (10K)
Response Time vs Number of Clients (10Kbyte
Responses)
27
Comparison Tests (19K)
Response Time vs Number of Clients (19Kbyte
Responses)
28
VR Comparison Tests Summary

• Tycho shows linear increase in response times
  regardless of message size and number of clients.
• Tycho had lower response times than R-GMA, MDS4
  and MDS2 without caching.
• MDS2 with caching is basically OpenLDAP - which
  is mature and performs well.
• Hawkeye performs better than Tycho for less than
  100 clients, but it does not scale well with
  larger number of clients.

29
Future Work

• Some performance improvements
• Caching of local mediator queries to reduce
  response times.
• The Simple store could be improved by making it
  more intelligent.
• We could use a hybrid VR-interconnect to use IRC
  for query routing and HTTP for transporting large
  responses.
• More functionality can be added to provide
  services that are more advanced
• HTTPS and SSL Sockets for secure comms.
• Support for transactions.
• Producers and consumers for computational
  steering.

30
Links

• Project Web page
• http//dsg.port.ac.uk/projects/tycho/
• The DSG Web page
• http//dsg.port.ac.uk/