The DDS Benchmarking Environment

1
The DDS Benchmarking Environment
James Edmondson jedmondson_at_gmail.com
Vanderbilt University Nashville, TN
2
Topics

• What is the DBE?
• What technologies does the DBE use?
• How does the DBE setup the environment?
• What problems had to be dealt with?
• What features will be added?
• Demonstration
• Conclusion and Wrap Up

3
What is the DBE?

• DDS Benchmarking Environment
• Sets up the testing environment and framework
• Configures and launches implementation specific
  services
• Launches Publishers and Subscribers
• Coordinates interaction, logging info, and stats
  collection
• Allows for easily scaling the number of
  participants
• Sets up the statistics for collection by Bagatel

Publishers
Throughput
Networking
Subscribers
4
What is the DBE?

• May be used to test other middleware or
  technologies
• Focus is on documented, reusable resources

Versatility
Dependability
Publishers
Throughput
Networking
Subscribers
5
What does the DBE do?
6
What does the DBE do?
7
What does the DBE do?
8
What does the DBE do?
9
What does the DBE do?
10
What does the DBE do?
Start the Subscribers
11
What does the DBE do?
12
What does the DBE do?
13
What does the DBE do?
14
What does the DBE do?
15
What does the DBE do?
Start the Publisher
16
What does the DBE do?
17
What does the DBE do?
18
What technologies does the DBE use?

• Scripts are written in Perl and use PerlAce
  module
• Remote execution is accomplished via SSH
• Logs, stats and test info are stored on a NFS
  partition
• All tests are currently ran on Fedora Core Linux
  machines
• Configured to test NDDS, TAO DDS, and Splice
• Synchronization between tests is maintained by
  using lockfiles

DBE
Linux
SSH
NFS
Perl
lockfiles
19
How does the DBE setup the environment?

• Most path information is updated using the
  /.ssh/environment file
• Some custom environment variables are set by DBE
  scripts on individual clients
• Individual Test IDs are assigned to each test
• Directories are created on the NFS partition for
  stats and logs

DBE
20
How does the DBE setup the environment?

• Most path information is updated using the
  /.ssh/environment file
• Some custom environment variables are set by DBE
  scripts on individual clients
• Individual Test IDs are assigned to each test
• Directories are created on the NFS partition for
  stats and logs

DBE
21
How does the DBE setup the environment?

• Most path information is updated using the
  /.ssh/environment file
• Some custom environment variables are set by DBE
  scripts on individual clients
• Individual Test IDs are assigned to each test
• Directories are created on the NFS partition for
  stats and logs

DBE
Change ENV
22
How does the DBE setup the environment?

• Most path information is updated using the
  /.ssh/environment file
• Some custom environment variables are set by DBE
  scripts on individual clients
• Individual Test IDs are assigned to each test
• Directories are created on the NFS partition for
  stats and logs

DBE
Change ENV
23
How does the DBE setup the environment?

• Most path information is updated using the
  /.ssh/environment file
• Some custom environment variables are set by DBE
  scripts on individual clients
• Individual Test IDs are assigned to each test
• Directories are created on the NFS partition for
  stats and logs

DBE
Change ENV
Generate ID
24
How does the DBE setup the environment?

• Most path information is updated using the
  /.ssh/environment file
• Some custom environment variables are set by DBE
  scripts on individual clients
• Individual Test IDs are assigned to each test
• Directories are created on the NFS partition for
  stats and logs

DBE
Change ENV
Generate ID
25
How does the DBE setup the environment?

• Most path information is updated using the
  /.ssh/environment file
• Some custom environment variables are set by DBE
  scripts on individual clients
• Individual Test IDs are assigned to each test
• Directories are created on the NFS partition for
  stats and logs

DBE
Change ENV
Generate ID
Create Directories
26
How does the DBE setup the environment?

• Publishers and subscribers are notified of test
  settings
• Result file locations
• QoS settings on a per subscriber basis
• Network settings
• Test specific settings (number of messages to
  send, etc.)

Pass Test Info
DBE
Repos
Publishers
Subscribers
27
What problems had to be dealt with?

• Synchronization amongst publishers and
  subscribers

Subscriber
Next Test
Subscriber
Wait
Publisher

• Lack of a terminal with remote command execution
  with SSH
• TCP and UDP buffer sizes were too small

Publisher
Publisher
Buffer
Buffer
28
What problems had to be dealt with?

• Flock() doesnt work as expected with stateless
  NFS

flock(myfile)
touch(myfile.lock)

• Shared libraries are referenced locally
• Block buffering of stdout

block buffering
stdout
file
stderr
file
immediate
29
What problems had to be dealt with?

• More problems on request )

30
What features will be added?

• New tests will require new features
• One such feature is per node scripting of
  subscribers and publishers
• Feature will allow dynamic starting and stopping
  of participants

DBE
31
What features will be added?

• New tests will require new features
• One such feature is per node scripting of
  subscribers and publishers
• Feature will allow dynamic starting and stopping
  of participants

DBE
Create Publisher
32
What features will be added?

• New tests will require new features
• One such feature is per node scripting of
  subscribers and publishers
• Feature will allow dynamic starting and stopping
  of participants

DBE
Create Publisher
33
What features will be added?

• New tests will require new features
• One such feature is per node scripting of
  subscribers and publishers
• Feature will allow dynamic starting and stopping
  of participants

DBE
Create Publisher
Publisher
34
What features will be added?

• New tests will require new features
• One such feature is per node scripting of
  subscribers and publishers
• Feature will allow dynamic starting and stopping
  of participants

DBE
Publisher
35
What features will be added?

• New tests will require new features
• One such feature is per node scripting of
  subscribers and publishers
• Feature will allow dynamic starting and stopping
  of participants

DBE
Create Subscriber
Publisher
36
What features will be added?

• New tests will require new features
• One such feature is per node scripting of
  subscribers and publishers
• Feature will allow dynamic starting and stopping
  of participants

DBE
Create Subscriber
Publisher
37
What features will be added?

• New tests will require new features
• One such feature is per node scripting of
  subscribers and publishers
• Feature will allow dynamic starting and stopping
  of participants

DBE
Create Subscriber
Publisher
Subscriber
38
What features will be added?

• New tests will require new features
• One such feature is per node scripting of
  subscribers and publishers
• Feature will allow dynamic starting and stopping
  of participants

DBE
Publisher
Subscriber
39
What features will be added?

• New tests will require new features
• One such feature is per node scripting of
  subscribers and publishers
• Feature will allow dynamic starting and stopping
  of participants

DBE
Create Subscriber
Publisher
Subscriber
40
What features will be added?

• New tests will require new features
• One such feature is per node scripting of
  subscribers and publishers
• Feature will allow dynamic starting and stopping
  of participants

DBE
Create Subscriber
Publisher
Subscriber
41
What features will be added?

• New tests will require new features
• One such feature is per node scripting of
  subscribers and publishers
• Feature will allow dynamic starting and stopping
  of participants

DBE
Create Subscriber
Publisher
Subscriber
Subscriber
42
What features will be added?

• New tests will require new features
• One such feature is per node scripting of
  subscribers and publishers
• Feature will allow dynamic starting and stopping
  of participants

DBE
Publisher
Subscriber
Subscriber
43
What features will be added?

• New tests will require new features
• One such feature is per node scripting of
  subscribers and publishers
• Feature will allow dynamic starting and stopping
  of participants

DBE
Kill Subscriber 1
Publisher
Subscriber
Subscriber
44
What features will be added?

• New tests will require new features
• One such feature is per node scripting of
  subscribers and publishers
• Feature will allow dynamic starting and stopping
  of participants

DBE
Kill Subscriber 1
Publisher
Subscriber
Subscriber
45
What features will be added?

• New tests will require new features
• One such feature is per node scripting of
  subscribers and publishers
• Feature will allow dynamic starting and stopping
  of participants

DBE
Kill Subscriber 1
Publisher
Subscriber
46
What features will be added?

• New tests will require new features
• One such feature is per node scripting of
  subscribers and publishers
• Feature will allow dynamic starting and stopping
  of participants

DBE
Kill Subscriber 1
Publisher
Subscriber
47
What features will be added?

• New tests will require new features
• One such feature is per node scripting of
  subscribers and publishers
• Feature will allow dynamic starting and stopping
  of participants

DBE
Publisher
Subscriber
48
What other additions are planned?

• The ability to launch logging services
• Memory and CPU usage will need to be recorded
• The ability to test other middleware and
  applications

49
Demonstration

• Primer messages 100
• Stats messages 1000
• Test type throughput
• Implementation NDDS

50
Conclusion and Wrap Up

• The DBE is an excellent suite of tools for
  automating DDS testing on a networked environment
• New test types will be added to gain further
  insight into the DDS implementations
• Many new features will also be added to better
  reveal problems and benefits of the various DDS
  products
• We would like to thank our AFRL and Vanderbilt
  Sponsors for their continued support