Interoperability Testing

1
Interoperability Testing

2
Work done so far

• WSDL subgroup
• Generated Web Service Description with aim for
  maximum interoperability between various SOAP
  stacks
• WSDL tested against tooling supplied by
• JAX-RPC RI
• AXIS
• .NET
• Basic and informal interop tests
• .NET lt-gt AXIS

3
WSDL subgroup (contd)

• Released a subgroup report (Andre)
• Philosophy followed
• e.g. choosing document/literal style
• Extension mechanism
• Found problems/concerns
• Raised bug reports against the stacks
• AXIS begun work on all of them
• .NET reported fixes were in progress
• JAX-RPC RI has not responded yet
• Fixes/workarounds reflected in the WSDL and
  interfaces

4
Interoperabilty

• WSDL subgroup work
• Allows various stacks to generate code from WSDL
  definitions
• Ensures interoperability on SOAP level
• Need for interoperability on WSRP protocol level
• cross-vendor testing
• Compliance testing ?

5
Cross-vendor testing

• Release open source project which provides sample
  implementation
• Allows vendors to test their Producers and
  Consumers against this implementation
• Does not guerantee compliance
• Vendors willing to make their Producers
  accessible to others for interop tests?
• Allows others to test their Consumers
• In turn vendors return feedback on their Producer
  implementations
• Requires similar (spec) level of implementation
• Should be easy once version 1 is released
• Still no compliance testing in a formal sense

6
Cross-vendor Testing - Matrix
Consumer/Producer Vendor1 Vendor2 Vendor3 Vendor4
Vendor1
Vendor2
Vendor3
Vendor4
7
Compliance Testing

• Testing a particular implementation for
  compliance with the specification.
• Compliance testing is strictly black box
  testing

8
Compliance Testing - Tasks

• Develop a set of testable assertions which
  represent the specification
• Derive testable assertions from the specification
• Determine test cases implied by each assertion
• Develop a Compliance Test Suite available for
  public download so that vendors and customers can
  verify that their implementations are compliant
  with the WSRP 1.0 Version of the spec
• Design, develop, and test the tests for each
  assertion
• Create a test framework to run, evaluate, and
  report on the tests

9
Deriving Testable Assertions

• Significant Effort to recast the spec as
  assertions
• Requires committee involvement

10
Test Suite Framework

• A technical framework for running compliance
  tests.
• The framework shall be used to run, evaluate, and
  report test cases.
• Should be highly scriptable and configurable so
  test cases may be added easily
• This may be a significant effort by itself,
  depending on our ability to leverage existing
  frameworks

11
Testing Producers

• Develop a Consumer which executes test cases
  against a Producer and reports the results.
• Test cases are configured as XML input
• Reports are generated as XML output

12
Testing Consumers

• Develop a RI Producer
• conforms to the spec (I.e. passes all the
  compliance tests)
• produces reports on the behavior of the Consumer,
  if in error.
• For example, validating the state that was
  supposed to be kept and returned by the Consumer

13
Testing - Roadmap

• Form interop test subgroup
• Start interop tests February, 15th?
• Establish interoperability April, 15th
• Form compliance test subgroup
• Draft assertion suite March, 15th
• Finalize assertion suite May, 1st

14
Interop

• Mike
• Alan
• Richard
• Andre
• Nigel
• Subbu
• Gil(?)

15
Compliance

• Ross
• Dan
• Rich

16
Backup
17
Existing Examples

• Java Community Process
• Calls for developing a Technology Compatibility
  Kit (TCK) side by side with Reference
  Implementation
• Well developed process and tools
• W3C
• The W3C standardization process only produces
  specification documents (Recommendations, etc),
  and does not formally address compliance testing.
• Apache
• Each project defines its own testing practices
  and standards

18
WS-I Testing Working Group2

• Deliverables 
• The Test Tools Development Working Group will
  produce the following deliverables
• 1.  WS-I Test Methodology White paper
• 2.  WS-I Test Tools Specification
• 3. Two or more implementations of Test Tools and
  supporting documentation
• 4.  Assertions and Test Conditions used as input
  to the Test Tools
• 5.   WS-I Tool Configuration Template
• 6. WS-I Experience Report Tool Development as a
  component of WS-I Profile Development

2WS-I Testing Working Group Charter,
http//www.ws-i.org/docs/charters/Test_Charter.pd
f
19
Scope of the Task1

• Derive testable assertions from the specification
• Determine test cases implied by each assertion
• Design, develop, and test the tests for each
  assertion
• Create a test framework to run, evaluate, and
  report on the tests

1TCK Project Planning and Development Guide, Sun
Microsystems, Inc
20
Existing Frameworks

• JUnit -- open source project
• BaRT Batch generator for Regression Tests --
  IBM open source tool used for testing JVMs.
• Suns Compatibility Test Toolkit licensed,
  limited to JCP projects

21
Testing Optional Functionality

• Separate, or at least configurable, test suites
  for discrete levels of function
• The discrete levels of function currently
  mentioned are Simple and Sophisticated
• Finer distinctions may have to be made
• Perhaps the test suite can determine at runtime
  which level of function the Producer supports and
  be able to apply the correct tests.