Testing Programs for Transportation Management Systems

1
Testing Programs for Transportation Management
Systems Practical Considerations
Testing is an important part of the deployment of
a TMS. The purpose of testing is two-fold.
First, testing is about verifying that what was
specified is what was delivered - it verifies
that the product (system) meets the functional,
performance, design, and implementation
requirements identified in the procurement
specifications. Therefore, a good testing
program requires that there are well-written
requirements for both the components and the
overall system. Without testable requirements,
there is no basis for a test program. Secondly,
testing is about managing risk for both the
acquiring agency and the systems
vendor/developer/integrator. The test program
that evolves from the overarching systems
engineering process, if properly structured and
administered, allows managing the programmatic
and technical risks to help assure the success of
the project. The testing program is used to
identify when the work has been completed so
that the contract can be closed, the vendor paid,
and the agency can transition the system to the
warranty and maintenance phase of the project.
Meaning of Requirements Statements Terms
The language, syntax, and structure of you
requirements statements are extremely important
as they directly affect the quality and
thoroughness of the test program that is based on
them. Certain terms used in requirements
statements have specific contractual implications.

• Shall is used to confer a requirement on the
  provider of the product or service and is
  typically understood to mean at the time of
  delivery.
• Will is used to confer a requirement on the
  receiver (the accepting agency) of the product or
  service when that product or service is
  delivered. Will is also used to imply a future
  requirement on the provider that should be clear
  from the context of the statement.
• May is a conditional term and implies that
  either the provider or the receiver has the
  option to meet the stated requirement. May
  statements are generally not testable unless
  additional conditions are included to indicate
  what is expected if the provider or receiver
  elects that option.
• Should falls into same category as may and is
  considered an optional requirement that may or
  may not be included in the system depending on
  the providers perspective.
• Must is used to add additional emphasis to the
  requirement statement that can be directed at
  either the provider or receiver, but more
  typically the provider. Must is typically used
  in a requirement that has specific legal or
  contractual ramifications such as may be invoked
  by requiring a particular State statute or
  governing regulation be strictly adhered to in
  the performance of the work. In the contract
  specifications, it has the same basic meaning as
  shall.

From a contracting perspective, only requirements
with MUST and SHALL statements are likely to be
provided by the contractor. All other
requirements should be considered part of a wish
list and will not be part of the testing
program.
2
Writing Testable Requirements Dos and Donts
The requirements statements contained within the
procurement specifications are the basis for test
and acceptance. Poor or missing requirements
statements result in requirements that cannot be
verified and products or services that dont meet
expectations. Requirements statements should be
written as clear, unambiguous, declarative
sentences. Proper grammatical sentence structure
is as important as is use of shall and must,
particularly in defining who is the responsible
party for providing the product or service and
who will be accepting delivery of that product or
service. The following are some dos and donts
for writing and reviewing testable requirements.
Dos

• Write the requirement in simple, understandable,
  concise terms be short and to the point. If
  complex technical terminology is necessary, make
  sure those terms are defined or well understood
  by the provider as well as the receiver.
• For each individual requirement there should be
  one shall or must statement. If the requirements
  are complex, then they should be subdivided into
  a string of individual requirements to the
  greatest extent possible.
• Write the requirement as a positive statement. If
  something is not desired, try to phrase the
  requirement to state what is desired.
• Have a test method, such as inspection,
  certificate of compliance, analysis,
  demonstration or test, and pass/fail criteria in
  mind when writing or reviewing the requirement.
  If you cant figure out how to verify the
  requirement or what criteria constitutes
  acceptance, you cant expect the provider to
  demonstrate compliance with the requirement.
• Consider the complexity, technical expertise, and
  expense of the testing that may be necessary to
  verify the requirement simplifying the
  requirement may result in the same end product or
  service, but at a reduced test expense.
• When preparing the requirements statement, make
  sure that the requirements will have the same
  interpretation regardless of the background and
  experience of the reader. Add clarifying
  information when necessary to ensure a common
  understanding by readers with radically different
  backgrounds.

Donts

• Avoid the use of may and should in the
  requirement statement unless you specifically
  want to give the provider an option in how that
  requirement can be met, or give the receiver an
  acceptance option or an out.
• Avoid negative requirements. Positive statements
  are better, because they can be definitively
  measured for acceptance testing.
• Dont mix dissimilar or unrelated requirements in
  the same statement. This practice complicates
  requirements traceability and verification
  testing. Unrelated requirements will usually be
  verified at different times, under different test
  conditions, and using different test procedures
  or methods.

Additional Resources
The Testing Handbook for Transportation
Management Systems is intended to provide
direction, guidance, and recommended practices
for test planning, test procedures, and test
execution for the acquisition, operation, and
maintenance of transportation management systems
and ITS devices. For a non-technical audience,
the TMS Testing Primer identifies key aspects of
testing, identifies the benefits of developing
and using testing programs, and profiles
successful practices of existing programs. The
material is available on the Federal Highway
Administration website at http//ops.fhwa.dot.gov/
publications/publications.htm/.

• Primer, FHWA-OP-07-089, EDL 14354

• Handbook, FHWA-OP-07-088, EDL 14353

Federal Highway Administration U.S. Department of
Transportation 400 7th Street, S.W.
(HOP) Washington, DC 20590 Toll-Free Help Line
866-367-7487 www.ops.fhwa.dot.gov Publication
No. FHWA-OP-0X-XXX EDL Document No. XXXXX
February 2007