SSS Validation and Testing

1
SSS Validation and Testing

• June 5, 2003
• Chicago, IL
• Erik DeBenedictis
• William McLendon III

Sandia is a multiprogram laboratory operated by
Sandia Corporation, a Lockheed Martin
Company,for the United States Department of
Energy under contract DE-AC04-94AL85000.
2
Overview

• Strategies for RT System Testing and Validation
• Effective System Performance Benchmark (ESP)
• APITEST

3
Distributed Runtime System Testing

• Complicated interactions
• Approach to testing
• Component Testing
• Light Applications
• Performance / Functionality
• Operational Profile
• Stress Testing
• Users expect a high-degree of quality in todays
  high end systems! Extensive (and smart) testing
  is critical to achieving this goal.

4
Effective System Performance Benchmark(ESP)

• Developed by NERSC for use in their system
  procurement process.
• Benchmark suite for MPPs for overall
  effectiveness.
• Measures total system utilization
• A day in the life of a MPP
• Designed to predict the effectiveness of a system
  before purchase.
• Theoretical peak is only part of the story
• Available from NERSC for download
• http//www.nersc.gov/aboutnersc/esp.html

5
ESP Goals

• Determine how well an existing system supports a
  particular scientific workload
• Assess systems for that workload before purchase
• Provide quantitative effectiveness information
  regarding system enhancements
• Compare different systems on a singe workload or
  discipline
• Compare system-level performance on workloads
  derived from different disciplines
• Compare different systems for different workloads.

6
ESP Execution Profile
7
APITEST
8
APITEST - Overview

• Unit-testing tool for network components
• Targeted for networked applications
• Scriptable Tests (XML Schema Grammar)
• Enabling Technology
• Black/White Box Testing
• Multi-Protocol Support
• TCP/IP, SSSLib, Portals

9
Status of APITEST

• recoded in Python
• for portability
• Libraries
• Compilers (ANSI ? ANSI)
• Integrated into SSSLib
• sss/apitest
• Tests well under development for ssslib
  components now.
• More detail on following slide

10
APITEST Status

• Coded in Python
• Scriptable Tests in XML
• Defined in XML Schema
• Test Entity Types
• Command Shell
• Network Transmission
• Dependencies
• Parent
• Sibling
• Distributed Test Execution

• PASS/FAIL Criteria Handling
• Regular Expression Matching
• Exact Matching
• Schema Validation
• Test Repetition
• Tests Sequences
• Test Chaining
• Protocol Support
• SSSlib, Portals, TCP/IP
• Result Handling
• Historical Data
• MySQL Database

Working Partial Planned
11
APITEST Enables

• Flexibility in unit testing without a lot of
  coding
• Black Box testing
• Does a component support the API?
• Fault injection?
• RT Integration testing
• Component abstraction
• Path analysis
• White Box testing
• Coverage Testing
• Internal States of Component X
• Unreachable States?

12
APITEST supports Regular Expressions

• Regular expression matching in tests

Sent to service-directoryltget-locationgtltlocation
component/gtlt/get-locationgt Expected
Bufferltlocationsgt ltlocations
componentservice-directory/gt ltlocations
componentevent-manager/gtlt/locationsgt Regular
Expressionltrecvbuf typeREGEXPgt
ltlocationsgt(\sltlocation\scomponent\S\s/gt\s
)lt/locationsgtlt/recvbufgt
13
Encoding XML into XML CDATA
XML parsers will interpret the lt character as a
newElement, NOT as a character buffer. So,we
must trickour SAX parse by using lt instead of
lt
ltssslibTestgt ltnamegttest1lt/namegt
ltdestinationgtsdlt/destinationgt
ltsendbufgt ltget-locationgt ltlocation
component''/gt lt/get-locationgt
lt/sendbufgt lt/ssslibTestgt
14
APITEST Component Architecture
15
Test Sequence Looping
16
Test Chaining
17
APITEST Example Script
Example 1 - Shell Test Example 2 - Shell
Test Example 3 - Service Directory Query Example
4 - Service Directory Query
18
apitest.py Command Line Options

• Supported Command Line Options
• REQUIRED -f ltinput file.xmlgt
• Specifies the test specification file
• OPTIONAL --sqldb
• Tests will save their results into a mysql
  database.
• Only in SHELL tests thus far
• OPTIONAL --test
• Apitest prints some text to stdout and stderr and
  exits (20).
• mainly used by shl-t1.xml and shl-t2.xml

19
MySQL Database Support

• MySQL database support
• MySQL is open source
• SQL is here to stay
• Benefits of using a database
• Historical data is easily accessed
• Detect response time creep
• Report generation flexibility
• Do not have to modify apitest for new reports.

20
SQL Fields for ShellTest Table

• TESTNAME
• SCRIPTNAME
• DATE
• AUTHOR
• CONTACT
• ELAPSEDTIME
• CHKSTDO
• CHKSTDE
• CHKSTUS
• STDO_PASS
• STDE_PASS
• STUS_PASS

• CCODE
• EXPECTED_STDO
• EXPECTED_STDE
• EXPECTED_STUS
• STDO_BUFTYPE
• STDE_BUFTYPE
• STUS_BUFTYPE
• STDO_ACTUAL
• STDE_ACTUAL
• STUS_ACTUAL
• ERRATA
• DETAIL

21
APITEST Summary

• Ongoing effort
• Concept from October 2001 F2F discussions.
• Good progress so far
• Included in SSSlib RPMs
• Developed XML Schema definition for test entities
• Changed from C to Python
• Already much more functionality than prototype

22
Conclusions

• Testing and Validation
• Existing capabilities
• Application Testing
• SNL Frameworks
• ESP Test Framework
• Stress Tests
• APITEST integration with SSSLIB distribution
• Packaged with the RPM
• Quite a lot of work still to do

23
Questions?