ATML Test Results

1
ATML Test Results

• Design Evaluation

2
Evaluation Methods

• Prototyping, for TestBase test executive
• Plug-in MTI Controller
• Style sheet
• Comparative analysis, for PAWS
• With the XML output of the current RTS Data Logger

3
Evaluation Conclusions

• With minor exceptions, the current schema
  accommodates all test results generated by our
  products, test results which we consider
  important for users
• The extension mechanism enables us to store other
  data that may be useful
• A few useful data items may be added to the schema

4
Requests for Clarification

• Semantics of boolean attribute entryPoint of
  TestGroup
• Records if executed test group was an entry
  point.
• Semantics of type attribute of Test
• Probably performance, diagnostics, etc.
• TestResult.Limits has the optional attribute
  operator its child LimitPair has the mandatory
  attribute operator are these redundant? or do
  they have different meanings?
• This is by design. Both are needed.
• TestResult.Limits.LimitPair has the mandatory
  attribute operator what goes there if there is a
  single pair of limits?
• Note Either AND or OR would work however,
  making the attribute optional may be a better
  solution.
• Use AND for inside limits use OR for outside
  limits.

5
Requests for Clarification

• Both TestResults its child ResultSet have
  attributes startDateTime, endDateTime,
  testOutcome
• Do these have different semantics?
• If not, one set should probably be removed
• The attributes of TestResults will be removed.

6
Requests for Clarification...

• What is the appropriate place for TPS file
  name?
• May add an optional attribute to SoftwareItemType
• Use SoftwareItemType.nomenclature
• What is the appropriate place for ATE Name?
• Attribute nomenclature of TestStation?
• Ask opinion from Bob Fox, Tim Davis.
• What is the appropriate place for UUT Name?
• Attribute nomenclature of UUT?
• Ask opinion from Bob Fox, Tim Davis.
• Is there a place for Operator Name?
• Maybe under WorkOrder?
• While we could use an extension, adding a schema
  item may be useful for other products/applications
  
• An element will be added under the root.

7
Requests for Clarification...

• Is there a place for the diagnostic conclusion
  (generated by the complete TPS execution)?
• The Indictment of the last executed Test may be
  used, but there may be use cases where individual
  tests are not recorded
• An element/attribute may be added under
  TestResults
• Indictment elements (0 n) will be added under
  TestResults.
• Is it possible to extend the set of data types,
  for ValueType?
• Yes.

8
Enhancement Proposals

• Add support for mapping TestGroup Test items to
  statements in the TPS
• Should also work when the TPS does not have an
  ATML representation?
• Possibly, as file name line/statement no.
• It may be useful to also record the start end
  statement for the entire TPS execution
• In ATLAS, the TPS start statement may be
  different than the start statement of the first
  Test/Test Group
• Possibly under TestResults
• A name attribute will be added to Test and
  TestGroup.

9
Enhancement Proposals...

• Add support for storing the parameters results
  of digital comparisons
• Possibly under Limits
• To support ATLAS PROVE, we should include Masks
  (ZERO ONE), Error Results, Error Locations,
  Fault Number
• As a minimum, we should add an extension
  hook-up under Limits
• Will add a 4-th choice under Limits. Ion to make
  a proposal.

10
Enhancement Proposals...

• Add support for non-binary test outcomes
• Ex Pass / Fail High / Fail Low
• Currently the attribute testOutcome of Test is of
  type PassFailType
• May add other values to PassFailType
• May change attribute type to string
• Will create Outcome element with attributes value
  and qualifier. Outcome.value will store what is
  stored now in testOutcome. A 4-th value, Error,
  will be added to PassFailType. qualifier may be
  used to store additional information for value
  Fail (ex. High, Low) or value Error (ex. error
  description)

11
Enhancement Proposals...

• Add support for identifying test procedures
• In the case of test executives, each Test
  corresponds to a test procedure, which can be a
  separate software module this test procedure
  should be identified in the Test Results file
  (along with version information)
• Proposal add an element of type SoftwareItem
  under Test (similar to TestResults.TestProgram)
• Not critical - the extension mechanism may be
  used as an alternative
• See Add support for mapping TestGroup Test
  items to statements in the TPS

12
Enhancement Proposals...

• Add support for identifying output parameters
• In the case of test executives, TestResult
  elements may store the values of output
  parameters of test procedures
• Since a test procedure can have multiple
  parameters, they must be identified (by name)
• For input parameters, Parameters.Data has a name
  attribute which may be used for identification
• No such support exists for output parameters
• Proposal add a name attribute for
  TestResult.TestData
• Alternative add a Reference element under
  TestResult.TestData (similar to the one under
  Parameters.Data) use these elements to record
  the name of the parameter
• Will add name attribute to TestResult. For
  consistency, will add name attribute to Event and
  Calibration. For consistency, will move name
  attribute of Parameter.Data to Parameter.

13
Enhancement Proposals...

• Add a place for recording simulated execution
• Use case filter out simulated executions, when
  uploading results to a central database
• Possibly, boolean attribute of TestResults
• For the case of test executives, similar
  attributes of Test may record the simulation mode
  for each test procedure execution
• A more elaborate structure may enable the listing
  of individual functional features or hardware
  assets that were simulated
• Will add attribute to Test and TestGroup. The
  attribute that appears at the ResultSet will
  apply to the overall TPS execution.
• Add a place for recording aborted execution
• Use case for storing test results for aborted
  executions execution was aborted due to an
  execution error want to look at test results and
  see what went wrong
• Possibly, boolean attribute of TestResults
• Use NoTest Outcome.value with Abort
  Outcome.qualifier.

14
Enhancement Proposals...

• Add support for storing the name versions of
  system software running on the test station
• May include run-time name version
• Possibly as element of type SoftwareItemType
  under TestResults (or TestStation?)
• For the case of test executivess, similar
  elements under Test may store the names
  versions of run-times used to execute each test
  procedure
• Not critical - the extension mechanism may be
  used as an alternative
• Use extensions. An alternative mechanism for
  recording versions could make use of a
  TestStation ATML document that is
  version-controlled.

15
Enhancement Proposals...

• Add support for storing the type of the operator
  action that triggered an event
• Possibly under Event
• Example PAWS RTS generates events in response to
  operator actions. Different types of events are
  generated for the following actions Reset,
  StartAt ltstatement_nogt, Halt, SkipDelay,
  UserCommand ltcommandgt
• As a minimum, add an extension hook-up under
  Event, so users can extend the schema to
  accommodate the above data
• Use Event.source.

16
Suggested Corrections

• ValueType inconsistency
• ATML type integer has value of type xsinteger
  (w/ unrestricted range), while ATML type
  unsignedInteger has value of type xsunsignedInt
  (w/ restricted range)
• Solutions
• Change type of integer.value to xsint (would
  restrict range)
• Better change type of unsignedInteger.value to
  xsnonNegativeInteger
• Change type of integer.value from xsinteger to
  xsint.
• ValueType simplification for ATML type
  hexadecimal, may use xshexBinary currently we
  use xsstring with a restriction
• Will not change. There was a good reason for
  using the current approach.
• SoftwareItemType.releaseDate is of type xsdate,
  while most of the other items storing date
  information are of type xsdateTime slight
  inconsistency?
• Will not change.