Automatic Test Generation from Semiformal Specifications for Functional Verification of SystemonChip

1
Automatic Test Generation from Semi-formal
Specifications for Functional Verification of
System-on-Chip Designs

• C.M.Kirchsteiger1, J.Grinschgl1, C.Trummer1,
  C.Steger1, R.Weiss1, M.Pistauer2

1Institute for Technical Informatics, Graz
University of Technology
²CISC Semiconductor Design Consulting
GmbH, Klagenfurt
2
Content

• Introduction
• Motivation
• Novel Verification Methodology for SoC
  (System-on-Chip) Designs
• Implementation
• Case Study of an Radio Frequency Identification
  (RFID) Controller
• Similar Approaches, Challenges, Conclusion
• Outlook and Future work

3
Introduction

• System-on-Chip (SoC)
• Entire system integrated on a single chip
• Increasing complexity (embedded processors,
  multi-bus architecture, memories) due to rising
  integration densities on chips
• Simulation-based Functional Verification of
  System-on-Chip Designs
• For functional verification the SoC design model
  is simulated with testbenches to determine if it
  corresponds to its specification

4
Motivation

• Rising number of requirements according to the
  increasing complexity of SoC (System-on-Chip)
  designs
• Requirements are limitations on the design and
  have to be fulfilled by the design
• Clearly, deriving testcases from the requirements
  manually by reading the usually large and
  textual-based SoC specification document is time
  and resource intensive as well as error-prone
• Due to the usually purely informal, non-machine
  readable structure of the specification document
  for HW designs, it is infeasible to perform the
  testcase generation automatically

5
SIMBA (Simulation-based Requirements Testing of
SoCs)

• A novel design flow, towards the integration of
  system requirements throughout the design flow of
  SoCs
• Goals
• Extract the requirements from the system
  specification
• Manage requirements throughout the design flow
• Select the optimal design for the given
  requirements
• Provide simulation-based techniques to ensure
  that the design fulfils the requirements

6
Novel Verification Methodology for SoC Designs

• A specification-based verification methodology
  for SoC designs that aims for
• The selection of a specification format suitable
  for SoC design verification
• The correction of the specification to reduce the
  risk of late redesigns
• The automated generation of testcases from the
  functional specification to speed up functional
  verification
• The execution of testcases with advanced
  verification techniques (direct-random) to
  increase test coverage

7
Which Requirements Specification to use for SoC
Designs?

• Purely informal specifications formats
• Common document-based specification formats
  with diagrams
• Widely accepted, easy to use
• - Very difficult to parse
• Formal specification formats
• Easy to parse
• - Hardware engineers typically shy away logical
  languages such astemporal logics and other
  programming paradigms like object-oriented
  techniques 1
• Semi-formal specification formats
• Desirable trade-off between the techniques
  above
• Easy to use and widely accepted
• Easy to parse automatically

8
Semi-Formal Specification Formats

• Textual Use Cases 3
• Define the interaction with the environment as
  scenarios in a step-by-step way
• Benefits for HW specification
• Table-based structure is suitable for automatic
  processing and extendable (e.g. new fields to
  cover non-functional requirements)
• Informal description within each field eases
  specification
• Easy integration with existing design practice,
  due to its document-based structure

9
Specification of an Radio Frequency
Identification (RFID) Controller
Source http//www.epcza.org/upload/epcglobal_2_uh
f_rfid_protocolv.pdf
10
RFID Specification as Textual Use Cases

• Name Set up Tag
• Description Use case accessed when tag enters
  the Reader field.
• Scope UHF RFID Tag (Tag).
• Primary actor Interrogator (Reader)
• Precondition/Trigger Tag comes from Hibernate
  use case
• Main Success Scenario
• 1. Tag receives Query command from Reader with
  Receiver unit _at_TREQ1
• 2. Tag detects Query command
• ..
• 11. Tag transmits 16bit Random number with
  Backscatter unit
• 12. Tag exits use case and goes to Reply Tag
  Use Case
• Alternate Flows
• 1a. Tag receives Select command from Reader with
  Receiver unit
• ...
• LocNonfunctional Requirements
• Timing Requirements
• _at_TREQ1 The tag shall respond within t1
  _at_TREQ3 Step 1 until step 11 shall be done within
  t1.

Non-Functional Requirement
Functional Requirements
11
Parsing the Use Case Requirements (1)

• Natural Language Processing
• Based on a language subset (lexicon) and given
  grammar structures to ease parsing
• User Interaction
• To specify unknown terms
• To rewrite invalid grammar
• Guidelines provided
• Self-learning Lexicon
• Common terms and their types for the
  specification of SoC Designs

12
Parsing the Use Case Requirements (2)

• Lexical Analysis
• Parses each interaction step
• Determines the type of each term
• Looks up in lexicon
• Syntax Analysis
• Compares the grammar with the given grammar
• Generates a syntax tree
• Semantic Analysis
• Identifies the meaning of each term (action,
  actors, message,)
• Output
• Corrected requirements file

13
Automatic Testbench Generation

• Corrected XML use case specification as input
• Generates a SystemC Testbench (TB) (state
  machine-based)
• A testcase is generated for each testable
  interaction step
• A use case scenario corresponds to a flow of
  testcases
• SystemC Transactor module required to connect
  TB to the SuV
• Separates the SuVs interface from the TB

SystemC Testbench State machine
Transactor between TB and SuV
14
Testcase Simulation

• Directed Tests 2
• - Have to be defined manually to cover expected
  behavior
• - Cannot execute all possible input sequence
  combinations
• - Unexpected behavior (i.e. corner-cases) is not
  tested
• Directed (Constrained-) Random Tests 2
• Faster, since input sequences are randomly
  generated
• Randomized input sequences cover also
  unexpected behavior

Source Automatic Stimulus Generation
Randomized Testing in Vera
15
Simulation-based Algorithm Using
Constrained-Random Verification

• Randomly select and execute testcases from the
  testbench state machine during simulation
• During each iteration every use case is executed
  at least once.
• In the first iteration, use cases are processed
  sequentially by executing the main success
  scenario
• All further iterations the SystemC Verification
  Library (SCV) for randomization regarding
• The selection of the use cases
• The selection of the scenario
• Data randomization The content of each data
  range

16
Implementation

• JAXB
• Generates a Java structure based on the use case
  XML specification
• CUP Parser
• Invokes JFlex to determine the type of each term
• Checks the grammar
• Semantic Analysis
• Identifies actions, actors and messages
• Automatic Testbench Generator
• Generates the state-based SystemC testbench
• Generates a SystemC transactor

17
SyAD (System Architect Designer)
SyAD Snapshot
18
Case Study of an RFID (Radio Frequency
Identification) Controller
19
Testcase Generation from the RFID Specification
RFID use case specification
Generated SystemC testbench
20
Mapping of Testcases to the RFID Interface
SyAD Snapshot
21
Results (1) Parsing the specification

• RFID Specification containing 53 use case
  scenarios
• Lexical Analysis
• Prompts the user to add the type of eight unknown
  terms to the lexicon
• Syntax Analysis
• Identifies four missing terms and two invalid
  word-orders. Prompts the user to rewrite the
  sentence

Results from parsing the specification
22
Results (2) Testcase Execution

• Five use cases (UC) with 131 testcases are
  simulated with 1, 5, 10, 15 and 20 iterations of
  the proposed random testcase selection and
  execution algorithm
• UC1 corresponds to the first state in the RFID
  state machine. It is executed most of all, since
  it is in the pre-condition list of all other use
  cases.
• UC5 is less executed since it corresponds to the
  final RFID state

Simulation results
23
Results (3) Testcases Coverage
Test coverage during simulation

• 20 iterations of the proposed algorithm execute
  80 of all use case scenarios and disclose 14
  design errors

24
Conclusion (1)

• Novel Specification-based Verification
  Methodology for SoC designs aiming for
• Uses semi-formal specifications
• Easy-to-read/specify and integration with
  existing practice
• Suitable for automatic processing
• Specification checking to reduce the risk of late
  redesigns
• Removes specification errors
• Detects missing terms and invalid grammar
• Automatic generated testcases to speed up
  verification
• Testcase are generated from the specification
  with minimum user interaction
• Testcases help the designer/verification engineer
  to verify if the design fulfils its specification
• Constrained-randomized testcase simulation to
  check corner cases
• Requirements changes are propagated to the
  testcases

25
Conclusion (2)

• Case Study of an Radio Frequency Identification
  (RFID) controller state machine
• 6 syntax errors detected in the specification
  document
• Test coverage of 80 achieved

26
Comparison to Other Approaches

• M. Riebisch and M. Hubner. Traceability-driven
  Model Refinement for Test Case Generation. 2005.
• Automatically transforms use cases to UML
  activity diagrams. This is done by lexical,
  linguistic and semantic analysis. The ability to
  automatically generate testcases from the UML
  diagrams is completely missing
• A. Fantechi, S. Gnesi, G. Lami, and A. Maccari.
  Application of Linguistic
• Techniques for Use Case Analysis. 2002.
• Linguistic techniques are applied to extract
  useful information from the use cases.
  Nevertheless, no testcases are automatically
  created either.
• Raven from Ravenflow
• Testspecifications are directly generated from
  natural language requirements and can be export
  to a testmanagement tool
• Testspecifications are not directly embedded in a
  simulation environment
• Testspecifications are not available in SystemC,
  VHDL,... Testexecution-scripts are generated and
  must be somehow linked to the modeled design
• No random-constrained testcase execution
  algorithm to handle large numbers of testcases

27
Challenges

• Main limitation is given by the external
  interface of the SuV
• Only black-box tests are generated
• Not every design can be specified by an
  interaction flow with its environment
• Non-functional requirements have to be covered
  differently

28
Outlook and Future work

• More designs to further confirm the performance
  benefits of the proposed methodology
• Approach is limited to black-box tests
• Future work focuses on the enhancement with
  white-box verification capabilities
• Ongoing work focuses on the integration and
  verification of power requirements to verify
  power-aware SoCs
• Adapt the algorithm to consider the functional
  coverage of the SuV when selecting the testcases
  randomly

29
Thank you! Questions?

• Institute for Technical Informatics, TU Graz,
  Austria
• In cooperation with
• CISC Semiconductor Design Consulting GmbH,
  Austria
• and University College Dublin, Ireland

30
Literature

• 1Formal Specifications Languages for Protocol
  Compliance Verification, A. Bunker, University
  of Utah, S. A. McKee, Cornell University
• 2Closing the loop in Testbench Automation,
  M. Olen, Mentor Graphics
• 3Writing effective Use Cases, A. Cockburn

31
Disclosed Design Errors

• No support for RFID commands
• -gt BlockWrite not supported
• -gt No Query supported, instead usage of
  Query_bat
• -gt No Error-code replied by a kill command with
  invalid handle
• -gt No escape from hibernate state

32
Implementation in SyAD