A Framework for Testing Database Applications

1
A Framework for Testing Database Applications
2
Outline

• Motivation
• Issues in testing DB application
• Framework
• State Generator
• Design
• Implementation
• Example
• What Need to be Done?

3
Motivation

• Substantial effort done to insure the correctness
  and efficiency of DBMS (database management
  system)
• Integrity of data
• Efficient access of data
• Little attention to the correctness of DB
  application programs

4
Motivation

• Correctness of DB application
• Does the application program behave as specified?
• Does the database schema correctly reflect the
  organization of the real world data being
  modeled?
• Are security and privacy protected appropriately?
• Are the data in the database accurate?
• Does the DBMS perform all insertions, deletions
  and updates of the data correctly?

5
Motivation

• Input and output space include explicit input and
  output as well as database states
• DB application written in a semi-declarative
  language (e.g. SQL), or a combination of an
  imperative language (e.g. C) and a declarative
  language
• Most existing testing techniques are for
  imperative languages

6
Tester Needs Background

• Tables, records, attributes, primary key, foreign
  key
• Domain constraints (e.g. integer, strings)
• Unique constraints (e.g. SSN)
• Not NULL (e.g. name)
• Referential integrity (e.g. supplier names in
  catalog table should appear in supplier table)
• Semantic constraints (e.g. salary of dept head
  should be higher than those of other members in
  that dept)

7
Background

• Example

8
Issues in Testing DB Application

• Role of DB states
• Input and output spaces have a complicated
  structure
• Expected and actual results (output) also depend
  on the state of the DB before executing.
• Problems
• Controllability put a system into the desired
  state before testing
• Observability observe its state after testing

9
Issues in Testing DB Application

• Role of DB states (cont)
• Failure of finite state model
• infinite states
• difficult to describe transitions between states
  concisely
• Controllability ? populate DB with appropriate
  data

10
Issues in Testing DB Application

• Populate DB with live or synthetic data?
• Live data may not reflect a sufficiently wide
  variety of possible situations.
• Difficult to identify appropriate user inputs to
  exercise interesting situations and to determine
  appropriate outputs.
• Security or privacy constraints prevent tester
  from seeing the live data.

11
Issues in Testing DB Application

• Generating synthetic data
• Generating values from given domains to make
  tables
• Integrity constraints
• We want valid and interesting data.
• Let tester specify situations likely to expose
  faults and to occur in practice.
• Domains, relations, constraints are specified
  explicitly in SQLs DDL (Data Definition
  Language).

12
Framework
13
Framework

• use DB schema and information from tester
  indicating useful values for attributes
• populate DB tables with data satisfying integrity
  constraints

14
Framework

• use information
• generated by 1st Component
• derived from parsing SQL statements
• supplied by tester
• generate input data to be supplied to the
  application

15
Framework

• capture applications outputs
• if possible, check them against expected values
  supplied by tester

16
Framework

• Investigate the DB state after execution

17
State Generator

• 1st component
• Parse schema definitions
• Collect relevant information about tables,
  attributes and constraints, used as the basis for
  an interaction with the user
• Query user to suggest values for each domain
• Combine values into tables while assuring that
  the constraints are satisfied
• Category-partition method

18
State Generator

• Values partitioned into several choices (data
  groups) containing values that are conceptually
  different

19
(No Transcript)
20
(No Transcript)
21
Example
22
Example
23
Example
24
What Need to be Done?

• Domain constraints
• Semantic constraints
• Boundary values among those in DB
• Special values believed to be fault-prone
• Needs to develop ways of validating the output of
  test cases
• Parts of DB that should be unchanged is indeed
  unchanged
• Parts of DB that should have changed really did
  so in appropriate ways