Finding MinRepros

1
Finding Min-Repros in Database Software
Rimma. V. Nehme Microsoft Jim Gray Systems
Lab Madison, WI rimman_at_microsoft.com
Nicolas Bruno Microsoft Research Redmond,
WA nbruno_at_microsoft.com
DBTest 2009 Providence, RI
2
The Problem
Database Testing
Database Debugging
(where bugs are found)
GAP
(where bugs are fixed)
Our Focus
Min-Repros in Database Software
3
What is a Min Repro?

• The simplest possible setup that reproduces the
  original problem

Input Configuration
DBMS Component
.exe
Problem
Output
Inputs
alternatively
Min Repro
4
Example (Product Defects Analysis Application)
-- Query 1 Select basic defect
information select isnull(summary, '') as
'Summary', isnull(disp.descriptor, '') as
'Disposition', isnull(type.descriptor, '')
as 'Type', isnull(prior.descriptor, '') as
'Priority', isnull(prod.descriptor, '') as
'Product', isnull(comp.descriptor, '') as
'Component', isnull(d.reference, '') as
'Reference', isnull(sevr.descriptor, '') as
'Severity', isnull(convert(varchar(12),
d.dateenter, 101), '') as 'Date Entered',
isnull(enteredby.firstname ' '
enteredby.lastname, '') as 'Entered By' from
defects d left join ( select projectid,
idrecord, descriptor from flddispo ) disp
on d.projectid disp.projectid and
d.idDisposit disp.idRecord left join (
select projectid, idrecord, descriptor from
fldtype ) type on d.projectid
type.projectid and d.idType type.idRecord
left join ( select projectid, idrecord,
descriptor from fldprior ) prior on
d.projectid prior.projectid and d.idPriority
prior.idRecord left join ( select
projectid, idrecord, descriptor from fldprod
) prod on d.projectid prod.projectid and
d.idProduct prod.idRecord left join (
select projectid, idrecord, descriptor from
fldcomp ) comp on d.projectid
comp.projectid and d.idCompon comp.idRecord
left join ( select projectid, idrecord,
descriptor from fldsever ) sevr on
d.projectid sevr.projectid and d.idSeverity
sevr.idRecord left join ( select
projectid, idrecord, firstname, lastname from
users ) enteredby on d.projectid
enteredby.projectid and d.idEnterBy
enteredby.idRecord
-- Query 3 Select defect event history select
d.defectnum as 'Defect ', isnull(d.summary,
'') as 'Summary', isnull(e.name, '') as
'Event', isnull(u.firstname ' '
u.lastname, '') as 'Applied By',
convert(varchar, evt.dateevent, 100) as 'Date
Applied', isnull(evt.notes, '') as 'Notes',
isnull(rslt.name, 'ltno changegt') as 'Resulting
State' from users u, defects d,
events e, defectevts evt left
join ( select projectid, idrecord, name
from states ) rslt on evt.projectid
rslt.projectid and evt.rsltstate rslt.idrecord
where d.projectid evt.projectid and
d.projectid e.projectid and d.projectid
u.projectid and d.idrecord evt.parentid
and evt.evtdefid e.idrecord and evt.iduser
u.idrecord order by d.idrecord,
evt.dateevent, evt.ordernum
-- Query 2 Select defect reported by
information select d.defectnum as 'Defect
', isnull(d.summary, '') as 'Summary',
isnull(foundby.firstname ' '
foundby.lastname, '') as 'Found By',
convert(varchar(12), rep.datefound, 101) as 'Date
Found', isnull(verfnd.descriptor, '') as
'Found in Version', isnull(repro.descriptor,
'') as 'Reproducable', isnull(rep.descrptn,
'') as 'Desription', isnull(rep.reprosteps,
'') as 'Steps To Reproduce', case
rep.TstConType when 2 then
isnull(testcnf.sysname, '') else 'User''s
Test Config' end as 'Test Config',
isnull(rep.otherhwsw, '') as 'Other Hw/Sw' from
defects d, reportby rep left join (
select projectid, idrecord, firstname, lastname
from users ) foundby on rep.projectid
foundby.projectid and rep.idFoundBy
foundby.idRecord left join ( select
projectid, idrecord, descriptor from fldrepro
) repro on rep.projectid
repro.projectid and rep.idReprod
repro.idRecord left join ( select
projectid, idrecord, descriptor from fldversn
) verfnd on rep.projectid
verfnd.projectid and rep.versnfound
verfnd.idRecord left join ( select
projectid, idrecord, sysname from sysconf )
testcnf on rep.projectid
testcnf.projectid and rep.idConfig
testcnf.idRecord where rep.iddefrec
d.idRecord and rep.projectid d.projectid
5
Min Repro Problem Statement
P
E
C
DB Min-ReproSystem
C

• Given
• input configuration C
• database executable(s) E
• problem specification (set of undesirable
  conditions) P
• Find the smallest configuration C from C to
  reproduce the problem P

6
Outline For The Rest of Talk

• Motivation
• Problem Statement
• Min-Repro System
• Overview
• Model
• Search
• Useful tools
• Conclusion

7
Min-Repro System Overview

• I - Inputs
• what are the inputs?
• P - Problem
• what is the problem?
• E - Execution component
• what is the executable?

RF(I inputs, E exe, P problem)
DBMS
Repro Function
Min-Repro System
1
2
DB Tester
3
Search Algorithm
Min-Repro
4
8
Modeling a Repro Examples

• Example 1 SQL workload causes DBMS server to
  crash
• Example 2 Variations in costs of different
  releases of the query optimizer

9
Search

• Implemented 2 known algorithms (Delta Debugging)

Isolating
Simplifying
DDMinDiff
DDMin
10
Simplifying Transformations
Transformations

• Intra-Transformations
• applicable to the internal content of an input
• can quickly reduce complexity of individual
  inputs
• - beneficial if repro contains few but complex
  inputs
• Not effective for repros with many but simple
  inputs

• Inter-Transformations
• applicable to a collection of inputs
• can quickly reduce the size of a repro in terms
  of input count
• - beneficial if repro contains many inputs
• Not effective for repros with few but complex
  inputs

Examples
Examples
11
Outline For The Rest of Talk

• Motivation
• Problem Statement
• Min-Repro System
• Overview
• Model
• Search
• Useful tools
• Conclusion

12
Language

• Test Language for Databases (short TLDB)
• Uses XML as its primary syntax
• similar in spirit to the XEXPR language
• Allows to create custom scripts
• enable logic reuse
• automation of search sub-tasks
• Scripts
• inter-scripts - applied to a set of inputs
• intra-scripts - applied to a particular input
• or both

13
Visualization

• GUI can be extremely useful in min-repro search
• A simple (yet intuitive) visualization can help
• in understanding what and why might have caused a
  problem
• come up with a more informed feedback to the
  system
• find min-repros faster

Search space visualization
Visual diff
14
Record-And-Replay
recorded and generalized as patterns for playback
Configuration C
C
transformations

• Record ? Generalize ? Replay
• Record a set of transformations into a pattern

• Benefits
• provide the DB tester with a proven solution
• encapsulates a solution that has demonstrated to
  work before
• improves communication between a DB tester and
  a DB developer
• describes the steps that were taken to produce
  the final min repro
• promotes logic reuse
• saves time and lowers debugging costs

15
Can Debugging Be a Game?

• Game Mode

• Application Mode

Similar

Point system
Time Challenge
Resource Challenge
Customizable challenges
16
Summary

• Our work focuses on the problem of finding
  min-repro in database software
• Our min-repro system attempts to bridge the
  current gap between DB testing and debugging
  disciplines
• Min-Repro system employs a set of useful tools
• General repro model
• Simplifying transformations
• High-level language
• Record-and-replay
• Application and game execution modes
• The min-repro system is designed with both
  experienced and novice DB testers in mind

17
Questions
Contact emails rimman_at_microsoft.com,
nicolasb_at_microsoft.com
18
Backup Slides
19
Motivation

• Database systems play an important role
• Mission critical applications depend on DBs
• Implicit assumption
• Data management services are well-tested,
  reliable and correct
• Testing/debugging DBMS problems is difficult and
  costly
• DBMS software is complex
• Large number of features
• Humans may lack experience and/or time
• Few tools exist using systematic approach to
  testing DB server and its accompanying tools

20
Database Testing and Debugging
fix
Database Debugging
Database Testing

• Simplify the problem
• Locate the error
• Fix bugs

DB Bug

• Find the bugs
• Find them early
• Make sure that they have been fixed

21
Challenges
Min-Repro System

• Challenge 1 how to model a repro? (Need to
  capture different types of inputs, various
  database executables, a wide range of problems)
• General Model for a Repro and a Problem
• Challenge 2 how to simplify (minimize) a repro?
• Simplification Transformations
• Challenge 3 how to reuse search logic and
  automate search?
• High-level Language
• Record-and-Replay Functionality
• Challenge 4 how to execute (interact with) the
  system?
• Application mode
• Game mode

Our Contributions
22
Searching for Min Repro

• Simplify split/transform input set
• into a simpler configuration

Challenges How to simplify? (Steps 1
and 4) Which subset to test first?
(Step 2) How to deal with multiple independent
inputs that reproduce the problem?
(Step 3)
Choose-to-Test test the simplified configuration
Choose-to-Continue continue with the subset that
reproduces the problem
Backtrack if no subset reproduces the problem,
backtrack and try different simplification method
23
Script Diagrammer
To Make Script-Writing Easier