Software Engineering Research Group,

1
International Conference on IT to Celebrate
S.Charmonmans 72nd Birthday (Charm09)

• Distributed t-way Test Suite Generation Algorithm
  for Combinatorial Interaction Testing

Zainal Hisham Che Soh, Mohammed. I Younis,
Syahrul Afzal Che Abdullah, Dr. Kamal Zuhairi
Zamli
2
T-way Combinatorial Testing

• Systematically reduce the number of test cases.
• Given any t parameters, every interaction element
  for t parameters should be covered by at least
  one test case.
• Allows faults triggered by interaction among t
  parameters to be detected.
• Advantages
• Higher test coverage better quality assurance
• Requires no access to internal source code SUT

Combinatorial Interaction Testing 2
3
Related Work

• Existing work has mainly focused on pairwise
  testing such as AETG, IPO, IRPS, DDA, G2Way and
  Tconfig.
• Many failures are caused by the interaction
  involving more than two parameters
• A recent NIST study by R. Kuhn indicates that
  failures can be triggered by interactions up to 6
  parameters
• For certain software, pairwise testing discovers
  a relatively low percentage of faults
• D. Cohen, et al. provided the AETG system for
  pair-wise, triple, or t-way combinations of a
  systems parameters.
• Y. Lei, et al. offered IPO strategy for pairwise
  testing and IPOG for t-way testing.

Combinatorial Interaction Testing 3
4
Research Problem

• In real and complex software product development,
  the number of input parameter, Pi and input
  parameter values, v is quite large.
• For high t-way testing with large parameter and
  input parameter values, the number of interaction
  element, IE grows rapidly as the value of t
  increases and leads toward the combinatorial
  explosion and the computational complexity
  problem.
• The test suite generation time will take longer
  time or can take day to complete or even stall.

Combinatorial Interaction Testing 4
5
Proposed Distributed Strategy for TS-CIT

• The proposed generation distributed strategy are
  based on parallel processing using master-worker
  pattern using tuple spaces technology.
• Tuple spaces is a middleware that enables
  communication between different process by means
  of exchanging tuples through a shared data-space
• The master and all workers are loosely connected
  through TSpaces server.
• Distributes the computational power and memory
  into different process/worker running on
  heterogeneous workstation.

Combinatorial Interaction Testing 5
6
Modelling Master-Worker
notify(X)
Worker X
read ParmSet(P1,P2,P3)
write
notify(Y)
read ParmSet(P1,P2,P3)
ParmSet(P1,P2,P3)
Master
Worker Y
IEset(a1,c1....)
write
Tuple Space
Worker Z
Combinatorial Interaction Testing 6
7
Strategy TS-CIT Master

• Roles of master process
• The master sends ParmSet, interaction strength t,
  and the IEG to shared data space.
• The master generates the exhaustive test set and
  send by batch to respective worker local memory.
• The master starts with an empty test suite (TS),
  and waits for all workers to send a notification
  on new test case t and maximum interaction
  element coverage value maxIEC to shared data
  space.
• The master reads the maxIEC value from each
  worker from shared data space. Then, the master
  chooses the test case t corresponding to the
  highest maxIEC value to be added to latest test
  suite, TS.
• The master issues command to the selected worker
  via shared data space to delete the selected test
  case t from their own local memory and to delete
  interaction elements covered by in IESet and
  notify for recalculate new highest maxIEC.
• The master checks if the IESet is empty or not.
  If empty, the master will consider the latest
  test suite, TS as the final generated test suite

Combinatorial Interaction Testing 7
8
Strategy TS-CIT Worker

• Roles of worker process
• The worker first connects to shared data space
  and reads the ParmSet, IEG and t in shared data
  space.
• Each worker reads the assigned IEG from shared
  data space.
• The worker generates all possible interaction
  elements for their assigned IEG and send back to
  shared data space as IESet.
• Each worker reads all test case t in their local
  memory and the worker determines the maxIEC of
  all t and sends the highest maxIEC with t to
  shared data space.
• The worker reads the command from shared data
  space, if it contains delete command the worker
  deletes t in their local memory and wait for
  notify recalculation of new maxIEC notification.

Combinatorial Interaction Testing 8
9
Master Generate Test Case
Tuple Space
Worker X
1.read TestCase1-10(v1,v2,v3) 2.Write TS and
maxIEC
Calculate IEC send to Master
write
1.read TestCase11-20(v1,v2,v3) 2. Write TS and
maxIEC
Test Set(v1,v2,v3)
Worker Y
Master
TS and maxIEC
Calculate IEC send to Master
Generate all possible Test Case.
IESet
Select TS with maxIEC send by all worker. Delete
IE of selected TS in IESet. If IESet empty then
Stop.
Worker Z
1.read TestCase21-30(v1,v2,v3) 2.Write TS and
maxIEC
Calculate IEC send to Master
Combinatorial Interaction Testing 9
10
Conclusion

• We also describe the ways to parallelize the
  TS-CIT using the tuple space implementation.
• Initial prove of correctness (accuracy) is
  encouraging. For 4 parameter with 3,2,2,2. All
  possible IE is 44. The exhaustive test case is
  24. After 13 iteration all 44 IE is covered by 12
  selected test case.
• In future we hope will be able to implement the
  algorithms and make experiment on their
  effectiveness with regard to other tools.

Combinatorial Interaction Testing 10
11
References

• D. M. Cohen, S. R. Dalal, M. L. Fredman, and G.
  C. Patton, The AETG System An Approach to
  Testing Based on Combinatorial Design, IEEE
  Transactions on Software Engineering, 1997, Vol.
  23, No. 7.
• M. B. Cohen, C. J. Colbourn, P. B. Gibbons and W.
  B. Mugridge, Constructing test suites for
  interaction testing, In Proc. of the Intl. Conf.
  on Software Engineering, (ICSE 2003), 2003, pp.
  38-48, Portland.
• R. Kuhn, D. Wallace, A. Gallo, Software Fault
  Interactions and Implications for Software
  Testing, IEEE Transactions on Software
  Engineering, June 2004, Vol. 30, No. 6.
• Alan Hartman, Leonid Raskin, Problems and
  algorithms for covering arrays, Discrete
  Mathematics 284(1-3) 149-156 (2004)
• Y. Lei and K. C. Tai , In-parameter-order a
  test generation strategy for pairwise testing,
  Proceedings Third IEEE Intl. High-Assurance
  Systems Engineering Symosium., 1998, pp. 254-261.
  
• K. C. Tai and Y. Lei, A Test Generation Strategy
  for Pairwise Testing, IEEE Transactions on
  Software Engineering, 2002, Vol. 28, No. 1.
• Changhai Nie, Baowen Xu, Liang Shi, Guowei Dong.
  Automatic test generation for n-way combinatorial
  testing. In Proceedings of Second International
  Workshop on Software Quality (SOQUA2005), Fair
  and Convention Center, Erfurt, Germany, 2005.
  Lecture Notes in Computer Science 3712, 2005
  203-211.
• Yu Lei et al, IPOG A General Strategy for T-Way
  Software Testing, 2006

Combinatorial Interaction Testing 11