Measuring and Modeling Usage and Reliability for Statistical Web Testing

1
Measuring and Modeling Usage and Reliability for
Statistical Web Testing
Usage Reliability
Statistics
Web Testing

• IEEE Transactions on Software Engineering
• 2001 Nov
• Chaitanya Kallepalli and Jeff Tian
• Presented by
• Kang, Jae-Sung

2
Contents

• Introduction
• Research Background
• Web Testing and Quality Assurance
• Statistical Web Testing
• The Usage Markov Chain
• UMMs for Statistical Web Testing
• Measurement and Analysis Using Web Logs
• A Case Study
• Conclusion

3
Introduction

• Quality assurance for the Web is becoming
  increasingly important
• Statistical testing and related reliability
  analysis
• Able to prioritize testing efforts based on
• usage scenarios
• frequencies for individual Web resources
• Navigation patterns
• A prerequisite to the statistical testing and
  reliability analysis
• The collection of Web usage information
• The construction of usage models
• Web servers are a good resource ex) various log
  files

4
Web Testing and Quality Assurance (1/3)

• Characteristics of Web applications
• Massive user population
• provides cross-platform universal access to Web
  resources
• Diverse usage environment
• Document and information focus
• as compared to computation focus for most
  traditional software
• document and information search and retrieval
  remain the dominant usage for most Web users

Client Web Browsers
Functionality distributed across Different layers
and subsystems
Web Server
Middleware
Traditional testing technique inappropriate for
Web application
Database - Backend
5
Web Testing and Quality Assurance (2/3)

• Define Web failure as the inability to
  correctly deliver information or documents
  required by Web users
• Host or network failure (middleware and Web
  server)
• Browser failures (client side)
• Source or content failures
• HTML documents
• Java, JavaScript, and ActiveX
• Cgi-Bin scripts
• Database
• Multimedia components

6
Web Testing and Quality Assurance (3/3)

• Existing Web Testing

HTML syntax checking
Link checking
Form testing
Java testing
Browser rendering
7
Statistical Web Testing (1/2)

• Due to the massive user population and enormous
  size of the interconnected Web
• Code-based and exhaustive testing can only be
  used in a small restricted area.
• Thus, selective testing based on some priority
  scheme is needed.
• Statistical testing
• Sequences from input domain are stochastically
  generated based on a probability distribution.
• A statistical analysis is performed on the test
  history that enables the measurement of various
  probabilistic aspects of the testing process.

8
Statistical Web Testing (2/2)

• General strategy
• Step 1. Construct the statistical testing models
  based on actual usage scenarios and related
  frequencies
• Step 2. Use these models for test case
  construction
• Step 3. Analyze the test results for reliability
  assessment and prediction, and help with decision
  making.

9
The Usage of Markov Chain (1/6)

• A Markov (Russian mathematician) chain
• A process containing a finite number of states
  for which the probability of being in each state
  in the future depends only on the present state
  of the process

Mon
Fish
.1
.6
.3
Liver
Ribs
Fish
.1
.3
.6
.3
.4
.6
.1
.3
.3
Wen
Fish
Liver
Ribs
Fish
Liver
Ribs
Fish
Liver
Ribs
P(fish on Web following fish on Mon) 0.01 0.09
0.18 28
10
The Usage of Markov Chain (2/6)
Usage Variables Cursor location( CL values Sel,
Ent, Anl, Prt, Ext) Project defined (PD values
Yes no)
11
The Usage of Markov Chain (3/6)
Usage chain for the example
12
The Usage of Markov Chain (4/6)
13
The Usage of Markov Chain (5/6)
Estimate the coverage Of usage chain states and
arcs
analytical results for Markov chains
14
The Usage of Markov Chain (6/6)
average sequence takes 105s Every state is
covered 12 105 1260s (21min) Every arc is
covered 36105 3780s (1hr, 3 min)
15
UMMs for Statistical Web Testing (1/2)

• Unified Markov Model(UMM)
• Execution flow, information flow, transaction
  processing and associated probabilistic usage
  information

16
UMMs for Statistical Web Testing (2/2)
UMMs for SMU/CSE Web pages
17
Testing based on Usage frequencies

• Test Case generation
• Possible test cases with probabilities above
  specific thresholds can be generated to cover
  frequently used operations
• Testing of individual Web pages can be performed
  by existing tools
• Failure or success rates, mean-time-to-failures,
  reliability growth trend, etc.

CSE home -gt Courses -gt Program -gt Exit The
probability of the sequence is 0.3 0.5 0.3
0.045 If this probability is above threshold the
corresponding test case will be selected and
executed
18
UMM construction

• For Web applications
• Related design documents and programs(HTML code)
  can be used to construct the basic usage model
• Ex) FastStats extract site architecture from
  the HTML code
• Transition probabilities can be obtained from
  various sources
• Unique to Web applications is that the customer
  accesses to Web resources can be easily recorded
  at the server side

19
Measurement and Analysis Using Web Logs (1/3)

• Hits
• The HTML file corresponding to a page is
  requested
• Any graphics within the HTML page is requested

Sample entries in an access log
20
Measurement and Analysis Using Web Logs (2/3)

• Error Types

Sample entries in an error log
21
Measurement and Analysis Using Web Logs (3/3)

• Reliability R
• Mean-time-between-failures(MTBF)

f
n - f
total number of f errors n hits Nelson model
R 1 - 1 -
n
n
1
?
ti
If usage time ti is available for each hit i the
summary reliability measure
MTBF
f
i
n
If usage time ti is not available for each hit
i we can use the number of hits as rough
time measure
MTBF
f
22
A Snapshot of Usage Analysis Tool
23
Error Analysis (1/2)
Summary of Errors for SUM
24
Error Analysis (2/2)

• Type A errors permission denied
• 1. Unauthorized access to restricted resources
• Expected behavior
• 2. Wrongfully denied to access to restricted or
  unrestricted resources
• Denied access for unrestricted resources with
  proper access authorization (failure)

25
Conclusion

• An approach for statistical Web testing and
  reliability analysis
• Consider characteristics and usage of Web
  applications
• Supported by automated information extraction
  from existing Web logs
• Uses the general Web model, Unified Markov Chain
• Able to apply various reliability anaysis