MD850: e-Service Operations

1
MD850 e-Service Operations

• Capacity Management

2
Overview

• Motivation
• Background
• Capacity Management
• Manufacturing Operations
• Person-to-Person Service Operations
• e-Service Operations
• Capacity Management Strategies
• Software Demonstration
• MercuryInteractives Astra Load Test

3
Motivation
4
Motivation

• Many e-Service Problems Exist that Affect Service
  Quality and Hurt Revenues
• Downtime
• Slow times (service slowdown)
• 37 of site pages exhibit unacceptable
  performance as defined by their managers (Mercury
  Interactive 2001)
• Unpredicted traffic volume spikes
• Transaction failure
• 1 in 20 transactions fail on the average
  corporate website (Mercury Interactive 2001)

5
Motivation

• 52 of e-Service applications failed to scale
  according to original design expectations
  (Newport Group 1999)
• Automated load testing tools were not used
• 60 used no load testing tool prior to deployment
• 34 used load testing late in development or post
  deployment only
• Budget and time overruns above average
• 23.5 days, 67,083
• Scalability expectations were not realistic
• Thought they would handle 4300 concurrent users
  on average
• Actually could only handle 2200 concurrent users
  on average

6
Background
7
BackgroundCapacity Concepts

• Capacity
• rate at which output can be produced by an
  operating unit (e.g., a machine, a process, a
  facility, a company)
• capacity number of units of output/time
  period
• Design Capacity
• maximum rate at which the process can operate on
  a continual basis under ideal conditions
• Effective Capacity
• rate of production that can be achieved for
  extended periods under normal conditions, taking
  into account various infrastructure and
  environmental factors

8
BackgroundCapacity Concepts

• Capacity Utilization
• actual output/design capacity
• Capacity Efficiency
• actual output/effective capacity

Managerial Focus On Loss in Production Capability
Actual Output Period 5
Actual Output Period 8
0
Design Capacity
Effective Capacity
9
Capacity Management in Manufacturing Operations
10
Capacity ManagementManufacturing Operations

• Supply Chain Processes
• Supply Chain Modularity
• Facilities
• Facility Location
• Facility Focus
• Process Design

• Product Design
• Product Variety
• Product Quality
• Production Scheduling
• Materials Management
• Maintenance
• Job Design and Personnel Management

11
Capacity ManagementManufacturing Operations

• Common Approaches
• Forecasting demand
• Translate demands (d1, d2, , dn) for products
  (1, , n) into capacity requirements
• Break-Even Analysis
• Decision Analysis

12
Capacity ManagementManufacturing Operations

• Demand-Side Management of Capacity
• Counter-cyclic products
• Toro - lawnmowers, snowblowers
• Promotion
• Pricing

13
Capacity ManagementManufacturing Operations

• Medium-term to Short-term approaches
• Aggregate planning
• chase demand strategy
• level demand/workforce strategy
• Inventory management
• Materials management
• Operations scheduling
• Personnel scheduling

• Typical capacity analysis management tools
• Optimization
• MRP CRP
• Inventory review
• Queueing
• Simulation
• Optimization

14
Capacity ManagementManufacturing Operations

• Capacity Strategies
• Facility
• Product-Organized facility makes one product
  type
• Process-Organized multiple products or parts
  made through one process
• Market-Organized close physical proximity to the
  customer
• Focused Factory/Plant-Within-A-Plant
• Capacity Expansion
• Demand-Leading maintain excess capacity
• Demand-Trailing capacity lags behind demand
• Demand-Matching match capacity to demand
• Steady Expansion add capacity on regular basis,
  based on long-term needs, but not on demand
  fluctuations

15
Capacity Management in Person-to-Person Service
Operations
16
Capacity ManagementPerson-to-Person Services

• Capacities to Manage
• Facilitating Goods
• When produced internally, same issues as
  manufacturing
• Often procured from supplier
• Services
• Persons who can deliver service at a certain rate
• Information
• Printed information either inventoried, or
  printed on demand at some rate
• Communicated in person at some rate

17
Capacity ManagementPerson-to-Person Services

• Demand Side Management of Customer Behaviors
• Appointment reminders
• Pricing
• penalty for being late
• yield-management
• Wave scheduling
• Capacity sharing
• One printer, 20 professors
• Ask customer to stop consuming/conserve
• Energy efficient light bulbs
• Stop filling demand/shut off service
• California electric service

18
Capacity ManagementPerson-to-Person Services

• Service Facility Design
• Queuing model of service system queues
• Simulation models to study activities within
  system and to estimate effective capacity,
  service times, and so forth
• Daily/Weekly Capacity Management
• Service Personnel/Staff Scheduling
• Nurses, police, paramedics
• optimization - integer linear programming

19
Capacity Management in e-Service Operations
20
Capacity Managemente-Service Operations

• e-Service Capacities to Manage
• Digital network capacities
• Goods network capacities
• Person-to-Person service network capacities
• Inter-layer digital service capacities

Service-Product
Service-Process
Digital Networks Digital Content
Service Networks, Intelligent Goods Networks
Service Personnel
21
Capacity Managemente-Service Operations
Service-Product
Service-Process
Distributed Component Applications
Client/ Server
Digital Networks
Info Digital Content
Networks of Physical Objects
People
22
Capacity Managemente-Service Capacity

• Determinants of e-Service Capacity
• Number of customers and internal users
• demand
• Service-Product
• site content
• Service-Process
• server capacity and configuration of hardware and
  software

23
Capacity ManagementDeterminers of e-Service
Capacity
Average Digital Content Demand Per User
Number of Users Per Time Period
Desired Capacity
x
Number of Supported Concurrent Users
Load on Hardware Per User
Hardware Capacity (design)

/
Goods Info Services Info Digital
Services Content Consumed Service-Product
Server Network Infrastructure Software
Modules Service-Process
Web browsers Wireless apps Other processes (e.g.,
ERP system) User Mix
24
Capacity ManagementDeterminers of e-Service
Capacity
25
Capacity ManagementDeterminers of e-Service
Capacity

• Common e-Service Problems and Causes
• Long response time from end users point of view
• Long response time as measured by the servers
• Memory leaks
• High CPU usage
• Too many open connections between the application
  and end users
• Lengthy queues for end user requests
• Too many table scans of the database
• Erroneous data returned
• HTTP errors
• Pages not available

26
Capacity Managemente-Service Operations

• Typical e-Service Infrastructure Managed to
  Achieve Desired Capacity
• Application Server Software
• Web Server Software
• Database Software
• Networking Software
• Load Balancing Software
• Application Server Hardware
• Web Server Hardware
• Database Hardware
• Networking Hardware

27
Capacity ManagementStrategies for e-Service
Capacity

• Strategies for Managing e-Service Capacity
• Manage demand
• Manage Service-Product
• Manage Service-Process

28
Managing e-Service Demand Patterns
29
Capacity Managemente-Service Demands
Cyclical
Random Infrequent
6am 12pm 6pm 12am 6am 12pm 6pm 12am Day 1
Day 2
Demand Surge
6am 12pm 6pm 12am 6am 12pm 6pm 12am Day 1
Day 2
30
Managing the e-Service Product
31
Capacity Management e-Service Reference Model
(Menasce and Almeida, Scaling for e-Business,
2000)
Business View
External Metrics
Characteristics of the Business
Business Model
Navigational Structure Function
Functional Model
Patterns of Customer Behavior
Customer Model
Site Architecture and Service Demands
Resource Model
Technological View
Internal Metrics
32
Capacity PlanningTranslating the Service-Product
into Loads
Characteristics of the Business
Business Model
Navigational Structure Function
Goods Services Information
Functional Model
Service- Product
Menasce and Almeida (2000)
Heim and Sinha (2000), Spiller and Lohse (1998)
A Network of Paths Between Pages/Objects
WWW Site Content
33
Managing the e-Service Process
34
Capacity Managemente-Service Process Strategies

• Implications of Classic Aggregate Planning
  Strategies
• chase demand strategy
• add a server when you sense it is needed
• take away a server when it is not needed
• level demand strategy
• build an inventory of e-Services (IMPOSSIBLE)

35
Capacity Managemente-Service Process Strategies

• Implications of Classic Capacity Expansion
  Strategies
• Demand-Leading maintain excess capacity
• expensive solution
• service level stays reasonable
• customers satisfied
• Demand-Trailing capacity lags behind demand
• queue of requests builds
• service slows as servers average capacity across
  all requests
• server computers grind to a halt when demand
  severely surpasses capacity time to purchase
  new servers
• dissatisfied customers

36
Capacity Managemente-Service Process Strategies

• Implications of Classic Capacity Expansion
  Strategies
• Demand-Matching match capacity to demand
• as demand grows (shrinks), in-house (and
  outsourced) servers are dynamically added
  (removed)
• capacity for static files (GIF, JPEG, .EXEs,
  etc.) outsourced to a third-party
• as demand grows (shrinks), third-party senses
  growth and re-allocates files across a larger
  (smaller) subset of their network
• you manage basic content (e.g., HOME.HTML file)
• caching strategy (Akamai, Inktomi, others
  vendors)
• Steady Expansion add capacity on regular basis,
  based on long-term needs, but not on demand
  fluctuations
• intermittent periods of poor service
  responsiveness
• if demand grows too fast, system may fail,
  customers dissatisfied

37
e-Service Capacity Planning, Analysis, and
Management
38
e-Service Capacity Planning, Analysis, and
Management

• Best Practices
• Set conservative goals with the intent to expand
  them out in a systematic fashion
• Have a plan for short-term incremental
  achievements of long-term goals
• Integrate load testing into the development
  process of web applications early and often
• Always test the limits of any e-Service
  application prior to going live
• Leverage pre-deployment test assets in the
  production environment to monitor live web
  application performance
• Set a load capacity threshold for growth which is
  continuously monitored
• At 70-80 of the existing systems handling
  capacity, start executing plans to add more
  capacity
• (Source Newport Group 1999, 2000)

39
e-Service Capacity Planning, Analysis, and
Management

• Best Practice Objectives
• Minimized Latency
• Keep waiting time between making a request and
  beginning to see a result as low as possible
• Maximized Throughput
• Number of items potentially processed per unit
  time should be as high as possible
• Mid-to-high Utilization
• Actual capacity utilization of components should
  be kept around 75 latency suffers as you go
  over this level
• Maximized Efficiency
• Keep overall performance high and cost low

40
e-Service Capacity Planning, Analysis, and
Management

• Best Practice Metrics
• End User Response Time
• Measures the performance of an application from
  the end-user perspective
• The amount of time required for an end user to
  receive a response or to execute a business
  transaction
• Application Availability
• Page availability
• Transaction availability
• User-perceived availability
• Reliability
• Correct Content Delivery in Multi-Step
  Transactions
• (Sources Newport Group 1999, Mercury Interactive
  2001)

41
e-Service Capacity Planning, Analysis, and
Management

• Best Practice Guidelines
• Focus on critical business processes
• Use the right monitoring solution to meet your
  business needs
• Get a consistent performance baseline and watch
  for trends
• Avoid an alerting flood send alerts
  conservatively
• Think recurring when acting on alerts
• Correlate end-user performance with back-end
  issues
• When a problem is identified, prioritize IT
  resources
• Optimize your existing infrastructure
• Define escalation procedures to follow to address
  performance issues
• Use application performance monitoring to avoid
  having every department scramble when performance
  goes bad
• (Source Mercury Interactive, 2001)

42
Capacity Management Techniques During e-Service
Design and Development
43
Capacity ManagementDesign Development Stage

• Basic Question How to design communication
  between e-Service application layers?
• Capacity Issue
• How to set up appropriate connections between
  tiers?
• How to avoid too few connections between tiers?
• How to avoid bad effects from failure of a server
  in one of the tiers?
• Methods
• Separation of presentation logic from business
  logic from database management
• Load balancing
• Outsource network storage for caching of content

44
Capacity ManagementDesign Development Stage

• Commonly Observed Design Problems
• General
• Insufficient memory
• Incompatible service packs and application
  extensions (e.g., .DLLs)
• Excessive queueing requests
• Too many secure HTTPS connections in use
• Web Server
• Insufficient memory
• Poor web server design
• High CPU usage
• Application Server
• Poor cache management and high CPU usage
• Lack of memory
• Poor session management
• Poor database tuning

45
Capacity ManagementDesign Development Stage

• Commonly Observed Design Problems
• Database
• Inefficient indexing
• Fragmented databases
• Out-of-date statistics
• Faulty application design
• Network
• Inadequate Internet pipe
• Hidden bottlenecks between the customers Web
  site and the ISP
• Faulty hops (misdirected traffic and lost
  packets)
• Misconfigured software and incompatible hardware
• (Source Mercury Interactive 2001)

46
Capacity ManagementDesign Development Stage

• Load Balancing
• Objective is to be able to allocate demand as
  it is requested to resources that will provide
  best service response
• Many sites use N-tier systems with many servers
  in each tier that perform same functions
• Reliability
• Backup in case of failure and when server
  maintenance must be performed
• Possible techniques
• RANDOM DISTRIBUTION Round-robin (random)
  allocation of a customer request to an IP address
  for a web server. If a server fails, customer
  requests will still be allocated to it until its
  IP address is removed from service.
• INTELLIGENT DISTRIBUTION Customer requests are
  allocated to servers based upon current
  utilization at each server. The lowest
  utilization server will receive the next job.

47
Capacity ManagementDesign Development Stage

• Outsource network storage for cache capacity
• Akamai, Inktomi, etc. provide services for
  storing and serving static content
• Many of these companies are also setting up
  caching procedures for dynamic content

48
Capacity ManagementDesign Development Stage

• Basic Question Theoretically, how many people
  can my homepage be served to simultaneously?
• Capacity Issue
• What is my websites design capacity?
• Back of the Envelope Method
• Home page (average) size 50,000 bits 6250
  bytes
• Rented T3 line 45,000,000 bits/sec
• 45,000,000 bits/sec / 50,000 bits/customer
• 900 happy customers/sec the design capacity
• 900 customers/sec 60 60 24
• 77,760,000 happy customers/day
• assuming 900 customers request the page
  simultaneously at the beginning of each second,
  and each has a fast enough modem to receive the
  file by the end of the second

49
Capacity ManagementDesign Development Stage

• Load Analysis and Testing Tools
• Brute Force Calculations
• Queueing theory
• Brute Force Load Testing
• Thousands of employees at their computers
• Simulation-Based Load Testing Monitoring
  Software
• Load-Test Can our site handle 25 users?
• Stress-Test Stable? Reliable? Over long
  period?
• Capacity-Test Maximum number of concurrent
  customers?

50
Capacity ManagementDesign Development Stage

• Simulation Based Load Testing
• Employ virtual (i.e. fake) users on client
  computers
• Have virtual users use the e-Service system based
  on scripted behaviors recorded for them by the
  load testing program
• Collect data about the performance that the
  virtual users experience
• Modify service process depending upon the typical
  performance observed

51
Capacity ManagementDesign Development Stage

• Major Vendors Applications
• Mercury Interactive
• LoadRunner WinRunner TestDirector
• Astra LoadTest Astra Quicktest Astra
  SiteManager
• Segue
• SilkPerformer
• Empirix
• Compuware
• Radware

52
Capacity ManagementDesign Development Stage
Capacity/Load Test
Monitoring
Development
Deployment
M.I. LoadRunner, etc. M.I. Astra LoadTest,
etc. Segue SilkPerformer
Software Product
Deployment
Deployment
Service
53
Capacity Management Techniques After e-Service
Deployment
54
Capacity ManagementAfter e-Service Deployment

• General Patterns
• As richness of content increases, capacity
  decreases (for a fixed service process)
• As customers converge to more actively involved
  behavior types (are retained, and thus consume
  more content), capacity decreases (for a fixed
  service process)
• As network of site paths increases in complexity,
  the complexity of managing capacity will increase

55
Capacity ManagementAfter e-Service Deployment

• General Responses
• Decrease content richness (when appropriate)
• Make customers more efficient in their behavior
• Manage site complexity

56
Capacity ManagementAfter e-Service Deployment

• Content Tuning
• Serve out content at a lower level of richness,
  or remove some content
• By lowering the number of bytes for each file,
  less capacity will be used by each individual
  user
• Saved capacity can then be used to serve
  additional customers

57
Capacity ManagementAfter e-Service Deployment

• Content Tuning
• Possible techniques
• DESIGN TIME Create separate sets of high
  resolution and low resolution images for your
  e-Service, and program your site so that you can
  change which images are dynamically inserted into
  your pages
• DESIGN TIME Create a program that loads an
  image directory on the web server with high or
  low resolution images, depending upon present
  demands
• RUN TIME Use a program procedure or filter to
  pre-process each image to a lower resolution, as
  the image is requested. Once the image has first
  been processed, the filter could just serve out
  the lower resolution image. (Note this approach
  lowers image bandwidth but increases page
  processing time.)
• RUN TIME Change heavily downloaded page from
  dynamic content (3.5 seconds on average to
  generate and then download) to static page (1.5
  seconds on average to download)

58
Capacity ManagementAfter e-Service Deployment

• Managing customers efficiency
• New customers difficult to do
• Better site design as you figure out how new
  customers behave
• Longer-term customers/users should experience a
  learning effect that will cause them to be more
  directed in their activities
• They may consume more content per visit as site
  stickiness keeps them around
• They may consume less content if they become more
  efficient at shopping and other tasks on the site

59
Capacity ManagementAfter e-Service Deployment

• Basic Question Is resource X running?
• Capacity Issue
• Has server failed?
• Has network router failed?
• Method
• Systems performance monitoring tools for each
  infrastructure component
• Tend to monitor and provide information in a
  stovepipe fashion, but useful if you need to know
  about a specific resource
• Note system monitors can report that each
  individual component is running fine, yet the
  system overall can exhibit poor performance

60
Capacity ManagementAfter e-Service Deployment

• Basic Question Is my site running?
• Capacity Issue
• Is capacity gt 0 right now?
• How long has it been 0?
• Method
• Simple site monitor, similar to ping service on
  any computer
• Example
• ArsDigita.coms free Uptime web site monitoring
  service ... Every 15 minutes, it tries to
  download a text file called http//www.mysite.com
  /textfile.txt from your web site
• If it fails, it emails you that your site is down

61
Capacity ManagementAfter e-Service Deployment

• Basic Question What portions of my
  service-product are popular?
• Capacity Issue
• Which files are being requested frequently?
• Which content configurations are requested
  frequently?
• Which processes deliver that content?
• Am I paying too much for my ISP service contract?
  Can I get by on a lower-bandwidth contract?
• Method
• Site log file analysis
• Add up all http transactions made to your web
  site during some time period

62
Capacity ManagementAfter e-Service Deployment

• Log File Analysis
• Analyze http requests stored on the server
• Translate requests into aggregate historical
  demands for certain time period buckets
• Vendors
• Webalizer (www.mrunix.com/webalizer)
• NetTracker (www.sane.com/products/NetTracker)
• many more

63
Capacity ManagementAfter e-Service Deployment

• Customer Profiling (via data mining)
• Translate customer behaviors within system into
  common customer profiles
• Link profiles to the resources that are required
  to service the respective activities

64
Capacity ManagementAfter e-Service Deployment
Network of Site Paths
is made up of ...
Translating the service product into demand loads
Subset of Paths Having Some Relationship
(Business Activities) Within an E-Service
has some business meaning, and is stored in ...
WWW Site Log (http GET, page referrals between
pages )
Data Mining
Subset 1 of paths commonly visited together
Subset 2 of paths commonly visited together
Subset 3 of paths commonly visted together
Three Consumer Types 1, 2, and 3
Subset 1 browse only Subset 2 directly
buy Subset 3 browse, then buy
additional analysis
65
Capacity ManagementAfter e-Service Deployment
WWW Site Content
is made up of ...
Translating the service product into demand loads
Content Located Together Within an E-Service
has some business meaning, and is stored in ...
WWW Site Log (http GET commands for pages visited)
Data Mining
Goods X Services X Content X
Goods Y Services Y Content Y
Goods Z Services Z Content Z
Three Consumer Types X, Y, and Z
additional analysis
66
Capacity ManagementAfter e-Service Deployment

• Basic Question How do popular areas of my site
  affect my capacity?
• Capacity Issue
• How to link common behaviors to specific capacity
  providing resources?
• Methods
• Queueing models
• Load testing tools
• Site monitoring tools

67
Capacity ManagementAfter e-Service Deployment
Identify user navigation paths within e-Service
then use queueing theory equations to determine
long-run impact of various paths
Home Page
0.4
prob. 0.5
0.3
0.6
0.8
0.1
Page 1
Page 2
Enter
Add to Cart
Pay
0.4
prob. 0.3
Page 3
Page 4
0.0
1.0
0.1
prob. 0.2
0.3
Exit
1.0
Search Page
Business Activity Browse
68
Capacity ManagementAfter e-Service Deployment
Three Simple Consumer Behavior Types Observable
in Navigation Model
Identify user navigation paths within e-Service
then use queueing theory equations to determine
long-run impact of various paths
Enter
Home
Exit
Enter
Home
Browse
Exit
Enter
Browse
Add to Cart
Pay
Exit
69
Capacity ManagementAfter e-Service Deployment

• Web Application Performance Monitoring
• Web application monitoring tools work to send up
  red flags when the application under surveillance
  fails to meet its performance objectives

70
Capacity ManagementAfter e-Service Deployment

• Web Performance Monitoring
• Active (synthetic) monitoring
• Identify several key transactions in the
  e-Service
• Create an emulated (or simulated) client for each
  key transaction
• Execute transactions against the simulated client
  on a regular basis
• Measure transactions in detail and collect
  service availability data
• If performance violates some predetermined
  threshold, send an alert to e-Service manager

71
Capacity ManagementAfter e-Service Deployment

• Web Performance Monitoring
• Passive (observational) monitoring
• Collect data from actual end user activity and
  store it in a database
• Analyze data for patterns
• Identify several key transactions in the
  e-Service
• Measure transactions in detail
• Generate performance metrics
• Link performance to activities experienced within
  the e-Service system

72
Capacity ManagementAfter e-Service Deployment
Active
Passive

• Based on actual end user data
• High degree of detail
• Capture client processing time
• Aids in root-cause analysis

• 24 x 7 monitoring
• Constant controlled data collection
• Data collection provides baseline
• Proactive in nature

Advantage

• Data collected opportunistically
• Reactive by nature to real customer problems
• Potential for collecting excessive amount of data

• Not based on true end-user data
• Limited insight into the back-end infrastructure
  performance
• Difficult to execute real transactions
• Creation/maintenance of scripts

Drawback
73
Capacity ManagementAfter e-Service Deployment

• Site Monitoring Software/Services
• Perform Systems Analysis
• Manager must understand the current system
  architecture
• Translate architecture into load testing
  objectives
• Define input data for testing
• Choose a testing/monitoring strategy
• Create Virtual User Scripts
• Record scripts that virtual users will use to
  interact with the e-Service system
• Define User Behaviors
• Specify how the virtual users will behave (random
  think times, browser types to use, dial-up
  speeds, etc.)

74
Capacity ManagementAfter e-Service Deployment

• Site Monitoring Software/Services
• Create a User Scenario
• A scenario is a set of user behaviors combined
  together to test a web site
• Monitor Performance
• Run the user scenario
• Collect data (client, network components,
  hardware components, server/software components)
  on performance for user requests
• Analyze Performance
• Statistics
• Graphs

75
Capacity ManagementAfter e-Service Deployment
76
Capacity ManagementAfter e-Service Deployment

• Prominent Vendors Applications
• Software Products
• Mercury Interactive Topaz
• Segue SilkPerformer
• Services
• Mercury Interactive ActiveWatch (Topaz-based
  service)
• Mercury Interactive ActiveTest (LoadRunner-based
  service)

77
Capacity ManagementAfter e-Service Deployment

• Test on Demand Services
• Deliverables
• Consulting time
• Set of tailored test scripts for an e-Service
• Application test run
• Final report providing details about the tests
• Suggestions for areas of e-Service process/
  infrastructure in need of improvement

78
Capacity ManagementAfter e-Service Deployment
Capacity/Load Test
Monitoring
Development
Deployment
M.I. LoadRunner, etc. M.I. Astra LoadTest,
etc. Segue SilkPerformer
M.I. Topaz
Software Product
Deployment
Deployment
M.I. ActiveWatch GomezNetworks
M.I. ActiveTest
Service
79
Software Demonstration
80
Software DemonstrationMercury Interactive Astra
LoadTest

• Some basic capabilities
• Record customer activities save as script
• Activities that have meaning
• Content consumption service-product subset
• Economic activity checkout system
• Network paths typical customer paths through
  system
• Site and/or service parameters automatically
  iterate through subset of or all possible values
• Multiple-option click boxes
• Drop-down options
• Combine multiple customer activities into a
  demand load scenario

81
Software DemonstrationMercury Interactive Astra
LoadTest

• Some basic capabilities
• Run scenario using actual web site or n-tier
  service process
• Prior to going live
• After going live
• Data collection
• Content transaction
• Process technology monitorsfor typical
• WWW servers hardware
• off-the-shelf WWW component software
• Data analysis
• Averages, stdev, graphs
• Drill-down
• Data export