Normalized evaluation scores

1
2008-2009 Magnify360 Computer Science
ClinicOptimizing Individuals' Online
Experience
Evaluation
Treating visitors as individuals
Underlying System
User Experience
For each algorithm, evaluation is done by
comparing its output with the results in the
training data. Its score is the sum over all of
the training data of the result categories shown
in the table below and to the right. We analyzed
the evaluation parameters for sensitivity final
values were both intuitive and insensitive to
slight changes.
Our system has four basic components
User navigates to a site
Conversion
User/site interaction
Magnify360 creates online web presences for many
different client companies. They do not take the
traditional approach of designing a single "best"
site for each client instead, Magnify360 creates
several different sites for each client in order
to appeal to different kinds of visitors. As an
example, here are three of the many versions of
Magnify360s own homepage
Classification
visitor_at_gmail.com
serve page
clickstream data
results
best page
Data
SVM
k-Means
classify user
Evaluation scores
Musician
Musician
Pasadena resident
choose page
Clustering
Evaluation
Insomniac
-0.1
-0.01
-0.001
-0.0001
0
0.1
Evaluation and user-analysis
Online classifier and server
Algorithms' evaluation table (left) and the
sensitivity analysis for the small negative
penalty (right)
An example visitor interaction with a client site
Dependencies and information flow in the
deliverable
Analytical
Visual
Insomniac
When the user navigates to a Magnify360 client,
the clickstream data is sent to our online
classifier. In this example, the system
determines that the user is a musician, and
serves an appropriate version of the target site,
HMC's front page. The conversion, or result, of
the visit is fed back into our evaluation engine.
That evaluation serves to improve the online
classifier and, thus, overall performance.
Magnify360 analyzes visitors' data packets, or
clickstream data, to decide which version of the
website to serve to a new user. Thus, this
project sought an AI system that would augment
and automate Magnify360's current approach for
matching users to sites.

• The data component provides the raw material for
  clustering and evaluation.
• The clustering system trains classification
  algorithms offline (overnight) using previously
  collected data.
• The best clustering system provides the next
  day's online classification to match users to
  suitable sites.
• The results feed back through the evaluation
  system in order to support further refinements.

Results
SVM error margin (e)
Evaluation scores
From clickstream data to profiles
Evaluation scores
This project's data is a large MySQL database
composed of 80 tables and over 100 million rows.
For each visitor the database stores hundreds of
clickstream attributes and a conversion, which is
each client's definition of a successful
interaction, e.g. a purchase, time-on-task, or a
contact email. An example table appears below.
SVM misclassification penalty (c)
Normalized evaluation
SVM parameter sensitivity
Clustering Algorithms
Classification and Testing
SVMs outperformed the best page for profile data,
while k-Means outperformed the best on modeled
continuous data. To preserve this versatility in
a live system with unknown data distribution, all
of the clustering algorithms that we considered
will be included in the final deliverable system.
Decision Trees automatically generate
classifications by subdividing user-profile space
into axis-aligned regions. A users cluster is
then found by navigating through a binary tree
answering yes/no questions.
Raw Data
k-Means
Overview of classification system
Normalized evaluation scores
When a visitor interacts with our system, that
visitor is classified by the current-best
artificial intelligence algorithm, which is
selected from the list that appears at left. In
order to test each algorithm's performance, the
available data is divided into training and
testing subsets. The performance of each approach
on the testing subset determines the next
iteration's best classification algorithm.
Profiles
k-Means is an iterative algorithm that randomly
chooses k centroids and then finds clusters such
that intracluster distances are minimized, and
intercluster distances are maximized.
k-Means number of clusters (k)
k-Means continuous data
visitor_at_gmail.com
Acknowledgments
Conversion
To simplify the data and leverage Magnify360's
insights, our system uses profiles to
characterize each user. Each profile is a bit
that summarizes some of the raw user data, e.g.,
FirefoxUser?, MacUser?, or LatenightVisitor?. In
these terms our system seeks to choose the site
that maximizes the expected conversion for the
profiles of each visitor.
Support Vector Machines (SVM) map data into
higher dimensions, then find the
maximally-separating hyperplanes in that new
space. In the original space, clusters of almost
any shape can result.
Training vs. testing AI classifiers choosing the
best one.
Faculty Advisor Zachary Dodds