Bias Variance Analysis of a Real World Problem

1
(No Transcript)
2
(No Transcript)
3
(No Transcript)
4
(No Transcript)
5
(No Transcript)
6
(No Transcript)
7
(No Transcript)
8
(No Transcript)
9
(No Transcript)
10
(No Transcript)
11
(No Transcript)
12
(No Transcript)
13
(No Transcript)
14
(No Transcript)
15
(No Transcript)
16
(No Transcript)
17
Bias Variance Analysis of a Real World Problem

• The problem
• To predict who will be interested in a particular
  insurance product, a caravan policy, and to
  provide an explanation of why people would be
  interested.
• Also to give an explanation of why people would
  buy
• Problem split into prediction and description
  task
• Nature of the problem
• Represented an important class of real world
  problems domains with noisy, correlated,
  redundant and high dimensional data with a weak
  relation between input and target.

18
Methodology

• This problem was presented in the CoIL Challenge
  2000.
• 43 participants in total.
• Bias-Variance Analysis of solutions of the
  participants
• Purpose of study
• To explain the results of different solutions of
  the CoIL competition to better understand the
  factors that determine the success of real world
  data mining projects
• Solution analyzed in terms of
• Feature Construction and Transformation
• Feature selection
• Model representation and the learning method
• Description task of the solution

19
Prediction task

• From a business perspective the goal of the
  prediction task is to rank current customers of
  the insurance company according to the
  probability that they will buy the policy, so
  that the highest ranking customers can be
  contacted through mailing.
• Given that only 6 actually own the policy, a
  regular zero-one loss or classification accuracy
  is not an appropriate evaluation metric.A model
  that predicts no one will buy has a high
  classification accuracy of 94!! But is useless
  for ranking and selecting customers

20
Description task

• The purpose of the description task is to provide
  insight into why customers have a policy.
• This is not necessarily the same as explaining
  the model underlying the prediction the
  description must be from a business perspective.
  The solutions should be explained in real world
  terms.

21
Feature construction and transformation

• Feature construction can reduce bias error by
  relaxing learning bias.
• Features added could allow for models that were
  initially excluded
• It could also change the search bias of a
  learning method
• Feature construction can also reduce the variance
  component of the error

22
Feature Selection

• Removing features that are irrelevant for the
  learner will not change the bias error. As far as
  the learner is concerned irrelevant feature are
  noise. Features are irrelevant for a learner
  either because there is no real relation with the
  target or the learning bias prevents capturing
  this relation.
• Removing relevant features may lead to an
  increase of intrinsic, bias and variance error.
• Risk of over fitting the relation between
  individual features and the target is high and
  eliminating irrelevant features is likely to
  reduce the variance error.

23
Continued

• Feature selection methods used by
  participants can be divided into three main
  categories
• Candidate features are evaluated independently of
  other features. Simple evaluation measures like
  correlation with the target feature are used.
  Feature selection is determined by prior domain
  knowledge as well.
• Subsets of features rather than just evaluate
  features individually and independently.
• Wrapper methods Learner which is used for model
  development is used for feature selection as well.

24
Model representation and learning method.

• An important characteristic of a method is
  therefore the strength and the content of the
  representation (or language) bias.
• Inadequate learning bias of a method can cause
  bias error.
• Strong bias will in general reduce the variance
  error because it forces the learner into a small
  class of models.
• If learning bias is incorrect then this will
  cause bias error.
• The advantage of methods with stronger bias is
  lower variance.
• Models that involve addition of features or of
  constructed predictors can exploit the fact that
  the noise in these predictors will average out
  and therefore the total prediction error will be
  smaller than that of the individual predictions.

25
Bias and Accuracy of CoIL solution methods

26
Accuracy and method Selection