Building Effective Predictive Models

1
Building Effective Predictive Models

• Ken Dyson
• School of Business

2
Modelling

• Its tricky
• No model is 100 effective
• Assist decision making
• Better informed decisions

3
Building Good Models

• Best model
• Performs the best on unseen, future data
• Process (three steps)
• Train using pre-classified data
• Refine using a test set prevent memorisation
• Evaluate model using a hold-out sample

4
Model Stability

• Performance should be stable
• Over time
• Different conditions
• Remember
• A predictive model is only as good as the data
  used to create it
• Generally more data better model

5
Division of Data

• The split
• Training data 60
• Test set 30
• Hold-out sample 10

6
The density of the Model

• Ratio of interesting events to the sample total
• Crucial when examining performance
• Select data randomly
• Over-sampling more rare (interesting) events
• Use over-sampling were the event is rare

7
Building Time-dependent Models

• Time frames play a critical role
• Both in model development prediction ability
• Time series may have missing data
• Need to build stable time series models
• Incorporate time frames within the data

8
Using Multiple Models

• Why build one model?
• Develop multiple models
• Experiment
• There is no one right way to build models
• No recipe for building the best model
• Try different
• Models
• time frames
• Parameters

9
Summary

• Model consists of
• Training set
• Test set
• Hold-out sample
• The density of the model is important
• Use
• Different models
• Time-sets
• Parameters
• Remember, there is no one solution that works for
  all problems