Data Warehousing and Data Mining Chapter 3 Data Preprocessing - PowerPoint PPT Presentation

1 / 146
About This Presentation
Title:

Data Warehousing and Data Mining Chapter 3 Data Preprocessing

Description:

2 statistics is approximately distributed as chi-square distribution with (kA-1) ... of hospitals and # of car-theft in a city are correlated ... – PowerPoint PPT presentation

Number of Views:670
Avg rating:3.0/5.0
Slides: 147
Provided by: jiaw196
Category:

less

Transcript and Presenter's Notes

Title: Data Warehousing and Data Mining Chapter 3 Data Preprocessing


1
Data Warehousing and Data Mining Chapter 3
Data Preprocessing
  • Spring
  • 2005

2
Chapter 3 Data Preprocessing
  • Why preprocess the data?
  • Data cleaning
  • Data integration and transformation
  • Data reduction
  • Discretization and concept hierarchy generation
  • Time-dependent data
  • Summary

3
Why Data Preprocessing?
  • Data in the real world is dirty
  • incomplete lacking attribute values, lacking
    certain attributes of interest, or containing
    only aggregate data
  • e.g., occupation
  • noisy containing errors or outliers
  • e.g., Salary-10
  • inconsistent containing discrepancies in codes
    or names
  • e.g., Age42 Birthday03/07/1997
  • e.g., Was rating 1,2,3, now rating A, B, C
  • e.g., discrepancy between duplicate records

4
Why Is Data Dirty?
  • Incomplete data comes from
  • n/a data value when collected
  • different consideration between the time when the
    data was collected and when it is analyzed.
  • human/hardware/software problems
  • Noisy data comes from the process of data
  • collection
  • entry
  • transmission
  • Inconsistent data comes from
  • different data sources
  • functional dependency violation

5
Why Is Data Preprocessing Important?
  • No quality data, no quality mining results!
  • Quality decisions must be based on quality data
  • e.g., duplicate or missing data may cause
    incorrect or even misleading statistics.
  • Data warehouse needs consistent integration of
    quality data
  • Data extraction, cleaning, and transformation
    comprises the majority of the work of building a
    data warehouse

6
Multi-Dimensional Measure of Data Quality
  • A well-accepted multidimensional view
  • Accuracy
  • Completeness
  • Consistency
  • Timeliness
  • Believability
  • Value added
  • Interpretability
  • Accessibility
  • Broad categories
  • intrinsic, contextual, representational, and
    accessibility.

7
Major Tasks in Data Preprocessing
  • Data cleaning
  • Fill in missing values, smooth noisy data,
    identify or remove outliers, and resolve
    inconsistencies
  • Data integration
  • Integration of multiple databases, data cubes, or
    files
  • Data transformation
  • Normalization and aggregation
  • Data reduction
  • Obtains reduced representation in volume but
    produces the same or similar analytical results
  • Data discretization
  • Part of data reduction but with particular
    importance, especially for numerical data

8
Forms of data preprocessing
9
Chapter 3 Data Preprocessing
  • Why preprocess the data?
  • Data cleaning
  • Data integration and transformation
  • Data reduction
  • Discretization and concept hierarchy generation
  • Time-dependent data
  • Summary

10
Data Cleaning
  • Importance
  • Data cleaning is one of the three biggest
    problems in data warehousingRalph Kimball
  • Data cleaning is the number one problem in data
    warehousingDCI survey
  • Data cleaning tasks
  • Fill in missing values
  • Identify outliers and smooth out noisy data
  • Correct inconsistent data
  • Resolve redundancy caused by data integration

11
Missing Data
  • Data is not always available
  • E.g., many tuples have no recorded value for
    several attributes, such as customer income in
    sales data
  • Missing data may be due to
  • equipment malfunction
  • inconsistent with other recorded data and thus
    deleted
  • data not entered due to misunderstanding
  • certain data may not be considered important at
    the time of entry
  • not register history or changes of the data
  • Missing data may need to be inferred.

12
How to Handle Missing Data?
  • Ignore the tuple usually done when class label
    is missing (assuming the tasks in
    classificationnot effective when the percentage
    of missing values per attribute varies
    considerably.
  • Fill in the missing value manually tedious
    infeasible?
  • Use a global constant to fill in the missing
    value e.g., unknown, a new class?!
  • Use the attribute mean,median or mode to fill in
    the missing value
  • Use the attribute mean for all samples belonging
    to the same class to fill in the missing value
    smarter
  • Use the most probable value to fill in the
    missing value inference-based such as Bayesian
    formula or decision tree

13
Most Probable Value
  • Use the most probable value to fill in the
    missing value inference-based such as Bayesian
    formula or decision tree
  • develop a submodel to predict the category or
    numerical value of the missing case
  • income f(education,sex, age,...)
  • A neural network model
  • K-NN model
  • Decision tree
  • Regression
  • ...

14
An Extreme Case
  • Suppose all values of an attribute is missing
  • This attribute may be considered as the unknown
    label in unsupervised learning
  • Assigning an object into a cluster can be thought
    of filling the missing values of an attribute

15
Time Series and Spacial Data
  • Filling missing values in time series and
    spatial data needs a different tratement

Stock index
x
x data exisits o missing
x
x
o
x
o
x
x
o
x
Time in days
16
Noisy Data
  • Noise random error or variance in a measured
    variable
  • Incorrect attribute values may due to
  • faulty data collection instruments
  • data entry problemshuman or computer error
  • data transmission problems
  • technology limitation
  • inconsistency in naming convention
  • duplicate records
  • Some noice is inavitable and due to variables or
    factors that can not be measured

17
How to Handle Noisy Data?
  • Binning method
  • first sort data and partition into (equi-depth)
    bins
  • then one can smooth by bin means, smooth by bin
    median, smooth by bin boundaries, etc.
  • Clustering
  • detect and remove outliers
  • Combined computer and human inspection
  • detect suspicious values and check by human
  • Regression
  • smooth by fitting the data into regression
    functions

18
Simple Discretization Methods Binning
  • Equal-width (distance) partitioning
  • It divides the range into N intervals of equal
    size uniform grid
  • if A and B are the lowest and highest values of
    the attribute, the width of intervals will be W
    (B-A)/N.
  • The most straightforward
  • But outliers may dominate presentation
  • Skewed data is not handled well.
  • Equal-depth (frequency) partitioning
  • It divides the range into N intervals, each
    containing approximately same number of samples
  • Managing categorical attributes can be tricky.

19
Binning Methods for Data Smoothing
  • Sorted data for price (in dollars) 4, 8, 9,
    15, 21, 21, 24, 25, 26, 28, 29, 34
  • Partition into (equi-depth) bins
  • - Bin 1 4, 8, 9, 15
  • - Bin 2 21, 21, 24, 25
  • - Bin 3 26, 28, 29, 34
  • Smoothing by bin means
  • - Bin 1 9, 9, 9, 9
  • - Bin 2 23, 23, 23, 23
  • - Bin 3 29, 29, 29, 29
  • Smoothing by bin boundaries
  • - Bin 1 4, 4, 4, 15
  • - Bin 2 21, 21, 25, 25
  • - Bin 3 26, 26, 26, 34

20
Cluster Analysis
21
Regression
y
Y1
y x 1
Y1
x
X1
22
Example use of regression to determine outliers
  • Problem predict the yearly spending of AE
    customers from their income.
  • Y yearly spending in YTL
  • X average montly income in YTL

23
Data set
X income, Y spending First data is probabliy an
outlier But examining only spending data that is
Distribution of Y 200 spending of the
first Customer can not be identified as an
outlier
  • X Y
  • 50 200
  • 100 50
  • 200 70
  • 250 80
  • 300 85
  • 400 120
  • 500 130
  • 700 150
  • 750 150
  • 800 180
  • 1000 200

24
Regression
y
Y1200
y 0.15x 20
Y1
x
X150
X101000
Examine residuals Y1-Y1 is high But Y10-Y10 is
low So data point 1 is an outlier
inccome
25
Notes
  • Some methods are used for both smoothing and data
    reduction or discretization
  • binning
  • used in decision tress to reduce number of
    categories
  • concept hierarchies
  • Example price a numerical variable
  • in to concepts as expensive, moderately prised,
    expensive

26
Inconsistent Data
  • Inconsistent data may due to
  • faulty data collection instruments
  • data entry problemshuman or computer error
  • data transmission problems
  • Chang in scale over time
  • 1,2,3 to A, B. C
  • inconsistency in naming convention
  • Data integration
  • Different units used for the same variable
  • TL or dollar
  • Value added tax included in one source not in
    other
  • duplicate records

27
Chapter 3 Data Preprocessing
  • Why preprocess the data?
  • Data cleaning
  • Data integration and transformation
  • Data reduction
  • Discretization and concept hierarchy generation
  • Time-dependent data
  • Summary

28
Data Integration
  • Data integration
  • combines data from multiple sources into a
    coherent store
  • Schema integration
  • integrate metadata from different sources
  • Entity identification problem identify real
    world entities from multiple data sources, e.g.,
    A.cust-id ? B.cust-
  • Detecting and resolving data value conflicts
  • for the same real world entity, attribute values
    from different sources are different
  • possible reasons different representations,
    different scales, e.g., metric vs. British units
  • price may include value added tax in one source
    and not include in the other
  • Name are entered by different convensions
  • name lastname or lastname name or ...

29
Handling Redundant Data in Data Integration
  • Redundant data occur often when integration of
    multiple databases
  • The same attribute may have different names in
    different databases
  • One attribute may be a derived attribute in
    another table, e.g., annual revenue
  • Redundant data may be able to be detected by
    correlational analysis
  • rx,y ?ni1(xi-mean_x)(yi-mean_y)/(n-1)?x?y
  • where ?x,?y are standard deviation of X and Y
  • as r?1 positive correlation x? y? or x? y?, one
    is redundant
  • r ? 0, X and X are independent
  • r ? -1 negative correlation x? y? or x? y? one
    is redundant

30
Positively and Negatively Correlated Data
31
Not Correlated Data
32
Exercise
  • Construct a data set or a functional relationship
    of two variables X and Y where there is a perfect
    relation between X and Y
  • knowing value of X, Y corresponding to that X can
    be predicted with no error
  • but correlation coefficient between X and Y is
  • ZERO

33
Handling Redundant Data in Data Integration
  • how to detect redundancy for categorical
    variables?
  • Find a measure of association for categorical
    variables
  • Careful integration of the data from multiple
    sources may help reduce/avoid redundancies and
    inconsistencies and improve mining speed and
    quality

34
Correlation Analysis (Categorical Data)
  • ?2 (chi-square) test
  • The larger the ?2 value, the more likely the
    variables are related
  • The cells that contribute the most to the ? 2
    value are those whose actual count is very
    different from the expected count

35
Chi-Square Test
  • Under the null hypothesis that attribute valuea
    are independent or not correlated
  • ? 2 statistics is approximately distributed as
    chi-square distribution with (kA-1)(kB-1) degree
    of freedom
  • KA and KB are number of distinct values of
    attribute A and B
  • Given a confidence level ?
  • Reject null hypothesis of no correlation if
  • ? 2gt ? 2(df,?)

36
Chi-Square Calculation An Example
  • ? 2 (chi-square) calculation (numbers in
    parenthesis are expected counts calculated based
    on the data distribution in the two categories)
  • ?2((2-1)(2-1),0.05)5.99lt507.93
  • It shows that like_science_fiction and play_chess
    are correlated in the group

37
Calculating expected frequencies
  • Expected values are computed under the
    independence assumption
  • E(i,j)pipjN
  • pipjN probability of observing variable A at
    state i
  • pj probability of observing variable B at state j
  • N total number of observations
  • E.g. E(like_scfic,play_chess)
    (450/1500)(300/1500)150090

38
Exercises
  • What is the limitation of chi-square measure of
    correlation between categorical variables
  • Suggest a better way of measuring correlation
    between categorical variables
  • Suggest a way of measuring association between a
    categorical and a numerical variable

39
Correlation and Causation
  • Correlation does not imply causality
  • of hospitals and of car-theft in a city are
    correlated
  • Both are causally linked to the third variable
    population

40
Exercise
  • Correlation coefficient show degree of linear
    association between two variables say X and Y
  • Suppose in the data set there is a third variable
    Z potentially correlated with X and Y
  • Then the association between X and Y is affected
    by the presence of the third variable Z
  • E.g. three variables
  • Y crop yield, Xrainfall, Z temperature
  • Assume
  • No association between Y and X
  • Positive association between Y and Z and
  • Negative association between X and Z

41
Exercise cont.
  • How do you interpret the associations
  • Design such a dataset
  • So the simple correlation coefficient between X
    and X is misleading
  • Contradicts with common sense knowledge
  • Develop an association measure showing the
    association between Y and X elliminating the
    effect of Z(third variable) on both X and Y

42
Another attribute relevance measureEntropy and
Information Gain
  • S contains si tuples of class Ci for i 1, ,
    m
  • Information measures info required to classify
    any arbitrary tuple
  • Entropy of attribute A with values a1,a2,,av
  • Information gained by branching on attribute A

43
Entropy and Information Gain
  • Choose the attributes as relevant
  • Whose information gains is greater than a
    critical value

44
Exercise
  • Construct a simple data set of two categorical
    attributes A and B such that
  • Knowing the values of A provides perfect
    information to predict B but
  • Knowing the values of B does not provide any
    information to precict A
  • Entroy is not a symetric measure of association
    like correlation coefficient is

45
Exercise
  • Consider a data set of three attributes A, B, and
    C. A has two distinct values as positive p
    and negative n and can be considered as the
    class label. The unconditional probabilities of p
    and n are 0.5 each. B and C have four and three
    distinct values as b1 b2 b3 b4 and c1 c2 c3 each
    having unconditional probabilities ¼ and 1/3
    .respectively. Construct a data set such that B
    is perfectly relevant in predicting the class
    type A but C is perfectly irrelevant for the
    prediction of A.

46
Data Transformation
  • Smoothing remove noise from data
  • Aggregation summarization, data cube
    construction
  • From daily sales to monthly sales
  • Generalization concept hierarchy climbing
  • From original ages to categories
    young,middle,senior
  • Normalization scaled to fall within a small,
    specified range
  • min-max normalization
  • z-score normalization
  • normalization by decimal scaling
  • Attribute/feature construction
  • New attributes constructed from the given ones
  • profit, financial ratios,

47
Data Transformation Normalization
  • min-max normalization
  • When a score is outside the new range the
    algorithm may give an error or warning message
  • in the presence of outliers regular observations
    are squized in to a small interval

48
Data Transformation Normalization
  • z-score normalization

Good in handling outliers as the new range is
between -?to ? no out of range error
49
Data Transformation Normalization
  • Decimal scaling

Where j is the smallest integer such that Max(
)lt1
  • Ex max v984 then j 3 v0.984
  • preserves the appearance of figures

50
Data TransformationLogarithmic Transformation
  • Logarithmic transformations
  • Y logY
  • Used in some distance based methods
  • Clustering
  • For ratio scaled variables
  • E.g. weight, TL/dollar
  • Distance between objects is related to percentage
    changes rather then actual differences

51
Linear transformations
  • Note that linear transformations preserve the
    shape of the distribution
  • Whereas non linear transformations distors the
    distribution of data
  • Example
  • Logistic function f(x) 1/(1exp(-x))
  • Transforms x between 0 and 1
  • New data are always between 0-1

52
Exercise
  • Device a transformation such that new data are
    always between 0-1 interval
  • but preserve the shape of the sample distribution
    as much as possible

53
Attribute/feature construction
  • Automatic attribute generation
  • Using product or and opperations
  • E.g. a regression model to explain spending of a
    customer shown by Y
  • Using X1 age, X2 income, ...
  • Y a b1X1 b2X2 a linear model
  • a,b1,b2 are parameters to be estimated
  • Y a b1X1 b2X2 cX1X2
  • a nonlinear model a,b1,b2,c are parameters to be
    estimated but linear in parameters as
  • X1X2 can be directly computed from income and
    age

54
Ratios or differences
  • Define new attributes
  • E.g.
  • Real variables in macroeconomics
  • Real GNP
  • Financial ratios
  • Profit revenue cost

55
Chapter 3 Data Preprocessing
  • Why preprocess the data?
  • Data cleaning
  • Data integration and transformation
  • Data reduction
  • Discretization and concept hierarchy generation
  • Time-dependent data
  • Summary

56
Data Reduction Strategies
  • Warehouse may store terabytes of data Complex
    data analysis/mining may take a very long time to
    run on the complete data set
  • Data reduction
  • Obtains a reduced representation of the data set
    that is much smaller in volume but yet produces
    the same (or almost the same) analytical results
  • Data reduction strategies
  • Data cube aggregation
  • Dimensionality reductionremove unimportant
    attributes
  • Data Compression
  • Numerosity reductionfit data into models
  • Discretization and concept hierarchy generation

57
Credit Card Promotion Database
58
Data Reduction
  • Eliminate redandent attributes
  • correlation coefficients show that
  • Watch and magazine are associated
  • Eliminate one
  • Combine variables to reduce number of independent
    variables
  • Principle component analysis
  • Sampling reduce number of cases (records) by
    sampling from secondary storage
  • Discretization
  • Age to categorical values

59
Data Cube Aggregation
  • The lowest level of a data cube
  • the aggregated data for an individual entity of
    interest
  • e.g., a customer in a phone calling data
    warehouse.
  • Multiple levels of aggregation in data cubes
  • Further reduce the size of data to deal with
  • Reference appropriate levels
  • Use the smallest representation which is enough
    to solve the task
  • Queries regarding aggregated information should
    be answered using data cube, when possible

60
Dimensionality Reduction
  • Feature selection (i.e., attribute subset
    selection)
  • Select a minimum set of features such that the
    probability distribution of different classes
    given the values for those features is as close
    as possible to the original distribution given
    the values of all features
  • reduce of patterns in the patterns, easier to
    understand
  • Heuristic methods (due to exponential of
    choices)
  • step-wise forward selection
  • step-wise backward elimination
  • combining forward selection and backward
    elimination
  • decision-tree induction

61
Example for irrelevent features
Average spending of a customer by income and age
Droping age Expain spending by Income only
Does not change results significantly
62
Example of Decision Tree Induction
Initial attribute set A1, A2, A3, A4, A5, A6
A4 ?
A6?
A1?
Class 2
Class 2
Class 1
Class 1
Reduced attribute set A1, A4, A6
63
Heuristic Feature Selection Methods
  • There are 2d possible sub-features of d features
  • Several heuristic feature selection methods
  • Best single features under the feature
    independence assumption choose by significance
    tests.
  • Best step-wise feature selection
  • The best single-feature is picked first
  • Then next best feature condition to the first,
    ...
  • Step-wise feature elimination
  • Repeatedly eliminate the worst feature
  • Best combined feature selection and elimination
  • Optimal branch and bound
  • Use feature elimination and backtracking

64
Example Economic indicator problem
  • There are tens of macroeconomic variables
  • say totally 45
  • Which ones is the best predictor for inflation
    rate three months ahead?
  • Develop a simple model to predict inflation by
    using only a couple of those 45 macro variables
  • Best-stepwise feature selection
  • The single macro variable predicting inflation
    among the 45 is seleced first try 45 models say
    /TL
  • repeat
  • The k th variable is entered among 45-(k-1)
    variables
  • Stop at some point introducing new variables

65
Example cont.
  • Best feature elimination
  • Develop a model including all 45 variables
  • Remove just one of them try 45 models each
    excluding just one out of 45 variables
  • repeat
  • Continue eliminating a new variable at each step
  • Unitl a stoping criteria
  • Rearly used comared to feature selection

66
Principal Component Analysis
  • Given N data vectors from k-dimensions, find c lt
    k orthogonal vectors that can be best used to
    represent data
  • The original data set is reduced to one
    consisting of N data vectors on c principal
    components (reduced dimensions)
  • Each data vector is a linear combination of the c
    principal component vectors
  • Works for numeric data only
  • Used when the number of dimensions is large

67
Principal Component Analysis
X2
Y1
Y2

X1

68
  • X Nk matrix of data in original form Xj is j th
    column of data matrix X where j 1,.,k
  • xij is the j th variable of i th observation
  • First normalize all variables X1..Xk to Z1.. Zk
  • such that mean of Zj j1,..,k 0 E(Zi) 0
  • If the scales of variables are differnt bring the
    scales of all variables to same units apply
  • Zi (Xi-mean_X)/std_X

69
  • The first principle component a1 is a linear
    combination of the data Z
  • y1 Za1 where y1 is N1 vector Z is NK and
    ak1
  • y1,1 z1,1a1,1 z1,2a2,1... z1,kak,1
  • y2,1 z2,1a1,1 z2,2a2,1... z2,kak,1

70
Za1 y1
71
  • a1 is a unit vector that is norm of a1 1
  • a1a1 1 or a1,12 a2,12... ak,121
  • Find the principle comp. vector a1 so as to get
    the maximum variation in y1
  • maximize the variance of y1
  • var(y1) var(y1,1) var(y2,1)... var(yN,1)
  • var(y1,1)i(y1,1-mean_y1)2
  • note that mean_y10
  • variation is y1 is y1,12 y2,12... yN,12
  • or in matrix notation (Za1)Za1 a1ZZa1
  • dimensions1k kN Nk k1 11 a scaler

72
a1ZZa1 y21
73
  • Z 1, 1.5, -0.5, 1
  • 2, 0.5, -1 ,-1.5
  • Z 1 2
  • 1.5 0.5
  • -0.5 -.1
  • 1 -1.5
  • ZZ 4.5 1.75
  • 4 7.5

74
The maximization problem
  • Max variation in y
  • subject to the restriction that a1a1
  • or max a1ZZa1
  • st a1a1
  • find a1 a1,1 a2,1 ak,1 so as to maximize the
    variation in the first transformed vector y1
  • Form the lagranien
  • L a1ZZa1 -?(a1a1-1)
  • Variation in y 0 constrain

75
  • L a1ZZa1 -?(a1a1-1)
  • dL/da0
  • take derivative of Lagrenian with respect to a
    and
  • dL/da 2ZZa1 -2?a10 or
  • ZZa1 ?a1 so
  • multiplying by a1
  • a1ZZa1 ? a1a1 or
  • a1ZZa1 ?
  • a1 is the first eigen vector of ZZ ,? is the
    maximum variation is the first eigen value of ZZ

76
Eigen Value problem
  • Ax ?x
  • Where A is an n by n matrix
  • X is an n by 1 vector
  • ? is a constant
  • Ax is multiplication of a matrix by a vector
  • A transformation from Rn to Rn
  • Find such vectors whose transformations are
    multiples of themselves

77
  • Ax- ?x 0
  • (A- ?I)x 0
  • Two unkonwns x and ?
  • But a homogenuous equation
  • X0 is a trivial solution
  • Rank A- ?I less then n
  • Determinant A- ?I should be zero

78
The Second principle component a2
  • The second principle component a2 can be found in
    a similar manner y2 Za2 where
  • y2,1 z1,1a1,2 z1,2a2,2... z1,kak,2
  • y2,2 z2,1a2,1 z2,2a2,2... z2,kak,2
  • and so on
  • a2 is a unit vector so a2a21 and orthogonal to
    a1

79
  • Max variation in y
  • subject to the restriction that a2a21 and
    a1a20
  • max a2ZZa2
  • st a2a21 and a1a20 the first and
    second PC unit vectors are orthogonal to each
    other
  • find a2 a2,1 a2,2 a2,k so as to maximize the
    variation in the transformed compnents y2
  • L a2ZZa2 -?2(a2a2-1) - ?a1a2

80
The Second principle component a2
  • L a2ZZa2 -?2(a2a2-1) - ?a1a2
  • dL/da20
  • take derivative of lagrenian with respect to a2
    and
  • dL/da 2ZZa2 -2?a2- ?a10 or
  • ZZa2 ?a2 ?a1
  • multiplying by a2
  • a2ZZa2 ? a2a2 ?a2 a1
  • a2ZZa2 ?
  • Variantion in second direction

81
  • Multiply by a1
  • a1ZZa2 ? a1a2 ?a1 a1
  • a1ZZa2 ? 0 as
  • a1ZZ ?a1 so ?a1a2 0 ?
  • Remember that
  • ZZa2 ?a2 ?a1
  • ZZa2 ?a2
  • the second transformation vector is the second
    eigen vector variation of the second principle
    component is the second eigen value ?2

82
C th principle component
  • Notice that ?2lt ?1
  • the first principle component is found in the
    direction of maximum variation
  • the second has the second maximum variation whose
    direction is orthogonal to the first PC
  • continue the optimization procedure until
    finining c lt k components
  • R(c) ?ci1 ?i/ ?ki1 ?i
  • Variation in c components/total variation
  • R(c) gt specified limit

83
Principal Component Analysis
X2
Y1
Y2

X1

84
Principal Component Analysis
X2
Y1
Y2
  • Coordinate of Z1 is
  • Z1,Z2 in
  • when using principle
  • components
  • Z y1,y2 expressed in
  • unit vectors a1 a2
  • a1
  • a2

Z1
85
Exercise
  • Z
  • (-0.5,0.5),(1,1.5), (2.5,1), (-1,-1.5),
    (-1.5,-1.5)
  • N5 k2
  • plot the point in Z1 Z2 axis
  • find the first principle component
  • without a matrix formulation
  • with a matrix formulation
  • express the data in terms of first PC
  • find the second principle component
  • Draw on a figure the principle components

86
Histograms
  • A popular data reduction technique
  • Divide data into buckets and store average (sum)
    for each bucket
  • Can be constructed optimally in one dimension
    using dynamic programming
  • Related to quantization problems.

87
  • Equiwidht the width of each bucket range is
    uniform
  • Equidepth (equiheight)each bucket contains
    roughly the same number of continuous data
    samples
  • V-Optimalleast variance
  • histogram variance is the is a weighted sum of
    the original values that each bucket represents
  • bucket weight values in bucket

88
MaxDiff
  • MaxDiff make a bucket boundry between adjacent
    values if the difference is one of the largest k
    differences
  • xi1 - xi gt max_k-1(x1,..xN)

89
V-optimal design
  • Sort the values
  • assign equal number of values in each bucket
  • compute variance
  • repeat
  • change buckets of boundary values
  • compute new variance
  • until no reduction in variance
  • variance (n1Var1n2Var2...nkVark)/N
  • N n1n2..nk,
  • Vari ?nij1(xj-x_meani)2/ni,
  • Note that V-Optimal in one dimension is
    equivalent to K-means clustering in one dimension

90
Clustering
  • Partition data set into clusters, and one can
    store cluster representation only
  • Can be very effective if data is clustered but
    not if data is smeared
  • Can have hierarchical clustering and be stored in
    multi-dimensional index tree structures
  • There are many choices of clustering definitions
    and clustering algorithms, further detailed in
    Chapter 8

91
Sampling
  • Allow a mining algorithm to run in complexity
    that is potentially sub-linear to the size of the
    data
  • Choose a representative subset of the data
  • Simple random sampling may have very poor
    performance in the presence of skew
  • Develop adaptive sampling methods
  • Stratified sampling
  • Approximate the percentage of each class (or
    subpopulation of interest) in the overall
    database
  • Used in conjunction with skewed data
  • Sampling may not reduce database I/Os (page at a
    time).

92
Sampling methods
  • Simple random sample without replacement (SRSWOR)
    of size n
  • n of N tuples from D nltN
  • P(drawing any tuple)1/N all are equally likely
  • Simple random sample with replacement (SRSWR) of
    size n
  • each time a tuple is drawn from D, it is recorded
    and then replaced
  • it may be drawn again

93
Sampling methods cont.
  • Cluster Sample if tuples in D are grouped into M
    mutually disjoint clusters then an SRS of m
    clusters can be obtained where m lt M
  • Tuples in a database are retrieved a page at a
    time
  • Each pages can be considered as a cluster
  • Stratified Sample if D is divided into mutually
    disjoint parts called strata.
  • Obtain a SRS at each stratum
  • a representative sample when data are skewed
  • Ex customer data a stratum for each age group
  • The age group having the smallest number of
    customers will be sure to be presented

94
Confidence intervals and sample size
  • Cost of obtaining a sample is propotional to the
    size of the sample, n
  • Specifying a confidence interval and
  • you should be able to determine n number of
    samples required so that the sample mean will be
    within the confidence interval with (1-p)
    confident
  • n is very small compared to the size of the
    database N
  • nltltN

95
Sampling
SRSWOR (simple random sample without
replacement)
SRSWR
96
Sampling
Cluster/Stratified Sample
Raw Data
97
Hierarchical Reduction
  • Use multi-resolution structure with different
    degrees of reduction
  • Hierarchical clustering is often performed but
    tends to define partitions of data sets rather
    than clusters
  • Parametric methods are usually not amenable to
    hierarchical representation
  • Hierarchical aggregation
  • An index tree hierarchically divides a data set
    into partitions by value range of some attributes
  • Each partition can be considered as a bucket
  • Thus an index tree with aggregates stored at each
    node is a hierarchical histogram

98
Chapter 3 Data Preprocessing
  • Why preprocess the data?
  • Data cleaning
  • Data integration and transformation
  • Data reduction
  • Discretization and concept hierarchy generation
  • Time-dependent data
  • Summary

99
Discretization
  • Three types of attributes
  • Nominal values from an unordered set
  • Ordinal values from an ordered set
  • Continuous real numbers
  • Discretization
  • divide the range of a continuous attribute into
    intervals
  • Some classification algorithms only accept
    categorical attributes.
  • Reduce data size by discretization
  • Prepare for further analysis

100
Discretization and Concept hierachy
  • Discretization
  • reduce the number of values for a given
    continuous attribute by dividing the range of the
    attribute into intervals. Interval labels can
    then be used to replace actual data values.
  • Types of discretization
  • Supervised (class information used)
  • unsupervised (not used or not available)
  • dynamic - static
  • local - global
  • top down - bottom up

101
Concept hierarchies
  • reduce the data by collecting and replacing low
    level concepts (such as numeric values for the
    attribute age) by higher level concepts (such as
    young, middle-aged, or senior).
  • detail is lost
  • generalized data
  • more meaningful
  • easier to interpret,
  • require less space
  • mining on a reduced data set require
  • fewer input/output opperations
  • be more efficient

102
Discretization Process
  • Univariate - one feature at a time
  • Steps of a typical discritization process
  • sorting the continuous values of the feature
  • evaluating a cut-point for splitting or adjacent
    intervals for merging
  • splitting or merging intervals of continuous
    value
  • top-down approach intervals are split
  • chose the best cut point to split the values
    into two partitions
  • until a stopping criterion is satisfied

103
Discretization Process cont.
  • bottom-up approach intervals are merged
  • find best pair of intervals to merge
  • until stopping criterion is satisfied
  • stopping at some point
  • lower better understanding less accuracy
  • high poorer understanding higher accuracy
  • Stopping criteria
  • number of intervals reach a critical value
  • number of observations in each interval exceeds a
    value
  • all observations in an interval are of the same
    class
  • a minimum information gain is not satisfied

104
Generalized Splitting Algorithm
  • S sorted values of feature f
  • splitting(S)
  • if stopping criterion satisfied
  • return
  • T GetBestSplitPoint(S)
  • S1GetLeftPart(S,T)
  • S2GetRightPart(S,T)
  • Splitting(S1)
  • Splitting(S2)

105
Discretization and concept hierarchy generation
for numeric data
  • Binning (see sections before)
  • Histogram analysis (see sections before)
  • Clustering analysis (see sections before)
  • Entropy-based discretization
  • Segmentation by natural partitioning

106
1R a supervised discretization method
  • After sorting the continuous values,
  • divides the range of continuous values into a
    number of disjoint intervals and adjusts the
    boundaries based on the class labels
  • Example
  • 11 14 15 18 19 20 21 ? 22 23 25 30 31 33 35 ? 36
  • R C C R C R C ? R C C R C R
    C ? R
  • C ? C
    ? R
  • interval of class C from 11-21
  • another interval of C from 22-35
  • last of class R including just 36
  • the two leftmost intervals are merged as they
    predict the same class
  • stoping each interval should contain a
    prespecified minimum number of instances

107
Entropy
  • A measure of (im)purity of an sample variable S
    is defined as
  • Ent(S) -?spslog2ps
  • s is a value of S
  • Ps is its estimated probability
  • average amount of information per event
  • Information of an event is
  • I(s) -log2ps
  • information is high for lower probable events and
    low otherwise

108
Entropy-Based Discretization
  • Given a set of samples S, if S is partitioned
    into two intervals S1 and S2 using boundary T,
    the entropy after partitioning is
  • The boundary that minimizes the entropy function
    over all possible boundaries is selected as a
    binary discretization.
  • The process is recursively applied to partitions
    obtained until some stopping criterion is met,
    e.g.,
  • Experiments show that it may reduce data size and
    improve classification accuracy

109
Age Splitting Example
  • A sample of ages for 15 customers
  • 19 27 29 35 38 39 40 41 42 43 43 43 45 55 55
  • y n y y y y y y n y y n n
    n n
  • classes are y or n response to life insurance
  • Responding to life promotion (y or n) v.s. Age of
    customer
  • y or n yes or no
  • entropy of the whole sample without partitioning
  • ent(S)(-6/15)log2(6/15)(-9/15)log2(9/15)
  • -0.4(-1.322)(-0.6)(-0.734)0.969

110
Age Splitting Example
  • Splitting value of age41.5
  • S1 agelt41.5 7 y, 1 n
  • S2 Tgt 41.5 2 y, 5 n
  • I(S,T)S1/Sent(S1)S2/Sent(S2)
  • (8/15)ent(S1)(7/15)ent(S2)
  • ent(S1)(-1/8)log2(1/8)(-7/8)log2(7/8)
  • -0.125(-3.000)(-0.875(-0.193)0.544
    ent(S2)(-2/7)log2(2/7)(-5/7)log2(5/7)
  • -0.286(-1.806)(-0.714(-0.486)0.863
  • ent(S1,S2) (8/15)0.544(7/15)0.8630.692

111
Age Splitting Example
  • Information gain
  • 0.969-0.6920.277
  • try all possible partitions
  • chose the partition providing maximum information
    gain
  • then reccursively partition each interval
  • until the stoping criteria is satisfied
  • when information gain lt a treshold value

112
Exercise
  • Consider a data set of two attributes A and B. A
    is continuous, whereas B is categorical, having
    two values as y and n, which can be
    considered as class of each observation. When
    attribute A is discretized into two equiwidth
    intervals no information is provided by the class
    attribute B but when discretized into three
    equiwidth intervals there is perfect information
    provided by B. Construct a simple dataset obeying
    these characteristics.

113
Segmentation by natural partitioning
  • 3-4-5 rule can be used to segment numeric data
    into
  • relatively uniform, natural intervals.
  • If an interval covers 3, 6, 7 or 9 distinct
    values at the most significant digit, partition
    the range into 3 equi-width intervals (equiwith
    intervals for 3,6,9)
  • 3 intervals in grouping of 2-3-2 for 7
  • If it covers 2, 4, or 8 distinct values at the
    most significant digit, partition the range into
    4 intervals
  • If it covers 1, 5, or 10 distinct values at the
    most significant digit, partition the range into
    5 intervals

114
Example of 3-4-5 rule
(-4000 -5,000)
Step 4
115
Example 3.5
  • profits at different branches of AE
  • Min-351,976.00 Max4,700,896.50
  • 5th percentile -159,876
  • 95th percentile 1,838,761
  • low-159,876, high1,838,761
  • msd1,000,000
  • rounding low to milliondigit low-1,000,000
  • rounding high to milliondigit high2,000,000
  • (2,000,000-(-1,000,000))/1,000,0003
  • partitioned into 3 equiwidth subsegments
  • (-1,000,000...0,(0...1,000,000,(1,000,000...2,00
    0,000

116
Example 3.5
  • First interval
  • low(-1,000,000)ltmin
  • adjust the left boundary
  • msd of min 100,000
  • rounding min to -400,000 min
  • first interval(-400,000...0
  • last interval (1,000,000...2,000,000 not cover
    max
  • max lthigh
  • a new interval (2,000,000...5,000,000
  • rounding max to 5,000,000

117
Example 3.5
  • Recursively each interval can be further
    partitioned according to 3-4-5 rule
  • first interval (-400,00...0 partitioned into 4
    subintervals
  • (-400,000...-300,000,(-300,000...-200,000,
  • (-200,000...-100,000,(-100,000...0
  • second interval (0...1,000,000 partitioned into
    5 subintervals
  • (0...200,000,(200,000...400,000,
  • (400,000...600,000,(600,000...800,000
  • (800,000...1,000,000
  • last interval into 3 subintervals
  • (2,000,000..3,000,000(3,000,000..4,000,000,(4,00
    0,000..5,000,000

118
Concept hierarchy generation for categorical data
  • finite but possibly large distinct values with no
    ordering
  • geographic location,job categories, item type
  • Specification of a partial ordering of attributes
    explicitly at the schema level by users or
    experts
  • Specification of a portion of a hierarchy by
    explicit data grouping
  • Specification of a set of attributes, but not of
    their partial ordering
  • Specification of only a partial set of attributes

119
Specification of a set of attributes
  • Concept hierarchy can be automatically generated
    based on the number of distinct values per
    attribute in the given attribute set. The
    attribute with the most distinct values is placed
    at the lowest level of the hierarchy.

15 distinct values
country
65 distinct values
province_or_ state
3567 distinct values
city
674,339 distinct values
street
120
Chapter 3 Data Preprocessing
  • Why preprocess the data?
  • Data cleaning
  • Data integration and transformation
  • Data reduction
  • Discretization and concept hierarchy generation
  • Time-dependent data
  • Summary

121
Time Series Data
  • A time series database consists of sequences of
    values or events changing with time.
  • The values are typically measured at equal time
    intervals
  • Examples
  • daily closing values of stock market index
  • sales of products
  • economic variables GNP, exchange rates...

122
Mining Time-Series and Sequence Data
Time-series plot
123
Objective (1)
  • forecasting future unknown values based on
    historical observed data
  • one step ahead
  • predict Yt1 from YtYt-1, Yt-2...
  • multiple steps ahead
  • predict Yt1Yt2 Yt3 .. from YtYt-1, Yt-2...
  • univariate just one variable
  • forecast of TL/ based on its own values

124
Objective (2)
  • multivariate more then one variable is
    forecasted simultaneously
  • inflation, stock index, exchange rate, interest
    rates influence each other
  • model based forecasts

125
Data preprosessing
  • Handling missing values
  • methods for cross-sectional data are not
    applicable
  • fill missing values by smoothing
  • Aggregation disaggregation problems
  • frequency of raw data and desired forecast
    frequency may be different
  • aggregation form low level to high level
  • sum,average, minimum,maximum, last,first
  • which function is used depends on the type of data

126
  • disaggregation from high to low frequency
  • different concept from OLAPs drill down where low
    level data is already available
  • here low level data is not available
  • GNP is available at the quarterly level
  • but may be needed in monthly forecast of GNP
  • Methods
  • constant value is used
  • equally partitioned
  • linear smoothing
  • cubic smoothing

127
Components of time series data
  • long-term or trend
  • indicate the general direction in which a time
    series variable is moving over a long interval of
    time
  • a line or curve
  • Cyclic movements or cyclic variations
  • long term oscillations about a trend line or
    curve may or may not be periodic

128
Components of time series data
  • Seasonal movements or variations
  • due to events that recur annually
  • sales of beverages varies from season to season
  • fuel oil sales in winter are higher than in
    summer
  • identical or nearly identical patterns during
    corresponding months or seasons of successive
    years
  • Irregular or random movements
  • sporadic motion of time series due to random or
    chance events

129
Decomposition of time series
  • product of the four variables
  • Yi TiCiSiIi,
  • or sum of the four variables
  • Yi Ti Ci Si Ii,
  • Identify each components and try to forecast each
    seperately
  • The I component can not be forecasted or
    predicted as it is irregular

130
How to determine trend
  • A moving average of order m
  • (y1y2...ym)/m
  • a moving average tends to reduce the amount of
    variation present in the data set
  • eliminates unwanted fluctuations
  • smoothing of time series
  • Weighted moving average of order m

131
Example
  • original data
  • 3 7 2 0 4 5 9 7 2
  • MA(3) 4 3 2 3 6 7 6
  • weighed
  • MA(3) 5.5 2.5 1 3.5 5.5 8 6.5
  • 1 4 1
  • the first WMA value
  • (134712)/(141)5.5
  • MA loses the data at the beginning and end of a
    series may sometimes

132
Detecting trend
  • By moving averages
  • cyclic, seasonal and irregular patterns in the
    data can be eliminated
  • resulting only the trend movement
  • Free-hand method
  • approximate curve or line is drawn to fit a set
    of data based on the users own judgement
  • costly and not reliable

133
least square method
  • Fit a line by regression
  • Yt a bt a linear trend
  • Yt a bt ct2 a quadratic trend
  • Yt atb,convert into linear by logarithmic
    transformation
  • lnYt lnablnt
  • Yt aebt, exponential
  • lnYt lna bt

134
Seasonal variations
  • identify and remove seasonal variations
  • deseasonalize the data for trend and cyclic
    analysis
  • A seasonal index set of number showing the
    relative values of a variable during the months
    of a year
  • yt yt/st,
  • where yt is deseasonalized variable
  • st is seasonal index value at time t

135
  • Estimation of seasonal variations
  • Seasonal index
  • Set of numbers showing the relative values of a
    variable during the months of the year
  • E.g., if the sales during October, November, and
    December are 80, 120, and 140 of the average
    monthly sales for the whole year, respectively,
    then 80, 120, and 140 are seasonal index numbers
    for these months
  • Deseasonalized data
  • Data adjusted for seasonal variations
  • E.g., divide the original monthly data by the
    seasonal index numbers for the corresponding
    months

136
(No Transcript)
137
  • Visually inspect the data
  • to identify trend cyclic or seasonal components

138
(No Transcript)
139
Mining Time-Series and Sequence Data
  • Time-series database
  • Consists of sequences of values or events
    changing with time
  • Data is recorded at regular intervals
  • Characteristic time-series components
  • Trend, cycle, seasonal, irregular
  • Applications
  • Financial stock price, inflation
  • Biomedical blood pressure
  • Meteorological precipitation

140
Mining Time-Series and Sequence Data Trend
analysis
  • A time series can be illustrated as a time-series
    graph which describes a point moving with the
    passage of time
  • Categories of Time-Series Movements
  • Long-term or trend movements (trend curve)
  • Cyclic movements or cycle variations, e.g.,
    business cycles
  • Seasonal movements or seasonal variations
  • i.e, almost identical patterns that a time series
    appears to follow during corresponding months of
    successive years.
  • Irregular or random movements

141
Estimation of Trend Curve
  • The freehand method
  • Fit the curve by looking at the graph
  • Costly and barely reliable for large-scaled data
    mining
  • The least-square method
  • Find the curve minimizing the sum of the squares
    of the deviation of points on the curve from the
    corresponding data points
  • The moving-average method
  • Eliminate cyclic, seasonal and irregular patterns
  • Loss of end data
  • Sensitive to outliers

142
Discovery of Trend in Time-Series (1)
  • Estimation of seasonal variations
  • Seasonal index
  • Set of numbers showing the relative values of a
    variable during the months of the year
  • E.g., if the sales during October, November, and
    December are 80, 120, and 140 of the average
    monthly sales for the whole year, respectively,
    then 80, 120, and 140 are seasonal index numbers
    for these months
  • Deseasonalized data
  • Data adjusted for seasonal variations
  • E.g., divide the original monthly data by the
    seasonal index numbers for the corresponding
    months

143
Discovery of Trend in Time-Series (2)
  • Estimation of cyclic variations
  • If (approximate) periodicity of cycles occurs,
    cyclic index can be constructed in much the same
    manner as seasonal indexes
  • Estimation of irregular variations
  • By adjusting the data for trend, seasonal and
    cyclic variations
  • With the systematic analysis of the trend,
    cyclic, seasonal, and irregular components, it is
    possible to make long- or short-term predictions
    with reasonable quality

144
Chapter 3 Data Preprocessing
  • Why preprocess the data?
  • Data cleaning
  • Data integration and transformation
  • Data reduction
  • Discretization and concept hierarchy generation
  • Summary

145
Summary
  • Data preparation is a big issue for both
    warehousing and mining
  • Data preparation includes
  • Data cleaning and data integration
  • Data reduction and feature selection
  • Discretization
  • A lot a methods have been developed but still an
    active area of research

146
References
  • D. P. Ballou and G. K. Tayi. Enhancing data
    quality in data warehouse environments.
    Communications of ACM, 4273-78, 1999.
  • Jagadish et al., Special Issue on Data Reduction
    Techniques. Bulletin of the Technical Committee
    on Data Engineering, 20(4), December 1997.
  • D. Pyle. Data Preparation for Data Mining. Morgan
    Kaufmann, 1999.
  • T. Redman. Data Quality Management and
    Technology. Bantam Books, New York, 1992.
  • Y. Wand and R. Wang. Anchoring data quality
    dimensions ontological foundations.
    Communications of ACM, 3986-95, 1996.
  • R. Wang, V. Storey, and C. Firth. A framework for
    analysis of data quality research. IEEE Trans.
    Knowledge and Data Engineering, 7623-640, 1995.

147
http//www.cs.sfu.ca/han
  • Thank you !!!
Write a Comment
User Comments (0)
About PowerShow.com