Title: Data Mining: Concepts and Techniques Slides for Textbook Chapter 3
1Data Mining Concepts and Techniques Slides
for Textbook Chapter 3
2Chapter 3 Data Preprocessing
- Why preprocess the data?
- Data cleaning
- Data integration and transformation
- Data reduction
- Discretization and concept hierarchy generation
- Summary
3Why Data Preprocessing?
- Data in the real world is dirty
- incomplete lacking attribute values, lacking
certain attributes of interest, or containing
only aggregate data - e.g., occupation
- noisy containing errors or outliers
- e.g., Salary-10
- inconsistent containing discrepancies in codes
or names - e.g., Age42 Birthday03/07/1997
- e.g., Was rating 1,2,3, now rating A, B, C
- e.g., discrepancy between duplicate records
4Why Is Data Dirty?
- Incomplete data comes from
- n/a data value when collected
- different consideration between the time when the
data was collected and when it is analyzed. - human/hardware/software problems
- Noisy data comes from the process of data
- collection
- entry
- transmission
- Inconsistent data comes from
- Different data sources
- Functional dependency violation
5Why Is Data Preprocessing Important?
- No quality data, no quality mining results!
- Quality decisions must be based on quality data
- e.g., duplicate or missing data may cause
incorrect or even misleading statistics. - Data warehouse needs consistent integration of
quality data - Data extraction, cleaning, and transformation
comprises the majority of the work of building a
data warehouse
6Multi-Dimensional Measure of Data Quality
- A well-accepted multidimensional view
- Accuracy
- Completeness
- Consistency
- Timeliness
- Believability
- Value added
- Interpretability
- Accessibility
- Broad categories
- intrinsic, contextual, representational, and
accessibility.
7Major Tasks in Data Preprocessing
- Data cleaning
- Fill in missing values, smooth noisy data,
identify or remove outliers, and resolve
inconsistencies - Data integration
- Integration of multiple databases, data cubes, or
files - Data transformation
- Normalization and aggregation
- Data reduction
- Obtains reduced representation in volume but
produces the same or similar analytical results - Data discretization
- Part of data reduction but with particular
importance, especially for numerical data
8Forms of data preprocessing
9Chapter 3 Data Preprocessing
- Why preprocess the data?
- Data cleaning
- Data integration and transformation
- Data reduction
- Discretization and concept hierarchy generation
- Summary
10Data Cleaning
- Importance
- Data cleaning is one of the three biggest
problems in data warehousingRalph Kimball - Data cleaning is the number one problem in data
warehousingDCI survey - Data cleaning tasks
- Fill in missing values
- Identify outliers and smooth out noisy data
- Correct inconsistent data
- Resolve redundancy caused by data integration
11Missing Data
- Data is not always available
- E.g., many tuples have no recorded value for
several attributes, such as customer income in
sales data - Missing data may be due to
- equipment malfunction
- inconsistent with other recorded data and thus
deleted - data not entered due to misunderstanding
- certain data may not be considered important at
the time of entry - not register history or changes of the data
- Missing data may need to be inferred.
12How to Handle Missing Data?
- Ignore the tuple usually done when class label
is missing (assuming the tasks in
classificationnot effective when the percentage
of missing values per attribute varies
considerably. - Fill in the missing value manually tedious
infeasible? - Use a global constant to fill in the missing
value e.g., unknown, a new class?! - Use the attribute mean to fill in the missing
value - Use the attribute mean for all samples belonging
to the same class to fill in the missing value
smarter - Use the most probable value to fill in the
missing value inference-based such as Bayesian
formula or decision tree
13How to Handle Missing Data?
- Use the most probable value to fill in the
missing value inference-based such as Bayesian
formula or decision tree - develop a submodel to predict the category or
numerical value of the missing case - income f(education,sex, age,...)
14Noisy Data
- Noise random error or variance in a measured
variable - Incorrect attribute values may due to
- faulty data collection instruments
- data entry problemshuman or computer error
- data transmission problems
- technology limitation
- inconsistency in naming convention
- Other data problems which requires data cleaning
- duplicate records
- incomplete data
- inconsistent data
15How to Handle Noisy Data?
- Binning method
- first sort data and partition into (equi-depth)
bins - then one can smooth by bin means, smooth by bin
median, smooth by bin boundaries, etc. - Clustering
- detect and remove outliers
- Combined computer and human inspection
- detect suspicious values and check by human
- Regression
- smooth by fitting the data into regression
functions
16Simple Discretization Methods Binning
- Equal-width (distance) partitioning
- It divides the range into N intervals of equal
size uniform grid - if A and B are the lowest and highest values of
the attribute, the width of intervals will be W
(B-A)/N. - The most straightforward
- But outliers may dominate presentation
- Skewed data is not handled well.
- Equal-depth (frequency) partitioning
- It divides the range into N intervals, each
containing approximately same number of samples - Managing categorical attributes can be tricky.
17Binning Methods for Data Smoothing
- Sorted data for price (in dollars) 4, 8, 9,
15, 21, 21, 24, 25, 26, 28, 29, 34 - Partition into (equi-depth) bins
- - Bin 1 4, 8, 9, 15
- - Bin 2 21, 21, 24, 25
- - Bin 3 26, 28, 29, 34
- Smoothing by bin means
- - Bin 1 9, 9, 9, 9
- - Bin 2 23, 23, 23, 23
- - Bin 3 29, 29, 29, 29
- Smoothing by bin boundaries
- - Bin 1 4, 4, 4, 15
- - Bin 2 21, 21, 25, 25
- - Bin 3 26, 26, 26, 34
18Cluster Analysis
19Regression
y
Y1
y x 1
Y1
x
X1
20Example use of regression to determine outliers
- Problem predict the yearly spending of AE
customers from their income. - Y yearly spending in millions TL
- X montly income in millions TL
21Data set
First data is probabliy an outlier But examining
only spending data that is Distribution of Y 200
spending of the first Customer can not be
identified as an outlier
- X Y
- 50,200
- 100,50
- 200,70
- 250,80
- 300,85
- 400,120
- 500,130
- 700,150
- 750,150
- 800,180
- 1000,200
22Regression
y
Y1200
y 0.15x 20
Y1
x
X150
X101000
Examine residuals Y1-Y1 is high But Y10-Y10 is
low So data point 1 is an outlier
inccome
23Notes
- Some methods are used for both smoothing and data
reduction or discretization - binning
- used in decision tress to reduce number of
categories - concept hierarchies
- Example price a numerical variable
- in to concepts as expensive, moderately prised,
expensive
24Chapter 3 Data Preprocessing
- Why preprocess the data?
- Data cleaning
- Data integration and transformation
- Data reduction
- Discretization and concept hierarchy generation
- Summary
25Data Integration
- Data integration
- combines data from multiple sources into a
coherent store - Schema integration
- integrate metadata from different sources
- Entity identification problem identify real
world entities from multiple data sources, e.g.,
A.cust-id ? B.cust- - Detecting and resolving data value conflicts
- for the same real world entity, attribute values
from different sources are different - possible reasons different representations,
different scales, e.g., metric vs. British units - price may include value added tax in one source
and not include in the other
26Handling Redundant Data in Data Integration
- Redundant data occur often when integration of
multiple databases - The same attribute may have different names in
different databases - One attribute may be a derived attribute in
another table, e.g., annual revenue - Redundant data may be able to be detected by
correlational analysis - rx,y ?ni1(xi-mean_x)(yi-mean_y)/(n-1)?x?y
- where is standard deviation
- as r?1 positive correlation x? y? or x? y?, one
is redundant - r ? 0 x and y are independent
- r ? -1 negative correlation when x? y? or x? y?
one redundant
27Positively and Negatively Correlated Data
28 Not Correlated Data
29Exercise
- Construct a data set of two variables X and Y
where there is a perfect relation between X and
Y - knowing value of X, Y corresponding to that X can
be predicted with no error - but correlation coefficient between X and Y is
- ZERO
30Handling Redundant Data in Data Integration
- how to detect redundancy for categorical
variables? - Find a measure of association for categorical
variables - Careful integration of the data from multiple
sources may help reduce/avoid redundancies and
inconsistencies and improve mining speed and
quality
31Data Transformation
- Smoothing remove noise from data
- Aggregation summarization, data cube
construction - From daily sales to monthly sales
- Generalization concept hierarchy climbing
- From original ages to categories
young,middle,senior - Normalization scaled to fall within a small,
specified range - min-max normalization
- z-score normalization
- normalization by decimal scaling
- Attribute/feature construction
- New attributes constructed from the given ones
- profit, financial ratios,
32Data Transformation Normalization
- When a score is outside the new range the
algorithm may give an error or warning message - in the presence of outliers regular observations
are squized in to a small interval
33Data Transformation Normalization
Good in handling outliers as the new range is
between -?to ? no out of range error
34Data Transformation Normalization
Where j is the smallest integer such that Max(
)lt1
- Ex max v984 then j 3 v0.984
- preserves the appearance of figures
35Attribute/feature construction
- Automatic attribute generation
- Using product or and opperations
- E.g. a regression model to explain spending of a
customer shown by Y - Using X1 age, X2 income, ...
- Y a b1X1 b2X2 a linear model
- a,b1,b2 are parameters to be estimated
- Y a b1X1 b2X2 cX1X2
- a nonlinear model a,b1,b2,c are parameters to be
estimated but linear in parameters as - X1X2 can be directly computed from income and
age
36Chapter 3 Data Preprocessing
- Why preprocess the data?
- Data cleaning
- Data integration and transformation
- Data reduction
- Discretization and concept hierarchy generation
- Summary
37Data Reduction Strategies
- Warehouse may store terabytes of data Complex
data analysis/mining may take a very long time to
run on the complete data set - Data reduction
- Obtains a reduced representation of the data set
that is much smaller in volume but yet produces
the same (or almost the same) analytical results - Data reduction strategies
- Data cube aggregation
- Dimensionality reductionremove unimportant
attributes - Data Compression
- Numerosity reductionfit data into models
- Discretization and concept hierarchy generation
38Data Cube Aggregation
- The lowest level of a data cube
- the aggregated data for an individual entity of
interest - e.g., a customer in a phone calling data
warehouse. - Multiple levels of aggregation in data cubes
- Further reduce the size of data to deal with
- Reference appropriate levels
- Use the smallest representation which is enough
to solve the task - Queries regarding aggregated information should
be answered using data cube, when possible
39Dimensionality Reduction
- Feature selection (i.e., attribute subset
selection) - Select a minimum set of features such that the
probability distribution of different classes
given the values for those features is as close
as possible to the original distribution given
the values of all features - reduce of patterns in the patterns, easier to
understand - Heuristic methods (due to exponential of
choices) - step-wise forward selection
- step-wise backward elimination
- combining forward selection and backward
elimination - decision-tree induction
40Example for irrelevent features
Average spending of a customer by income and age
Droping age Expain spending by Income only
Does not change results significantly
41Example of Decision Tree Induction
Initial attribute set A1, A2, A3, A4, A5, A6
A4 ?
A6?
A1?
Class 2
Class 2
Class 1
Class 1
Reduced attribute set A1, A4, A6
42Heuristic Feature Selection Methods
- There are 2d possible sub-features of d features
- Several heuristic feature selection methods
- Best single features under the feature
independence assumption choose by significance
tests. - Best step-wise feature selection
- The best single-feature is picked first
- Then next best feature condition to the first,
... - Step-wise feature elimination
- Repeatedly eliminate the worst feature
- Best combined feature selection and elimination
- Optimal branch and bound
- Use feature elimination and backtracking
43Example Economic indicator problem
- There are tens of macroeconomic variables
- say totally 45
- Which ones is the best predictor for inflation
rate three months ahead? - Develop a simple model to predict inflation by
using only a couple of those 45 macro variables - Best-stepwise feature selection
- The single macro variable predicting inflation
among the 45 is seleced first try 45 models say
/TL - repeat
- The k th variable is entered among 45-(k-1)
variables - Stop at some point introducing new variables
44Example cont.
- Best feature elimination
- Develop a model including all 45 variables
- Remove just one of them try 45 models each
excluding just one out of 45 variables - repeat
- Continue eliminating a new variable at each step
- Unitl a stoping criteria
- Rearly used comared to feature selection
45Data Compression
- String compression
- There are extensive theories and well-tuned
algorithms - Typically lossless
- But only limited manipulation is possible without
expansion - Audio/video compression
- Typically lossy compression, with progressive
refinement - Sometimes small fragments of signal can be
reconstructed without reconstructing the whole - Time sequence is not audio
- Typically short and vary slowly with time
46Data Compression
Original Data
Compressed Data
lossless
Original Data Approximated
lossy
47Wavelet Transforms
- Discrete wavelet transform (DWT) linear signal
processing - Compressed approximation store only a small
fraction of the strongest of the wavelet
coefficients - Similar to discrete Fourier transform (DFT), but
better lossy compression, localized in space - Method
- Length, L, must be an integer power of 2 (padding
with 0s, when necessary) - Each transform has 2 functions smoothing,
difference - Applies to pairs of data, resulting in two set of
data of length L/2 - Applies two functions recursively, until reaches
the desired length
48Principal Component Analysis
- Given N data vectors from k-dimensions, find c lt
k orthogonal vectors that can be best used to
represent data - The original data set is reduced to one
consisting of N data vectors on c principal
components (reduced dimensions) - Each data vector is a linear combination of the c
principal component vectors - Works for numeric data only
- Used when the number of dimensions is large
49- X Nk matrix of data in original form Xj is j th
column of data matrix X where j 1,.,k - xij is the j th variable of i th observation
- First normalize all variables X1..Xk to Z1.. Zk
- such that mean of Zj j1,..,k 0 E(Zi) 0
- bring the scales of all variables to same units
- The first principle component a1 is a linear
combination of the data Z - y1 Za1 where y1 is N1 vector Z is NK and
ak1 - y1,1 z1,1a1,1 z1,2a1,2... z1,ka1,k
- y1,2 z2,1a1,1 z2,2a1,2... z2,ka1,k
50- a1 is a unit vector that is norm of a1 1
- a1a1 1 or a1,12 a1,22... a1,k21
- Find the principle comp. vector a1 so as to get
the maximum variation in y1 - maximize the variance of y1
- var(y1) var(y1,1) var(y1,2)... var(y1,N)
- var(y1,1)i(y1,1-mean_y1)2
- note that mean_y10
- variation is y1 is y1,12 y1,22... y1,N2
- or in matrix notation (Za1)Za1 a1ZZa1
- dimensions1k kN Nk k1 11 a scaler
51The maximization problem
- Max variation in y
- subject to the restriction that a1a1
- or max a1ZZa1
- st a1a1
- find a1 a1,1 a1,2 a1,k so as to maximize the
variation in the first transformed vector y1 - L a1ZZa1 -?(a1a1-1)
- dL/da0
- take derivative of Lagrenian with respect to a
and - dL/da 2ZZa1 -2?a10 or ZZa1 ?a1 so
- multiplying by a1 a1ZZa1 ? a1a1 or
a1ZZa1 ? - a1 is the first eigen vector,? is the maximum
variation
52The Second principle component a2
- The second principle component a2 can be found in
a similar manner y2 Za2 where - y2,1 z1,1a2,1 z1,2a2,2... z1,ka2,k
- y2,2 z2,1a2,1 z2,2a2,2... z2,ka2,k
- a2 is a unit vector so a2a21
- Max variation in y
- subject to the restriction that a1a1
- max a2ZZa2
- st a2a21 and a1a20 the first and
second PC unit vectors are orthogonal to each
other
53- find a2 a2,1 a2,2 a2,k so as to maximize the
variation in the transformed compnents y2 - L a2ZZa2 -?2(a2a2-1) - ?a1a2
- dL/da20
- take derivative of lagrenian with respect to a2
and - dL/da 2ZZa2 -2?a2- ?a10 or ZZa1 ?a1 - ?a1
- multiplying by a2 a2ZZa2 ? a2a2 - ?a2 a1
- a2ZZa2 ?2 the second transformation vector is
the second eigen vector variation of the second
principle component is the second eigen value ?2
54- Notice that ?2lt ?1
- the first principle component is found in the
direction of maximum variation - the second has the second maximum variation whose
direction is orthogonal to the first PC - continue the optimization procedure until
finining c lt k components
55Principal Component Analysis
X2
Y1
Y2
X1
56Principal Component Analysis
X2
Y1
Y2
- Coordinate of Z1 is
- Z1,Z2 in
- when using principle
- components
- Z y1,y2 expressed in
- unit vectors a1 a2
Z1
57Exercise
- Z
- (-0.5,0.5),(1,1.5), (2.5,1), (-1,-1.5),
(-2,-1.5) - N5 k2
- plot the point in Z1 Z2 axis
- find the first principle component
- without a matrix formulation
- with a matrix formulation
- express the data in terms of first PC
- find the second principle component
- Draw on a figure the principle components
58Numerosity Reduction
- Parametric methods
- Assume the data fits some model, estimate model
parameters, store only the parameters, and
discard the data (except possible outliers) - Log-linear models obtain value at a point in m-D
space as the product on appropriate marginal
subspaces - Non-parametric methods
- Do not assume models
- Major families histograms, clustering, sampling
59Regression and Log-Linear Models
- Linear regression Data are modeled to fit a
straight line - Often uses the least-square method to fit the
line - Multiple regression allows a response variable Y
to be modeled as a linear function of
multidimensional feature vector - Log-linear model approximates discrete
multidimensional probability distributions
60Regress Analysis and Log-Linear Models
- Linear regression Y ? ? X
- Two parameters , ? and ? specify the line and are
to be estimated by using the data at hand. - using the least squares criterion to the known
values of Y1, Y2, , X1, X2, . - Multiple regression Y b0 b1 X1 b2 X2.
- Many nonlinear functions can be transformed into
the above. - Y aXb a nonlinear model
- logY loga blogX transform to a linear one
- Y aX1b1X2b2X3b3 a multiple nonlinear model
- logY loga b1logX1 b2logX2 b3logX3
transform to a linear one in parameters
61Regress Analysis and Log-Linear Models
- Log-linear models
- The multi-way table of joint probabilities is
approximated by a product of lower-order tables. - Probability p(a, b, c, d) ?ab ?ac?ad ?bcd
62Xa Xb
- A two dimensional cuboid each dim has two values
- measure is count each count is indicated in the
cells - there are 4 values to store for a 22 cuboid but
- P(xa,yc) 2/17 joint probabilities
- estimate these probabilities from marginal
probabilities - P(xa) 5/17 and P(yc) 7/17
- so store only the marginal counts
63- If the cuboid has 2 dimensions 10 values for each
in the base cuboid there are 1010100 values to
store - but there are 10 values for dim_1 and
- 10 values for dim_2 totally 20 counts
- by using log-linear models estimate the counts or
cell probabilities for the base or lower level
cuboids using stored higher level counts or
probabilities for higher level cuboids
64- If the cell are totally independent
- P(a,b) P(a,)P(,b)
- or taking the logarithms
- logP(a,b) logP(a,)logP(,b)
- if they are dependent to some extend a model is
used to estimate P(a,b) based on marginal
probabilities ???? - P(a,b) P(a,)?P(,b)? or taking logs
- logP(a,b) ?logP(a,)?logP(,b)
- a linear model in logs to estimate ? and ?
654-dimensional example
- P(a,b,c,d)kP(a,)?P(,b)?P(,c)?P(,d)?
P(,a,b)eP(,a,c)f - taking logs
- logP(a,b,c,d)logk ?logP(a,) ?logP(,b)
?logP(,c) ?logP(,d) elogP(,a,b)
flogP(,a,c) - estimate the parameters of the model
- predict the probability of a 4-D cell using
information about probabilities for 1-D cuboids
and 2-D cuboids
66Histograms
- A popular data reduction technique
- Divide data into buckets and store average (sum)
for each bucket - Can be constructed optimally in one dimension
using dynamic programming - Related to quantization problems.
67- Equiwidht the width of each bucket range is
uniform - Equidepth (equiheight)each bucket contains
roughly the same number of continuous data
samples - V-Optimalleast variance
- histogram variance is the is a weighted sum of
the original values that each bucket represents - bucket weight values in bucket
68- MaxDiff make a bucket boundry between adjacent
values if the difference is one of the largest k
differences - xi1 - xi gt max_k-1(x1,..xN)
69V-optimal design
- Sort the values
- assign equal number of values in each bucket
- compute variance
- repeat
- change buckets of boundary values
- compute new variance
- until no reduction in variance
- variance (n1Var1n2Var2...nkVark)/N
- N n1n2..nk,
- Vari ?nij1(xj-x_meani)2/ni,
70Clustering
- Partition data set into clusters, and one can
store cluster representation only - Can be very effective if data is clustered but
not if data is smeared - Can have hierarchical clustering and be stored in
multi-dimensional index tree structures - There are many choices of clustering definitions
and clustering algorithms, further detailed in
Chapter 8
71Sampling
- Allow a mining algorithm to run in complexity
that is potentially sub-linear to the size of the
data - Choose a representative subset of the data
- Simple random sampling may have very poor
performance in the presence of skew - Develop adaptive sampling methods
- Stratified sampling
- Approximate the percentage of each class (or
subpopulation of interest) in the overall
database - Used in conjunction with skewed data
- Sampling may not reduce database I/Os (page at a
time).
72Sampling methods
- Simple random sample without replacement (SRSWOR)
of size n - n of N tuples from D nltN
- P(drawing any tuple)1/N all are equally likely
- Simple random sample with replacement (SRSWR) of
size n - each time a tuple is drawn from D, it is recorded
and then replaced - it may be drawn again
73Sampling methods cont.
- Cluster Sample if tuples in D are grouped into M
mutually disjoint clusters then an SRS of m
clusters can be obtained where m lt M - Stratified Sample if D is divided into mutually
disjoint parts called strata. - Obtain a SRS at each stratum
- a representative sample when data are skewed
- Ex customer data a stratum for each age group
74- Sampling complexity is sublinear to the size of
data - An aggregation query
- given Rr1,..,rN N records
- estimate ?(f)(1/N(?Ni1 f(ri)) based on a SRSWR
of size n lt N , where - f is specified real valued function
- if n sufficiently small to N
- no distinction between SRSWR and SRSWOR
75Confidence intervals and sample size
- Specifying a confidence interval and
- you should be able to determine n number of
samples required so that the sample mean will be
within the confidence interval with (1-p)
confident - n is very small compared to the size of the
database N - nltltN
76Sampling
SRSWOR (simple random sample without
replacement)
SRSWR
77Sampling
Cluster/Stratified Sample
Raw Data
78Hierarchical Reduction
- Use multi-resolution structure with different
degrees of reduction - Hierarchical clustering is often performed but
tends to define partitions of data sets rather
than clusters - Parametric methods are usually not amenable to
hierarchical representation - Hierarchical aggregation
- An index tree hierarchically divides a data set
into partitions by value range of some attributes - Each partition can be considered as a bucket
- Thus an index tree with aggregates stored at each
node is a hierarchical histogram
79Chapter 3 Data Preprocessing
- Why preprocess the data?
- Data cleaning
- Data integration and transformation
- Data reduction
- Discretization and concept hierarchy generation
- Summary
80Discretization
- Three types of attributes
- Nominal values from an unordered set
- Ordinal values from an ordered set
- Continuous real numbers
- Discretization
- divide the range of a continuous attribute into
intervals - Some classification algorithms only accept
categorical attributes. - Reduce data size by discretization
- Prepare for further analysis
81Discretization and Concept hierachy
- Discretization
- reduce the number of values for a given
continuous attribute by dividing the range of the
attribute into intervals. Interval labels can
then be used to replace actual data values. - Types of discretization
- Supervised (class information used)
- unsupervised (not used or not available)
- dynamic - static
- local - global
- top down - bottom up
82Concept hierarchies
- reduce the data by collecting and replacing low
level concepts (such as numeric values for the
attribute age) by higher level concepts (such as
young, middle-aged, or senior). - detail is lost
- generalized data
- more meaningful
- easier to interpret,
- require less space
- mining on a reduced data set require
- fewer input/output opperations
- be more efficient
83Discretization Process
- Univariate - one feature at a time
- Steps of a typical discritization process
- sorting the continuous values of the feature
- evaluating a cut-point for splitting or adjacent
intervals for merging - splitting or merging intervals of continuous
value - top-down approach intervals are split
- chose the best cut point to split the values
into two partitions - until a stopping criterion is satisfied
84- bottom-up approach intervals are merged
- find best pair of intervals to merge
- until stopping criterion is satisfied
- stopping at some point
- lower better understanding less accuracy
- high poorer understanding higher accuracy
- Stopping criteria
- number of intervals reach a critical value
- number of observations in each interval exceeds a
value - all observations in an interval are of the same
class - a minimum information gain is not satisfied
85Generalized Splitting Algorithm
- S sorted values of feature f
- splitting(S)
- if stopping criterion satisfied
- return
- T GetBestSplitPoint(S)
- S1GetLeftPart(S,T)
- S2GetRightPart(S,T)
- Splitting(S1)
- Splitting(S2)
86Discretization and concept hierarchy generation
for numeric data
- Binning (see sections before)
- Histogram analysis (see sections before)
- Clustering analysis (see sections before)
- Entropy-based discretization
- Segmentation by natural partitioning
871R a supervised discretization method
- After sorting the continuous values,
- divides the range of continuous values into a
number of disjoint intervals and adjusts the
boundaries based on the class labels - Example
- 11 14 15 18 19 20 21 ? 22 23 25 30 31 33 35 ? 36
- R C C R C R C ? R C C R C R
C ? R - C ? C
? R - interval of class C from 11-21
- another interval of C from 22-35
- last of class R including just 36
- the two leftmost intervals are merged as they
predict the same class - stoping each interval should contain a
prespecified minimum number of instances
88Entropy-Based Discretization
- Given a set of samples S, if S is partitioned
into two intervals S1 and S2 using boundary T,
the entropy after partitioning is - The boundary that minimizes the entropy function
over all possible boundaries is selected as a
binary discretization. - The process is recursively applied to partitions
obtained until some stopping criterion is met,
e.g., - Experiments show that it may reduce data size and
improve classification accuracy
89Entropy
- A measure of (im)purity of an sample variable S
is defined as - Ent(S) -?spslog2ps
- s is a value of S
- Ps is its estimated probability
- average amount of information per event
- Information of an event is
- I(s) -log2ps
- information is high for lower probable events and
low otherwise
90- it can be interpreted as the number of bits to
encode that event - if p1/2
- -log21/2 1 bit is required to encode that event
- if p1/4
- -log21/4 2 bits are required to encode that
event - if p1/3
- -log21/3-(loge1/3)/loge21.58 bits are required
to encode that event
91- Entropy is the highest when each event is equally
likely - Example tossing a coin
- p is probability of head
- 1-p probability of tail
- Entropy is 0 if p1 or P0
- ent(s) -1log2(1)-0log20 -10-0log200
- The lower the entropy the higher the information
content of the event
92- Entropy is 1 p 0.5 1-p0.5 event are equally
likely for a fair coin - ent(s) -1/2log2(1/2)-1/2log21/2
- ent(s) -1/2(-1) -1/2(-1)1
- if head or tail probabilities are unequal
- entropy is between 0 and 1
- class entropy is computed in a similar manner
- Entropy is 0 if all members of S belong to the
same class - entropy of a partition is the weighted average of
all entropies of all classes - ent(S1,s2)S1/Sent(S1)S2/Sent(S2)
93Temperature Example
- A sample of temperatures for 14 days
- 64 65 68 69 70 71 72 75 80 81 83 85
- y n y y y n n y n y
y n - y y
- classes are y or n
- playing tennis in a day whose temperature is
given - y or n yes or no
- entropy of the whole sample without partitioning
- ent(S)(-6/14)log2(6/14)(-8/14)log2(8/14)
- -0.428(-1.22)(-0.57)(-0.80)0.96
94- Splitting value of T71.5
- S1 Tlt71.5 4 y, 2 n
- S2 Tgt 71.5 5 y, 3 n
- I(S,T)S1/Sent(S1)S2/Sent(S2)
- (6/14)ent(S1)(8/14)ent(S2)
- ent(S1)(-2/6)log2(2/6)(-4/6)log2(4/6)
- -0.333(-1.58)(-0.666(-0.58)0.90
ent(S2)(-5/8)log2(5/8)(-3/8)log2(3/8) - -0.625(-0.67)(-0.375(-1.41)0.94
- ent(S1,S2) (6/14)0.90(8/14)0.940.3850.5379
22
95- Information gain
- 0.96-0.9220.04
- try all possible partitions
- chose the partition providing maximum information
gain - then reccursively partition each interval
- until the stoping criteria is satisfied
- when information gain lt a treshold value
96Segmentation by natural partitioning
- 3-4-5 rule can be used to segment numeric data
into - relatively uniform, natural intervals.
- If an interval covers 3, 6, 7 or 9 distinct
values at the most significant digit, partition
the range into 3 equi-width intervals (equiwith
intervals for 3,6,9) - 3 intervals in grouping of 2-3-2 for 7
- If it covers 2, 4, or 8 distinct values at the
most significant digit, partition the range into
4 intervals - If it covers 1, 5, or 10 distinct values at the
most significant digit, partition the range into
5 intervals
97Example of 3-4-5 rule
(-4000 -5,000)
Step 4
98Example 3.5
- profits at different branches of AE
- Min-351,976.00 Max4,700,896.50
- 5th percentile -159,876
- 95th percentile 1,838,761
- low-159,876, high1,838,761
- msd1,000,000
- rounding low to milliondigit low-1,000,000
- rounding high to milliondigit high2,000,000
- (2,000,000-(-1,000,000))/1,000,0003
- partitioned into 3 equiwidth subsegments
- (-1,000,000...0,(0...1,000,000,(1,000,000...2,00
0,000
99Example 3.5
- First interval
- low(-1,000,000)ltmin
- adjust the left boundary
- msd of min 100,000
- rounding min to -400,000 min
- first interval(-400,000...0
- last interval (1,000,000...2,000,000 not cover
max - max lthigh
- a new interval (2,000,000...5,000,000
- rounding max to 5,000,000
100Example 3.5
- Recursively each interval can be further
partitioned according to 3-4-5 rule - first interval (-400,00...0 partitioned into 4
subintervals - (-400,000...-300,000,(-300,000...-200,000,
- (-200,000...-100,000,(-100,000...0
- second interval (0...1,000,000 partitioned into
5 subintervals - (0...200,000,(200,000...400,000,
- (400,000...600,000,(600,000...800,000
- (800,000...1,000,000
- last interval into 3 subintervals
- (2,000,000..3,000,000(3,000,000..4,000,000,(4,00
0,000..5,000,000
101Concept hierarchy generation for categorical data
- finite but possibly large distinct values with no
ordering - geographic location,job categories, item type
- Specification of a partial ordering of attributes
explicitly at the schema level by users or
experts - Specification of a portion of a hierarchy by
explicit data grouping - Specification of a set of attributes, but not of
their partial ordering - Specification of only a partial set of attributes
102Specification of a set of attributes
- Concept hierarchy can be automatically generated
based on the number of distinct values per
attribute in the given attribute set. The
attribute with the most distinct values is placed
at the lowest level of the hierarchy.
15 distinct values
country
65 distinct values
province_or_ state
3567 distinct values
city
674,339 distinct values
street
103Time Series Data
- A time series database consists of sequences of
values or events changing with time. - The values are typically measured at equal time
intervals - Examples
- daily closing values of stock market index
- sales of products
- economic variables GNP, exchange rates...
104Objective (1)
- forecasting future unknown values based on
historical observed data - one step ahead
- predict Yt1 from YtYt-1, Yt-2...
- multiple steps ahead
- predict Yt1Yt2 Yt3 .. from YtYt-1, Yt-2...
- univariate just one variable
- forecast of TL/ based on its own values
105Objective (2)
- multivariate more then one variable is
forecasted simultaneously - inflation, stock index, exchange rate, interest
rates influence each other - model based forecasts
106Data preprocessing
- Handling missing values
- methods for cross-sectional data are not
applicable - fill missing values by smoothing
- Aggregation disaggregation problems
- frequency of raw data and desired forecast
frequency may be different - aggregation form low level to high level
- sum,average, minimum,maximum, last,first
- which function is used depends on the type of data
107- disaggregation from high to low frequency
- different concept from OLAPs drill down where low
level data is already available - here low level data is not available
- GNP is available at the quarterly level
- but may be needed in monthly forecast of GNP
- Methods
- constant value is used
- equally partitioned
- linear smoothing
- cubic smoothing
108Components of time series data
- long-term or trend
- indicate the general direction in which a time
series variable is moving over a long interval of
time - a line or curve
- Cyclic movements or cyclic variations
- long term oscillations about a trend line or
curve may or may not be periodic
109- Seasonal movements or variations
- due to events that recur annually
- sales of beverages varies from season to season
- fuel oil sales in winter are higher than in
summer - identical or nearly identical patterns during
corresponding months or seasons of successive
years - Irregular or random movements
- sporadic motion of time series due to random or
chance events
110Decomposition of time series
- product of the four variables
- Yi TiCiSiIi,
- or sum of the four variables
- Yi Ti Ci Si Ii,
111How to determine trend
- A moving average of order m
- (y1y2...ym)/m
- a moving average tends to reduce the amount of
variation present in the data set - eliminates unwanted fluctuations
- smoothing of time series
- Weighted moving average of order m
112Example
- original data
- 3 7 2 0 4 5 9 7 2
- MA(3) 4 3 2 3 6 7 6
- weighed
- MA(3) 5.5 2.5 1 3.5 5.5 8 6.5
- 1 4 1
- the first WMA value
- (134712)/(141)5.5
- MA loses the data at the beginning and end of a
series may sometimes
113Detecting trend
- By moving averages
- cyclic, seasonal and irregular patterns in the
data can be eliminated - resulting only the trend movement
- Free-hand method
- approximate curve or line is drawn to fit a set
of data based on the users own judgement - costly and not reliable
114least square method
- Fit a line by regression
- Yt a bt a linear trend
- Yt a bt ct2 a quadratic trend
- Yt atb,convert into linear by logarithmic
transformation - lnYt lnablnt
- Yt aebt, exponential
- lnYt lna bt
115Seasonal variations
- identify and remove seasonal variations
- deseasonalize the data for trend and cyclic
analysis - A seasonal index set of number showing the
relative values of a variable during the months
of a year - yt yt/st,
- where yt is deseasonalized varliable
- st is seasonal index value at time t
116- Visually inspect the data
- to identify trend cyclic or seasonal components
117Chapter 3 Data Preprocessing
- Why preprocess the data?
- Data cleaning
- Data integration and transformation
- Data reduction
- Discretization and concept hierarchy generation
- Summary
118Summary
- Data preparation is a big issue for both
warehousing and mining - Data preparation includes
- Data cleaning and data integration
- Data reduction and feature selection
- Discretization
- A lot a methods have been developed but still an
active area of research
119References
- D. P. Ballou and G. K. Tayi. Enhancing data
quality in data warehouse environments.
Communications of ACM, 4273-78, 1999. - Jagadish et al., Special Issue on Data Reduction
Techniques. Bulletin of the Technical Committee
on Data Engineering, 20(4), December 1997. - D. Pyle. Data Preparation for Data Mining. Morgan
Kaufmann, 1999. - T. Redman. Data Quality Management and
Technology. Bantam Books, New York, 1992. - Y. Wand and R. Wang. Anchoring data quality
dimensions ontological foundations.
Communications of ACM, 3986-95, 1996. - R. Wang, V. Storey, and C. Firth. A framework for
analysis of data quality research. IEEE Trans.
Knowledge and Data Engineering, 7623-640, 1995.
120http//www.cs.sfu.ca/han